Serverless inferencing on Kubernetes
Key Notes
Example #1
Provide Inference Location
Canary Location
Monitoring and explainability of models in production
Success Metrics for ML Model
1. Monitoring model performance
2. Monitoring metrics related to incoming data
3. Detecting outliers and drift
4. Explaining model predictions
Key aspects
Monitoring system requires functionality to determine when significant changes to data and predictive distributions happen
Seldon Core provides a dedicated /send-feedback API endpoint accepting labels and performing user-defined metric calculations
Drift Detector - The goal of the drift detector is therefore to identify when the distribution of the requests for the deployed model starts to diverge from the training data and model predictions
Model Monitoring - a KNative broker which can farm these out as desired via programmable triggers to serverless components such as outlier, drift and adversarial detection
More Reads - Minio - High performance object storage
Keep Thinking!!!
Key Notes
- KNative serverless paradigm to provide a serverless machine learning inference solution
- Frameworks - MLFlow, Kubeflow
- Handling multiple machine learning frameworks in a consistent manner.
- Updating running models with new versions.
- Scaling models appropriately with constraints.
- Monitoring models.
- Canaries allow users to split a small percentage of traffic to their new model
- KFServing is a project that was created within the Kubeflow
- Transformers allow focused data transformations of the request and response from the model
Example #1
Provide Inference Location
- Create a storage initializer to download the artifacts from any popular storage (Google Storage, Amazon S3, Azure, local disk) and load onto the server.
- Wire up networking so an endpoint is made available for inference requests
Canary Location
Monitoring and explainability of models in production
Success Metrics for ML Model
1. Monitoring model performance
2. Monitoring metrics related to incoming data
3. Detecting outliers and drift
4. Explaining model predictions
Key aspects
Monitoring system requires functionality to determine when significant changes to data and predictive distributions happen
Seldon Core provides a dedicated /send-feedback API endpoint accepting labels and performing user-defined metric calculations
Drift Detector - The goal of the drift detector is therefore to identify when the distribution of the requests for the deployed model starts to diverge from the training data and model predictions
Model Monitoring - a KNative broker which can farm these out as desired via programmable triggers to serverless components such as outlier, drift and adversarial detection
More Reads - Minio - High performance object storage
Keep Thinking!!!
