Research paper read - Serverless inferencing on Kubernetes

Serverless inferencing on Kubernetes

Key Notes

• KNative serverless paradigm to provide a serverless machine learning inference solution
• Frameworks - MLFlow, Kubeflow

Deployment / Inference Challenges

• Handling multiple machine learning frameworks in a consistent manner.
• Updating running models with new versions.
• Scaling models appropriately with constraints.
• Monitoring models.
• Canaries allow users to split a small percentage of traffic to their new model

KFServing

• KFServing is a project that was created within the Kubeflow
• Transformers allow focused data transformations of the request and response from the model

Example #1 
Provide Inference Location

• Create a storage initializer to download the artifacts from any popular storage (Google Storage, Amazon S3, Azure, local disk) and load onto the server.
• Wire up networking so an endpoint is made available for inference requests

Example #2
Canary Location

Monitoring and explainability of models in production
Success Metrics for ML Model
1. Monitoring model performance
2. Monitoring metrics related to incoming data
3. Detecting outliers and drift
4. Explaining model predictions

Key aspects
Monitoring system requires functionality to determine when significant changes to data and predictive distributions happen

Seldon Core provides a dedicated /send-feedback API endpoint accepting labels and performing user-defined metric calculations

Drift Detector - The goal of the drift detector is therefore to identify when the distribution of the requests for the deployed model starts to diverge from the training data and model predictions

Model Monitoring - a KNative broker which can farm these out as desired via programmable triggers to serverless components such as outlier, drift and adversarial detection

More Reads - Minio - High performance object storage

Keep Thinking!!!