Weekend Learning - Introducing KFServing: Serverless Model Serving on Kubernetes

KF Serving

• Serving story for Kubeflow
• The concepts behind Kubernetes

• Serving Frameworks (Seldon core, ml lambdas, Tensorflow Serving)

• Consistent interfaces for different frameworks

• ServiceAccount for access
• Canary - 2 way split (New / Old)
• Default Standard Deployment
• Canary - Addressable primary / default
• Experimental traffic handling
• Similar to A / B Testing

• KFServing for ML Problems
• Knative - Resource Model
• Production Features of ML KFServing 0.2

• Use cases at Bloomberg

• Serving models in production
• Scaling and handling traffic
• End to End implementation/scalability and load handling

• Model production requirements
• A lot of out of box features for production-grade implementation

• Bloombergs use cases
• All data / NLP

KF Serving Transformer concept

• Implement pre and post processing
• Add transformer to inference service

Model Explanation

• Alibi library

• Accessibility to prediction URL

Kafka Implementation Example
A / B Testing Approach
CI / CD Pipeline

MNIST kfserving

• Preprocess / postprocess in transformer
• Download image
• Run prediction
• Result upload to bucket
• Custom model to process
• Upload to bucket

More Reads
Link1
Sample for KFServing SDK with a custom image
Predict on a InferenceService using Tensorflow

Happy Learning!!!