ML Model Deployment Strategies

Real time - API as Service

• Google Assistant speech recognition
• Twitter feed

Streaming Approach

Ref - Link

Batch Prediction

Examples

• TripAdvisor hotel ranking
• Netflix recommendations

Hybrid Approach

Door Dash

• Restaurant recommendations use batch predictions
• Within each restaurant, item recommendations use online predictions

Netflix

• Title recommendations use batch predictions
• Row orders use online predictions

Ref - Link

Model Testing

• Canary testing - New model alongside existing system. Slowly increase the traffic to new model
• A/B testing - New model alongside existing system. A percentage of traffic is routed to new model based on routing rules
• Interleaved experiments - Especially useful for ranking/recsys. Take recommendations from both model A & B. Mix them together and show them to users
• Shadow testing - New model in parallel with existing system. New model’s predictions are logged, but not show to users

Ref - Link

Production ML

Ref - Link

Keep Exploring!!!