Model Deployment Architecture - The two sides of coin

 My implementation experience and lessons :)

Product Implementation (2012-2014)

• Integrated in product
• Jobs scheduled for midnight
• Workflow to monitor variations
• Forecast updated every day for store
• Everything custom-coded formula embedded
• Weighted moving average 
• Step up / Step down moving average approach

Batched State of Art (2021)

Recommendations AWS

• ETL / Glue jobs to get featured
• Full pull/delta pull scripts
• Feature engineering scripts
• Custom segmentation scripts
• Batch jobs to run models
• Large-scale recommendations generation
• Infra kubeflow setup 
• Leverage existing Kubeflow monitoring setup

Forecasting State of Art (2021)

Kubeflow + AWS

• ETL / Glue jobs to get features
• Full pull/delta pull scripts
• Feature engineering scripts
• Custom segmentation scripts
• Batch jobs to run models
• Kubeflow pipelines for the forecast
• Results persist in Redshift DB
• Infra kubeflow setup 
• Leverage existing kubeflow monitoring setup

Realtime State of Art (2022)

Real-time streaming / Vision Solution

• AWS Lamdbda-based approach
• Vision + Docker + AWS Lambda
• Request monitoring / logging

Keep Exploring!!!

AWS SaaS Blueprint

Interesting session on building SaaS Apps. Include different aspects hosting, keeping multiple versions, Isolating resources, sharing

• Categorize patterns / Building Blocks
• Manage and Operate Business
• Tenant aware, Tier aware metrics

• Flavors of Isolation
• Parition workloads
• Partition data

• Domain / Sub domain / Tenant work load
• Users to tenants mapping
• Orchestrate
• Tier - Throttle - Pattern

• Control Plane - Horizontal Services, Manage / scale. Services manage tenants
• Application Plane - Nature of SaaS Application, Isolation, Data Partitioning, Micro-Services

• Serverless - Control Plane
• Containers - App Plane
• Silo - Dedicated to Tenant
• Pool Side - Everything shared

• Silo - Deployment (Data Privacy / API Deployment)
• Serveless for Control Plane (Monitor / Measure)
• Onboard Architecture
• Self Service / Sign up
• Registration Service
• Create tenant / Routing / Policies
• Roles / Tiers / Isolation Policies
• Bill / Provision / Policies / Routing

• Provisioning experience
• Register - Provision / Self Service 
• Pooled Tenants

• Tenant Aware identity
• Access / Tenant Context
• Tokens with client context

• Tenant aware
• sub-domain
• which user pool to authenticate against
• Tenant management middle of execution

• Multiple / Global user pool
• MFA exclusive to user pool

• User pool mapped to

• User Management
• Admin - Tenants - Users

• Amazon Cognito - Implement secure, frictionless customer identity and access management that scales
• Billing providers - Third party
• Setup account with billing provider configure plans
• Send Activity data

• Metrics

• Application Plane
• Full Stack Silo/FUll Stack Pool

• Provisioning / Tiered 

• Full Stack Silo patterns

• Full Stack Pooled patterns

• Mixed models

• Multi-region

• EKS

• Isolation Implementation
• Full Stack Silo = Isolation off
• Full Stack Isolation
• Resource level isolation

 

• Deployment driven isolation

• Runtime enforced isolation

Keep Exploring!!!

SaaS Architecture References

A few questions on building SaaS.

• Are we going to create instances for each customer
• What is going to be common / pooled vs dedicated

Dedicated Service and DB Layer (Silo)

Ref - Link

Ref - Link

Key aspects

• Identity Management
• Monitoring
• Tenant onboarding

• Amazon Cognito - User onboarding is handled here
• Tenant Details are stored in DynamoDB
• Amazon code Pipeline to create custom namespaces and services

For AWS Lambda

Ref - Link

Ref - Link

Pooled Service and Dedicated DB (Pooled)

• Multi-Tenant Architecture: Designing a SaaS Application
• Multi tenant Architecture for a SaaS Application on AWS
• AWS Serverless SaaS Workshop
• Full stack isolation
• SaaS Architecture
• Building a Multi-Tenant SaaS Solution Using Amazon EKS
• AWS SaaS Boost

Keep Exploring!!!

Unwritten rules to prepare architecture slides

• Use transparent logos, Keep all logos professional and not different backgrounds
• Use logos available from the company websites
• Follow the background as-is, GCP has a grayish background, AWS most of it has black background, and Azure blue
• Keep the page full, well spread out like the architecture slide should look and include every possible flow
• Reference templates
• Keep all font sizes consistent, Component level fonts, module level fonts
• Differentiate different blocks based on functionality
• Add a narrative wherever needed
• Technical Content is king, Communication is impact but presentation is the catalyst
• Iterate/review / get inspiration and keep working

AWS - Link1

Azure - Link1, Link2

GCP - Link1, Link2

Keep Exploring!!!

AWS - Personalization - Good Read

Personalize cross-channel customer experiences 

Key steps

• Collect data from multiple sources to store in Amazon Simple Storage Service (Amazon S3).
• Use AWS Step Functions to orchestrate data onboarding and feature engineering.
• Build segments and predictions using SageMaker.
• Use propensity scores for display targeting.
• Send personalized messaging using Amazon Pinpoint.
• Integrate real-time personalized suggestions using Amazon Personalize.

Code Examples - Link

User Segmentation - Link, Amazon Personalise

Data sources connect to AWS Kinesis Link

Build, tune, and deploy an end-to-end churn prediction model using Amazon SageMaker Pipelines

Keep Exploring!!!

AWS Lambda - Docker - Building Models

Sometimes outdated blogs do not get results but to know something from high-level to ground level need to try it end to end.

Sharing some lessons in building example link

On the Mac system installed docker, and python already.

Lesson #1 - Creating empty files in mac

• file creation - touch requirements.txt
• touch app.py copy code
• touch Dockerfile, copy script

Lesson #2 - Installing Amazon CLI

• CLI Install - Steps mentioned in link
• For Mac - Download from link

Post Installation Validation commands

$ which aws

/usr/local/bin/aws 

$ aws --version

aws-cli/2.7.24 Python/3.8.8 Darwin/18.7.0 botocore/2.4.5

Lesson #3 - User Creation

Under IAM create users - https://us-east-1.console.aws.amazon.com/iamv2/home#/users

Create user, For the user create key, You will have access id, access key

Configure your AWS CLI with command

aws configure

Lesson #4 - Amazon ECR Creation and push image

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 261091142.dkr.ecr.us-west-2.amazonaws.com

docker build -t demo .

docker tag demo:latest 261091142.dkr.ecr.us-west-2.amazonaws.com/demo:latest

docker push 261091142.dkr.ecr.us-west-2.amazonaws.com/demo:latest

Lesson #5 - Create Lambda function from the Image

Lesson #6 - Give AWS S3 Read access for the user to read the input

It is not rocket science but some debug science is needed. You know what to do but finding the flows in UI takes time

The image did not compile some errors. This is an outdated blog. 

AWS CLI, Dockerizing, ECR, Lambda + S3. It was a good end-to-end flow.

Keep Exploring!!!

AWS Lambda Basic Examples

 

An example method for parameter handling

Keep Exploring!!!