Learning Materials - DL - ML - DE - MLOps - NLP

Ready to Learn collated examples

• machine-learning-zoomcamp
• zoomcamp-analytics
• data-engineering-zoomcamp
• mlops-zoomcamp
• MLOps project with Loan Data
• Large Language Models Course
• Google-Machine-Learning-for-Solutions-Architects
• A collection of design patterns/idioms in Python
• How to add tests to your data pipelines
• Data Pipeline Design Patterns - #1. Data flow patterns
• Data Pipeline Design Patterns - #2. Coding patterns in Python
• energy-forecasting
• How to build and automate a python ETL pipeline with airflow on AWS EC2 

Good Data related papers - Link

Neatly organized here: Link

Good Data Engineering Papers

• BtrBlocks: Efficient Columnar Compression for Data Lakes
• The Story of AWS Glue
• Building a serverless Data Lakehouse from spare parts⋆
• The Snowflake Elastic Data Warehouse

Updated list of engineering papers worth reading.

1. Google File System - https://lnkd.in/d2-wnyqZ

2. Map Reduce Big Data Algorithm - https://lnkd.in/dvE8-s8M

3. BigTable NoSQL Document Store - https://lnkd.in/drmvvSAK

4. Colossus Next Gen File Store - https://lnkd.in/dERKhwMf

5. Megastore Large Object Store - https://lnkd.in/d5JDs2-K

6. Monarch Time Series DB - https://lnkd.in/d3kH_NCp

7. Chubby Distributed Lock Management - https://lnkd.in/dYy-w5rW

8. Spanner Distributed Database - https://lnkd.in/d6Emnycp

9. Spanner - CAP theorem considerations - https://lnkd.in/dq29BAWQ

10. Dapper Tracing System - https://lnkd.in/dm36-6jn

11. Borg Cluster Management - https://lnkd.in/dnveV-HU

12. Zanzibar Authentication System - https://lnkd.in/d5Vf7sRD

13. Pregel Graph Processing - https://lnkd.in/daq4576Y

14. Napa - Data Warehousing - https://lnkd.in/dbEfsa5B

15. Napa - Partitioning Algorithm - https://lnkd.in/dkhA7efJ

16. TensorFlow - Machine Learning at Scale - https://lnkd.in/d-4NfV2Z

17. Google F1 - Fast Analytics - https://lnkd.in/dbZqEKuf

18. HALP - YouTube Content Delivery Network - https://lnkd.in/dHzJtUc7

19. Mesa - Data Warehousing - https://lnkd.in/dFJ_Jrz6

20. Google Firestore - https://lnkd.in/drtEN9qR

21. Amazon Aurora DB Architecture - https://lnkd.in/dcevpwFt

22. Dynamo DB NoSQL Database - https://lnkd.in/dMD8C_WK

23. Apple Foundation DB - NewSQL database - https://lnkd.in/dG75i_9K

24. TikTok Monolith - Embedding in real-time - https://lnkd.in/dcjBXCnc

25. Scalability at what COST - https://lnkd.in/dJ9ScYKq

26. Gorilla - Time Series DB - https://lnkd.in/d3AeN2kB

27. Cassandra - NoSQL DB - https://lnkd.in/d-_nhtED

28. FlexiRaft - Distributed Consensus Tradeoffs - https://lnkd.in/dX3nMvmt

29. Memcache - In-memory Cache at Facebook - https://lnkd.in/dKeYK67g

30. Millisampler Network Sampling - https://lnkd.in/dsj9FuD6

31. TAO Graph Database - https://lnkd.in/daasJpYf

32. MineSweeper - Root Cause Analysis - https://lnkd.in/dEsd6iwj

33. Facebook Prophet - Forecasting at Scale - https://lnkd.in/daCmAjak

34. Facebook ShardManager - https://lnkd.in/dDy9Dp2h

35. Hive - Map Reduce Jobs - https://lnkd.in/dpV8BM2R

36. Apache Thrift - Definition Language - https://lnkd.in/d7NzhP54

37. Meta Twine - Cluster Management System - https://lnkd.in/d5t7VFKE

38. Meta ServiceRouter - Service mesh - https://lnkd.in/dVnkv_bV

39. Apache Hadoop - Distributed File System - https://lnkd.in/dHsQu9FN

40. Apache Kafka - Event Bus - https://lnkd.in/dyxuKbMb

41. Apache Flink - https://lnkd.in/dn_gMvaR

NLP

NLP-Python

Advanced_NLP_Spacy

Vision Notes

The U-Net (actually) explained in 10 minutes

U-Net Implementation With PyTorch

Efficient Python Tricks and Tools for Data Scientists

mathdatasimplified

It takes time to read - learn - experiment - build expertise. Learning Never Ends !!!!

Keep Exploring!!!

MLOps Notes

Why MLOps

Quicker experimentation cycle -> More models

Quicker productization cycle -> More models in production

Full traceability for all models -> More models in production safely and scalability"

Tools for MLOps

• Data Analysis - Python, Pandas
• Source Control  - Git
• Test & Build Services - PyTest & Make
• Deployment Services - Git, DVC
• Model & Dataset Registry - DVC[aws s3]
• Feature Store - Project code library
• ML Metadata Store - DVC
• ML Pipeline Orchestrator - DVC & Make
• Experimentation Tracking - MLFlow

Mops Stack

GCP MLOPs

• MLOps
• Featurestore
• Model Monitoring

AWS MLOps

• Feature store
• Model Monitoring

Azure MLOps

• Feature store
• Model Monitoring
• Set up MLOps

MLFlow

MLflow - Tracking experiments, Packaging ML code, Managing and deploying models, central model store  

Made with ML

• Made With ML
• CI/CD for Machine Learning
• Model Serving
• Distributed training
• Monitoring Machine Learning Systems

MLOps Courses

• MLOps Zoomcamp
• MLOps-Basics
• MLOps Guide
• awesome-mlops
• dtu_mlops
• MLOps Course
• MLOps Tutorial #1: Intro to Continuous Integration for ML

Kubernetes

• Intro to Kubernetes Workshop
• Kubernetes slides
• Kubernetes The Hard Way
• From Notebook to Pipeline in No Time with LineaPy and DVC - Thomas Fraunholz at PyData 

Keep Exploring!!!

MlOps - Lets Learn it - Day #1

Many thanks to Stanford and MIT for sharing knowledge. In 2000 I used to download PPTs and refer to slides. Now you have a ton of materials plus a lot of distraction. Filtering knowledge vs chaos need time, focus and bookmarking.

Course materials are shared in Link

I wanted to review the first lecture/notes and bookmark my lessons

Notes - Link

Summary from it

Machine learning is an approach to (1) learn  (2) complex  (3) patterns from (4) existing data and use these patterns to make (5) predictions on (6) unseen data.

• Learn: DB has explicit relationships but ML learns relationships
• Complex: Across attributes ML finds relationships
• Patterns: Influences, categories, segments ML finds
• Existing data: Learn from data, improve on ongoing data collection. 
• Predictions: Use the learnt knowledge to apply for incoming data

Use cases distributions, Top 3 outside costs

• Customer insights
• Improve experience
• Retain Customers

Data Hierarchy - Data - OLTP - OLAP - ML - AI

Difference between Latency and throughput

Keep Exploring!!!

MLops Tools

MLOps tools link 

• CI/CD For Machine learning: ClearML, CML, Gitlab
• CronJob Monitoring: Cronitor, HealthchecksIO 
• Data Exploration: Apache Zeppelin, BambooLib, Google Colab, Jupyter Notebook, JupyterLab
• Data Management: DVC, Arrikto, BlazingSQL, Delta Lake, Dolt, DVC, Git LFS
• Data Processing: AirFlow, Hadoop
• Data Validation: Cerberus, Great Expectations
• Data Visualization: SuperSet, Tableau, Facet, Dash
• Feature Engineering: Featuretools, TSFresh
• Feature Store: Butterfree, ByteHub, Feast, Tecton
• Hyperparameter Tuning: Hyperas, Hyperopt, Kabit, KerasTuner, Optuna, Scikit Optimize, Optuna
• Machine Learning Platform: SageMaker, Kubeflow, H2O, MLReef, algorithmia, DataRobot, DAGsHub
• Model FairNess: AI 360, FairLearn, Opacus
• Model Interpretability: Alibi, Captum, ELI5, InterpretML, LIME, Lucid, SAGE, SHAP, Skater
• Model LifeCycle: MLflow, NeptuneAI, Comet, Keepsake, ModelDB, Weights and Biases
• Model Serving: BentoML, Tensorflow Serving, KFServing, SeldonCore, Streamlit, TorchServce, Gradio, Graphpipe, Hydrosphereout
• Model Testing and Validation: DeepChecks
• Optimization Tools: Dask, DeepSpeed, Horovod, Tpot, Ray Rapids
• Simplification Tools for ML: Pycaret, Hermione, Hydra, Koalas, TuriCreate(apple), TrainGenerator
• Visual Analysis and Debugging: Aporia, Evidently, Yellowricks, Netron, Fiddler, Manifold
• Workflow Tools: MLRun,Flyte, Metaflow, Ploomber, ZenML, Kedro

Ref - Link

Big Picture - Different phases of Model Development

Ref - Link

Overall Landscape - Monitor, Manage, Retrain, Tools Stack

MLOps vs Data Engineering

I always had a mixed opinion of different tasks in ML vs Data Engineering overlapping. This article I align to the views

MLOps is Mostly Data Engineering.

Data in different forms and the reporting aspects

• Transaction Data
• BI Reports
• ML Features
• ML Dashboards
• Everything operates on same data. 

Key Questions from article

• How different is the observability of model quality metrics like drift different to any product-related monitoring? 
• In product we keep monitoring the performance of our features, do people engage with them in the way we expect?

Cycle of Activities

Keep Exploring!!!

ML beyond Models - MLOps

Good Read, Helped me spot the gaps / Already things I know :)

Good Read - Reproducible Deep Learning

• Code Versioning (GIT)
• Data Versioning (DVC)
• Model Dockerization / Deployment

Codes / Examples

• Git Notes
• DVC Workflow
• Building Containers
• Reproducible Deep Learning

Most of this flow is applicable for any existing software dev process :)

Ref - Link

Keep Checking!!!

Pipelines - Pipelines

This concept of pipelines sometimes I feel the reality vs state of art is way too different

1. As of today %% of companies that have data consolidated for Building, models would be 5%, Rest all could be connect and extract data as needed
2. ML is not a separate skill, Data - OLTP, OLAP, Reporting, ML everything has to co-exist. 

The intent of the pipeline is to automate Model Building / Deployment. I have not seen direct training/deployment.

In Actual Implementation

• Training code will be separate
• Test data Location / Connectors to Pull data
• Trained models storage / Saving their metrics
• Deploying trained model as API

Still, we can achieve everything with the skills the team has across DB / ML, We don't need to have a dedicated ML pipeline. This post on DIY pipeline demonstrates the same DIY machine learning training pipeline

More Read

• Architecting a Machine Learning Pipeline
• An Introduction to Directed Acyclic Graphs (DAGs) for Data Scientists

Keep Exploring!!!