Pipelines - Pipelines

This concept of pipelines sometimes I feel the reality vs state of art is way too different

1. As of today %% of companies that have data consolidated for Building, models would be 5%, Rest all could be connect and extract data as needed
2. ML is not a separate skill, Data - OLTP, OLAP, Reporting, ML everything has to co-exist. 

The intent of the pipeline is to automate Model Building / Deployment. I have not seen direct training/deployment.

In Actual Implementation

• Training code will be separate
• Test data Location / Connectors to Pull data
• Trained models storage / Saving their metrics
• Deploying trained model as API

Still, we can achieve everything with the skills the team has across DB / ML, We don't need to have a dedicated ML pipeline. This post on DIY pipeline demonstrates the same DIY machine learning training pipeline

More Read

• Architecting a Machine Learning Pipeline
• An Introduction to Directed Acyclic Graphs (DAGs) for Data Scientists

Keep Exploring!!!