Pytube error - RegexMatchError: get_throttling_function_name: could not find match for multiple
Earlier roles / Feeds Processing
Had some critical tasks getting supplier feeds/data
DB supporting it
Technically from a scale point of view
Validations
Key design tweaks
ML Context
10 different models
Execute Job
Predict Job
Design Ideas
Python Options
Fast API - Uvicorn also has an option to start and run several worker processes. Link
uvicorn main:app --host 0.0.0.0 --port 8080 --workers 4
Flask API - Link
if __name__ == '__main__':
app.run(threaded=True)
flask run --with-threads
app.run(threaded=True)
More Reads
System Design — Design a distributed job scheduler (Keep It Simple Stupid Interview series)
Orchestrating a Background Job Workflow in Celery for Python
System Design: Designing a distributed Job Scheduler | Many interesting concepts to learn
Examples
Keep Exploring!!!
Many times I read up basics again and again, Over the years, I started with Windows98 Testing, C/C++ Adapters, Nestle production support, Application support, Supply chain QA / Performance / OLTP Development, SQL Developer, BI Developer, Setting up Teams, Warranty, Refurbishment, API / Supply chain, Website A/B testing, On call support. Retail product team setup/forecasting/scaling and then a long 2-year learning curve / paid lectures / back to basics mode. More learning started after that. Getting a break needs a lot of freelance / consulting/training / applied learning. Past 3 years very focused on learning/projects/production deployments.
Now when I teach the flow/work, there are different areas overall to understand products/domain/use cases
Sometimes I wonder how many things we can teach someone to switch to AI / ML. Always leverage your strengths in domain/data knowledge. It is vast and increasing day by day the scope of it. To succeed it is hard to know everything but the end goal is to add value to the business / use it to fix current challenges. Balance both learning and implementation. It will be a long journey to just learn forever.
Always blend your ideas in DATA + DOMAIN + AI + Business Value to find the right use cases.
Keep Exploring!!!
Let's list 10 reasons why GenAI will succeed
Let's list 10 reasons why GenAI will Fail
In Azure Portal
Fetch Endpoint and Keys after deployment
Example code in link
The different types of prompts - zero shot with limited prompts, reasoning preserve states, simplified step by step prompts
A prompt template can contain:
A few shot prompt template can be constructed from either a set of examples, or from an Example Selector object.
Few shot examples for chat models
Ref - Link
Keep Exploring!!!
We have AWS Lambda, GCP Cloud run servless function options. This will help effectively to autoscale.
For custom apps / rest / flask / fast end points how to we auto scale
Ref - Link
HorizontalPodAutoscaler Walkthrough
Key Notes
kubectl autoscale subcommand, part of kubectl, that helps you do this.
kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
# You can use "hpa" or "horizontalpodautoscaler"; either name works OK.
kubectl get hpa
How to Test Autoscaling in Kubernetes
Keep Exploring!!!
Warping -
Warping involves transforming an image's geometry, usually to correct distortions, align images, or change the perspective.
There are different types of warping, such as:
In the context of computer vision libraries like OpenCV, warping functions are available to apply these transformations to images, given the appropriate transformation matrix and input/output coordinates.
OpenCV Warping functions
OpenCV Blending functions
Paper - link
Things to note
Keep Exploring!!!
Kullback-Leibler (KL) divergence, also known as relative entropy. It is a measure of how one probability distribution is different from a second, reference probability distribution.
chi-square test is a statistical test used to determine whether there is a significant association between two categorical variables in a sample.
Keep Exploring!!!
Expertise = Experience = Domain + Data + Tech (Blend of all)
Easier said, I can echo it to my past journey/projects. Solving for things the customer needs vs I know a tech what all I can do.
Ref - Link
Keep Exploring!!!
Reduction in Cart Abandonment
Key Notes
Ref - Link
Solutions Architect
Key Notes
Ref - Link
MLOps
Key Projects
Ref - Link
System Design for Recommendations and Search
Key Notes
Key Concepts
Ref - link
My implementation experience and lessons :)
Product Implementation (2012-2014)
Batched State of Art (2021)
Recommendations AWS
Forecasting State of Art (2021)
Kubeflow + AWS
Realtime State of Art (2022)
Real-time streaming / Vision Solution
Keep Exploring!!!
Milvus Notes - Index/consistency / availability options
#1. Index type - usecase
Inverted File (IVF): An IVF index divides the vector space into several clusters and holds an inverted file for each cluster, recording which vectors belong to the cluster.
IVF Flat: This is a combination of IVF and flat index. It uses the IVF index to partition the data into clusters and then uses the flat index (brute-force search) within each cluster.
Hierarchical Navigable Small World (HNSW): HNSW builds a multi-layer navigation graph to represent the vector space.
#2. Consistency levels - Strong, Bounded, Session or Eventually
#3. HA - In-memory replicas help Milvus recover faster if a query node crashes.
#4. Vector search & Hybrid Search params offset, limit
offset - Number of results to skip in the returned set
limit - Number of the most similar results to return
How indexing and querying works
Keep Exploring!!!
Every problem statement needs to have
1. Define products/layouts
Notes
Stack
MosaicML for training
Future
Keep Exploring!!!
DBSCAN (Density-Based Spatial Clustering of Applications with Noise) and K-means are two popular clustering algorithms used for unsupervised learning tasks. They have different approaches to clustering and are suitable for different types of data. Here's a comparison of the two algorithms:
Approach:
DBSCAN: DBSCAN is a density-based clustering algorithm. It groups together points that are closely packed together based on a distance measure (e.g., Euclidean distance) and a density threshold. It can find clusters of arbitrary shapes and is also able to identify noise points that do not belong to any cluster.
K-means: K-means is a centroid-based clustering algorithm. It partitions the data into K clusters by minimizing the sum of squared distances between the data points and their corresponding cluster centroids. K-means assumes that clusters are spherical and have similar sizes.
Number of clusters:
DBSCAN: The number of clusters is determined automatically by the algorithm based on the input parameters (distance threshold and minimum number of points). You don't need to specify the number of clusters beforehand.
K-means: You need to specify the number of clusters (K) beforehand. Choosing the optimal value of K can be challenging and often requires domain knowledge or using techniques like the elbow method or silhouette analysis.
Cluster shapes:
DBSCAN: DBSCAN can find clusters of arbitrary shapes, making it suitable for datasets with complex structures.
K-means: K-means assumes that clusters are spherical and have similar sizes, which may not be suitable for datasets with complex structures or clusters with different shapes and sizes.
Handling noise:
DBSCAN: DBSCAN can identify and separate noise points that do not belong to any cluster.
K-means: K-means is sensitive to noise and outliers, as they can significantly affect the position of the cluster centroids.
Scalability:
DBSCAN: DBSCAN can be slower than K-means for large datasets, especially if the distance matrix needs to be computed. However, there are optimized versions of DBSCAN (e.g., HDBSCAN) that can handle large datasets more efficiently.
K-means: K-means is generally faster and more scalable than DBSCAN, especially when using optimized implementations (e.g., MiniBatchKMeans in scikit-learn).
In summary, DBSCAN is more suitable for datasets with complex structures, arbitrary cluster shapes, and noise, while K-means is faster and more scalable but assumes spherical clusters with similar sizes. The choice between DBSCAN and K-means depends on the characteristics of the data and the specific requirements of the clustering task.
Keep Exploring!!!
Keep Thinking!!!
Paper - Empowering cash managers to achieve cost savings by improving predictive accuracy
Time Series Forecasting with Transformer Models and Application to Asset Management
Self-attention is designed to capture the dependencies in the sequence, such as the relationship between each word with each other word in a senten
For a given query, we compare it with all keys K and get different weights for different values
Self-attention and multi-head attention are permutation-equivariant with respect to its inputs
In our experiment, we consider three different portfolio allocation methods:
How to Build a Cash Flow Forecast
Cash payments and receipts. - Short-period forecasts: Short-term forecasts typically look two to four weeks into the future and contain a daily breakdown of cash payments and receipts.
The most common medium-term forecast is the rolling 13-week cash flow forecast.
Long-period forecasts: Longer-term forecasts typically look 6–12 months into the future and are often the starting point for annual budgeting processes
Mixed-period forecasts: Mixed-period forecasts use a mix of the three periods above and are commonly used for liquidity risk management.
Preparing a cash flow forecast: Simple steps for vital insight
Empirical analysis of daily cash flow time series and its implications for forecasting
Cash management is concerned with the efficient use of a company’s cash and short-term investments such as marketable securities.
From these and other works, we observe that common assumptions on the statistical properties of cash flow time-series include:
Empowering cash managers to achieve cost savings by improving predictive accuracy
Kurtosis is a measure of the tailedness of a distribution. Tailedness is how often outliers occur
Transforming Financial Forecasting with Data Science and Machine Learning at Uber
Modeling strategic investments as an optimization problem
With each optimization problem, we can also specify constraints, such as:
Short-term use cases: Short-term use cases for cashflow forecasting include budgeting, forecasting sales, and managing cash flow. It can also be used to identify potential areas of overspending and to plan for future investments
Long-term use cases: Cashflow forecasting can be used to plan for long-term investments, such as capital expenditures and acquisitions. It can also be used to develop strategies for managing cash flow over the long-term, such as budgeting and debt management
Under the hood training from tons of images we are generating distributions based on
Keep Exploring!!!
