Vision Use case

How to Implement the Use Case Correctly

• Field of View
• Stable Infrastructure
• Minimal Occlusion
• No Manual Calibration
• With a good setup, half of the complexity and noise can be eliminated.

Keep Exploring!!!

AI Skills

 Keep Learning!!!

My Consulting Journey - AI - DL - GenAI Projects

As I wrap up my consulting tenure, I reflect on my success stories in the past 4 years. Here are some key projects that serve as my badges of success:

Bundle Recommendations Project #1 - Bundle recommendations for a specialty retailer of children’s apparel, from newborns to pre-teens (2020) Work/Impact - Transitioned from automated merchandiser-based recommendations to ML-based bundle recommendations. Achieved a 100% match with the ML approach. For a category level, we analyzed 6 months of transactions, comprising 1.5 million orders, and generated recommendations in 15 minutes.

Performance Optimization Project #2 (2021) - For a multinational mining company, optimized an existing app, more akin to a trading app, deployed between OLAP vs. OLTP. Applied a blend of DB/user and usage analysis/patterns/ML-based techniques to provide a list of recommendations to optimize.

GenAI + Vision Project #3 (2023-2024) - For a British multinational fast-moving consumer goods company, My key contribution is solution architecture based on Vision + GenAI for product detection and personalized recommendations, for its customers' products and brands.

Plants Classification Project #4 - Developing vision-based state-of-the-art classification models for the world's leading gardening charity. This work involved data curation, augmentation, and training, and ended as a paper :). Link

GenAI and CX improvement Project #5 - For a US-based leading specialty retailer of organizing solutions, custom spaces, and in-home services, leveraging GenAI + Vision to improve the customer journey. Pitched/deployed selected use cases. This is similar to what you see in Amazon/Swiggy GenAI Changes.

Forecasting Project #6 - Domain played a key role for me to contribute. For a leading South American beauty retailer, developing forecast models.

I had a mix of responsibilities as a Solution Architect, DB, and ML Engineer. I relied mostly on SA/DB/ML. In all projects, The team was a mix of platform, MLOps, and ML engineers. Sometimes the platform is a vendor cloud or an in-prem solution.

Hoping to undertake a few more similar projects in my next self-employed consulting roles.

If you are a startup, or SMB looking to apply AI/ML in your solutions, We can connect and collaborate on your AI Strategy. My coordinates [sivaram2k10][at][gmail]

Keep Exploring!!!

AI - Applied use case - Vision in Action

 

Spot the right use case, solve with the balance of data / strategy to meet the market on time

More read - Link

Keep Exploring!!!

Video Summarization

Learning to Summarize Videos by Contrasting Clips

1. Feature Extractor
2. Score Predictor
3. Summary Extractor
4. Highlight detection as a special case of the summarization task

Video Summarization: Towards Entity-Aware Captions - Summarizing video content into a natural language description

Video Summarization Using Deep Neural Networks: A Survey

Option #1

• Feature Extractor
• Score Predictor
• Summary Extractor
• Highlight detection as a special case of the summarization task

Option #2

• Frame 1 - Feature Vector
• Frame 2 - Feature Vector2
• Frame 3 - Feature Vector 3
• Feature vector score comparison to pick / unpick
• Object score comparison to pick / unpick

Other Techniques

• Hashing based
• Clustering based
• Feature based

Keep Exploring!!!

Can ML Solve this Problem ? Vision Problem - How to approach Damage Detection in Mobile Phones ?

How do you approach Damage Detection in Mobile Phones? 

Detecting defects on phones during exchange

Question - Can it be done with ML? 

• Student Answers - DL Vision

Question - Data Prerequisites?

Student Answers

• Physical damage to vision
• Images of the phone from various angles
• Software issues
• System diagnostics
• Images of cracked screens

Question - Model building

Student Answers

• Cnn classification 2 classes
• Damaged, not damaged
• Multiclass - damaged, degrees of damage (so that can identify price negotiation)
• inside parts, maybe images of phone when it is not damaged?

Real-world Way of Solving 

My Recommendation

• Detect Type of Phone, - Flip / Smart Phone
• Brand Detection (OCR)
• Image Similarity (Good Screen vs Similarity score to what you have)
• Line Detection - Count Cracks on Screen
• Segmentation to detect %% of cracked area
• Measure the deformation in the picture
• Yes / NO - Cracks
• Low / Medium / High
• Centre, Lower, Top

This is not a single model for all needs. This has to be based on brands, models, categories, Defect types, Data Collection, Labelling and Phased Adoption.

Keep Exploring!!!

Simplifying Neural Network Training Under Class Imbalance

Simplifying Neural Network Training Under Class Imbalance

• Small batch size - Class-imbalanced - settings, where small batch sizes shine.
• Data augmentations have an amplified impact on performance under class imbalance, especially on minority-class accuracy
• Adding a self-supervised loss during training can improve feature representations
• Label smoothing, especially on minority class examples, helps prevent overfitting. We adapt label smoothing for the class-imbalanced setting by applying more smoothing to minorityclass examples than to majority-class  examples
• A small modification of Sharpness-Aware Minimization (SAM) pulls decision boundaries away from minority samples and significantly improves minority-group accuracy
• Loss reweighting. Reweighting methods assign different weights to majority and minority class loss functions, increasing the influence of minority samples which would otherwise play little role in the loss function

Label smoothing is a technique often used in training deep learning models, particularly for classification tasks. It modifies the target labels, making them a blend of the original hard labels and some uniform or prior distribution. This can lead to better generalization by preventing the model from becoming too confident about its predictions. In a class-imbalanced setting, where some classes have significantly more examples than others, label smoothing can help by reducing the model's bias towards the more frequent classes.

Label smoothing for the class-imbalanced setting python example

In practice, label smoothing does not change the dataset's inherent imbalance but softens the target distributions by moving a portion of the mass from the peak (corresponding to the hard label) to other classes, which can help during the training of a model, preventing it from becoming overly confident on the majority class.

Loss reweighting for the class-imbalanced setting python example

class_weight: Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only).

Let's import the module first

from sklearn.utils import class_weight

In order to calculate the class weight do the following

class_weights = class_weight.compute_class_weight('balanced', np.unique(y_train), y_train)

Thirdly and lastly add it to the model fitting

model.fit(X_train, y_train, class_weight=class_weights)

Keep Exploring!!!

Learning Materials - DL - ML - DE - MLOps - NLP

Ready to Learn collated examples

• machine-learning-zoomcamp
• zoomcamp-analytics
• data-engineering-zoomcamp
• mlops-zoomcamp
• MLOps project with Loan Data
• Large Language Models Course
• Google-Machine-Learning-for-Solutions-Architects
• A collection of design patterns/idioms in Python
• How to add tests to your data pipelines
• Data Pipeline Design Patterns - #1. Data flow patterns
• Data Pipeline Design Patterns - #2. Coding patterns in Python
• energy-forecasting
• How to build and automate a python ETL pipeline with airflow on AWS EC2 

Good Data related papers - Link

Neatly organized here: Link

Good Data Engineering Papers

• BtrBlocks: Efficient Columnar Compression for Data Lakes
• The Story of AWS Glue
• Building a serverless Data Lakehouse from spare parts⋆
• The Snowflake Elastic Data Warehouse

Updated list of engineering papers worth reading.

1. Google File System - https://lnkd.in/d2-wnyqZ

2. Map Reduce Big Data Algorithm - https://lnkd.in/dvE8-s8M

3. BigTable NoSQL Document Store - https://lnkd.in/drmvvSAK

4. Colossus Next Gen File Store - https://lnkd.in/dERKhwMf

5. Megastore Large Object Store - https://lnkd.in/d5JDs2-K

6. Monarch Time Series DB - https://lnkd.in/d3kH_NCp

7. Chubby Distributed Lock Management - https://lnkd.in/dYy-w5rW

8. Spanner Distributed Database - https://lnkd.in/d6Emnycp

9. Spanner - CAP theorem considerations - https://lnkd.in/dq29BAWQ

10. Dapper Tracing System - https://lnkd.in/dm36-6jn

11. Borg Cluster Management - https://lnkd.in/dnveV-HU

12. Zanzibar Authentication System - https://lnkd.in/d5Vf7sRD

13. Pregel Graph Processing - https://lnkd.in/daq4576Y

14. Napa - Data Warehousing - https://lnkd.in/dbEfsa5B

15. Napa - Partitioning Algorithm - https://lnkd.in/dkhA7efJ

16. TensorFlow - Machine Learning at Scale - https://lnkd.in/d-4NfV2Z

17. Google F1 - Fast Analytics - https://lnkd.in/dbZqEKuf

18. HALP - YouTube Content Delivery Network - https://lnkd.in/dHzJtUc7

19. Mesa - Data Warehousing - https://lnkd.in/dFJ_Jrz6

20. Google Firestore - https://lnkd.in/drtEN9qR

21. Amazon Aurora DB Architecture - https://lnkd.in/dcevpwFt

22. Dynamo DB NoSQL Database - https://lnkd.in/dMD8C_WK

23. Apple Foundation DB - NewSQL database - https://lnkd.in/dG75i_9K

24. TikTok Monolith - Embedding in real-time - https://lnkd.in/dcjBXCnc

25. Scalability at what COST - https://lnkd.in/dJ9ScYKq

26. Gorilla - Time Series DB - https://lnkd.in/d3AeN2kB

27. Cassandra - NoSQL DB - https://lnkd.in/d-_nhtED

28. FlexiRaft - Distributed Consensus Tradeoffs - https://lnkd.in/dX3nMvmt

29. Memcache - In-memory Cache at Facebook - https://lnkd.in/dKeYK67g

30. Millisampler Network Sampling - https://lnkd.in/dsj9FuD6

31. TAO Graph Database - https://lnkd.in/daasJpYf

32. MineSweeper - Root Cause Analysis - https://lnkd.in/dEsd6iwj

33. Facebook Prophet - Forecasting at Scale - https://lnkd.in/daCmAjak

34. Facebook ShardManager - https://lnkd.in/dDy9Dp2h

35. Hive - Map Reduce Jobs - https://lnkd.in/dpV8BM2R

36. Apache Thrift - Definition Language - https://lnkd.in/d7NzhP54

37. Meta Twine - Cluster Management System - https://lnkd.in/d5t7VFKE

38. Meta ServiceRouter - Service mesh - https://lnkd.in/dVnkv_bV

39. Apache Hadoop - Distributed File System - https://lnkd.in/dHsQu9FN

40. Apache Kafka - Event Bus - https://lnkd.in/dyxuKbMb

41. Apache Flink - https://lnkd.in/dn_gMvaR

NLP

NLP-Python

Advanced_NLP_Spacy

Vision Notes

The U-Net (actually) explained in 10 minutes

U-Net Implementation With PyTorch

Efficient Python Tricks and Tools for Data Scientists

mathdatasimplified

It takes time to read - learn - experiment - build expertise. Learning Never Ends !!!!

Keep Exploring!!!

Machine Learning Interpretability / Explainability

Key Notes / Ideas 

Key items from blog / Reposted 

• Create White-Box / Interpretable Models (Intrinsic): e.g., Linear Regression, Decision Trees.
• Explain Black-Box / Complex Models (Post-Hoc): e.g., LIME, SHAP.
• Enhance the Fairness of a Model: e.g., Fairness Indicators, Adversarial Debiasing.
• Test Sensitivity of Predictions: e.g., Perturbation Analysis.

Local vs Global Interpretations:

• Local: Dive into a single prediction to understand it. e.g., Individual SHAP values.
• Global: Grasp the overall model behavior. e.g., Feature Importance Rankings.

Data Types & Applicable Interpretability Methods:

• Tabular: e.g., Partial Dependence Plots.
• Text: e.g., Word Embedding Visualizations.
• Image: e.g., Grad-CAM for CNNs.
• Graph: e.g., Node Influence Metrics.

Model Specificity:

• Model Specific: Techniques that apply to a single model or a group of models. e.g., Feature Importances for Trees.
• Model Agnostic: General methods applicable to any model. e.g., LIME.

Ref - Link

From AI Ethics institute key points Link

• Transparency and explainability gains may be significant
• Explainable by justification - Examples could get a better understanding 
• Explainability through feature importance - understanding of the effect of features - SHAP (SHapley Additive exPlanations
• Abstracting key patterns identified in the deep learning models as actual features
• Implications of different types of errors have, as well as what the right way of evaluating these errors should be.

Keep Exploring!!!

CNN Learning One pagers

Product and Example

• https://tangoeye.ai/
• Retail solutions built on
• Age Detection Models
• Gender Detection Models
• Face Detection
• Re-identification

Models for Training - tensorflow hub

How CNN works - Visualizer

How Features are Learned - 10 class classification

But what is a neural network

The Complete Mathematics of Neural Networks and Deep Learning

Learning Procedure

• Step 1: Take a batch of training data and perform forward propagation to compute the loss.
• Step 2: Backpropagate the loss to get the gradient of the loss with respect to each weight.
• Step 3: Use the gradients to update the weights of the network.

Backprop summary

• Chain rule derivate
• The procedure repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between actual output and desired output
• Ability to create new distinguishing features
• The aim is to find the set of weights that ensure that for each input vector the output vector produced by the network is same as the desired output vector
• The drawback in learning procedure is that the error surface may contain local minima so that gradient descent is not guaranteed to find a global minimum

Activation Functions

• Introduce non-linearity into a model
• We need non-linearity, to capture more complex features and model more complex variations that simple linear models can not capture.
• neural networks use non-linear activation functions, which can help the network learn complex data, compute and learn
• Signmoid, Tanh, Relu

Designing CNN

• The first rule of thumb is that you should not try to design your own architecture from scratch
• If you are working on generic problem, it never hurts to start with ResNet-50. If you are building a mobile-based visual application where there is limited computation resources, try MobileNets

Keep Learning!!!

Lightweight Deep Learning - Model Tuning - Model Compression

Key Techniques

• Pruning
• Weight Sharing
• Quantization
• Low-rank Approximation
• Sparse Regularization
• Distillation

Pruning Weights

• Motivated by how real brain learns
• Remove weights which 𝑤𝑒𝑖𝑔ℎ𝑡 < 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑
• Retrain after pruning weights
• Learn effective connections by iterative pruning
• Between L1 and L2, which regularization is better

Criteria for Pruning 

• Minimum weight - Pruning by the magnitude of Kernel weight (L2 norm)
• Smallest activation - Prune Kernels that lead to feature maps with the, smallest activations

Weight Sharing

• Compress the neural network with weight sharing
• Use a low-cost hash function to randomly group connection weights into hash buckets

Quantization

• Binarize all the weights and activations, turning DNN into Binarized Neural Network (BNN), in order to reduce memory consumption and increase power-efficiency.
• The method binarizes both weights and activations, in contrast to BinaryConnect which binarizes only weights

Low-rank Approximation

• SVD

Sparse Regularization

• Zero-out groups of weights using sparsity-inducing penalty

Distillation

• Based on teacher-student model
• Given trained deep neural network (DNN), called ‘teachers’, make a compressed ‘student’ model, with similar accuracy using quantization and distillation
• Teacher = Original Deep Model, Student = Quantized Model

Ref  - Link

Keep Exploring!!!

What a Life

Trying to find similar groups based on behavior - Apply Clustering
Predicting the intuition, this is my job connecting to it - Classification
With a new height of happiness/hope in life - Forecast more sleep hours
Look at people, decode their intentions - Read data/correlate and spot intentions
Finding my liked-minded folks with cosine, euclidean, and manhattan but still not getting relevant recommendations
On mistakes still learning to backpropagate and change/improve my optimistic learning rates
Everything is deep in life... the deeper you feel, the farther you go...
Have more attention to the present. Live more, Feel the Life.

It's a Deep Life :)

What does from_logits=True do in SparseCategoricalcrossEntropy loss function?

•  The from_logits=True attribute inform the loss function that the output values generated by the model are not normalized
• In other words, the softmax function has not been applied on them to produce a probability distribution
• Basically we need to softmax and pick max value or the max value from list is the prediction

Keep Learning!!!

Deep Learning Revisions

It's always good to take a pause/revise / add a few more learning pointers :)

Key Notes

• ML operates by handcrafted features
• DL features learned directly from data
• Data prevalent, Parallelizable models / hardware, GPU/ CUDA, TF / Pytorch
• Activation functions and their differentiation

• Non-Linear functions help to build boundaries

• Text - sequence of characters / words
• Stock prices / DNA sequences
• Temporal dimension to models
• Same series once for each Timestep
• Horizontal to vertical view
• Each output is connected/is input to the next timestamp
• Internal memory / state-maintained

• Individual Loss for each timestep
• Backprop for all timestamps
• Forward pass across time

• Back propagate through time
• Loss with respect to the internal state
• Attention

Ref - Course Link

Keep Thinking!!!

Loss Functions - Deep Learning

The choice of loss depends on the desired output (e.g., classification vs. regression)

Regression Loss Functions

• Mean Squared Error Loss
• L1 Loss
• L2 Loss
• Mean Squared Logarithmic Error Loss
• Mean Absolute Error Loss

L2 Norm, mean squared error. Mean Squared Error - The mean square error is probably straight forward. You take the difference of the result and the ground truth for this sample and square it.

The L1 loss is basically the Absolut value of the difference between the current sample’s actual output and the desired output.

Binary Classification Loss Functions

• Binary Cross-Entropy
• Hinge Loss
• Squared Hinge Loss

Multi-class Classification Loss Functions

• Multi-class Cross Entropy Loss
• Sparse Multiclass Cross-Entropy Loss
• Kullback Leibler Divergence Loss

The Negative log-likelihood loss is based on the idea that every output represents a likelihood for example a particular class. It aims to make the output for the correct class has high as possible and for others as small as possible.

Cross entropy loss - The cross entropy loss is very popular for classification problems. The losses are averaged across observations for each minibatch

Kullback-Leibler Divergence Loss - Measures distance between distributions

Ref - Link1, Link2

Keep Thinking!!!

Zero Shot Learning

Zero Shot Learning

My Summary 

• Extract Attributes from Data (Images - Edges, Corners, Contours, Fetaure vectors)
• Textual embedding or feature vector
• Using this Classify known or unknown Class

Zero-Shot Learning - Feature / Attribute extraction and prediction based on those features of known class and heard features of unknown class

Feedback - Good concept, For all these cases we need reasonable data to extract, build features, and discriminative features.

Some conceptual notes/papers

• CNN learning algorithm to learn to detect the features of the word-embeddings like stripes, animalness, and whiteness in images as well.
• Replace the label of the image with its word-embedding during training.
• Pre-trained word-embeddings can be downloaded and used with the object recognition CNN model.

ZSL

• Zero-shot methods basically work by combining the observed/seen and non-observed/unseen categories
• There are two common approaches used to solve the zero-shot recognition problems.
• Embedding based approach
• Generative model-based approach
• Zero-shot classification model is trained on both seen and non-observed category images at train time

From classification - Set of X, Not belongs to X, Belongs to set ox X class vs Not belongs to X set

Zero Shot Learning 

• Zero-shot classification refers to the problem setting where we want to recognize objects from classes that our model has not seen during training
• Seen classes: These are classes for which we have labelled images during training
• Unseen classes: These are classes for which labelled images are not present during the training phase.
• Auxiliary information: This information consists of descriptions/semantic attributes/word embeddings

If I had to sum up ZSL in a few words, I’d say that it is:

• Pattern recognition without training examples
• Based on semantic transfer

Representation Learning

Zero-shot learning approach

• Training is the process of capturing knowledge about the qualities.
• Inference where the information is utilized to classify examples into a new set of classes.

Rethinking Zero-Shot Learning: A Conditional Visual Classification Perspective

• Classifying visual features based on the classifiers learned from the semantic descriptions
• Highly discriminative classifiers for seen classes and the generated classifiers for unseen classes to classify visual features of all classes

Zero-shot Learning with Deep Neural Networks for Object Recognition∗

• Zero-Shot Learning with Semantic Output Codes
• Zero-shot Learning with Deep Neural Networks for Object Recognition∗

Keep Exploring!!!

CNN Learning Tools - Deep Learning Tools

• Interactive Visualization - Link
• CNN Interactive Explorer - Link
• Tensorflow Playground - Link
• Embedding Projector - Link
• Sketch RNN Demo - Link , Link1

Keep Exploring!!!