Transfer Learning Notes

Teach the process/approach for students to get better clarity :). My answers for below question 

 When does Transfer Learning work?

Keep Exploring!!!

2023 - Lessons - AI - GenAI

After every year learning extends Data, AI, Products, and Domain. 2023 had a blend of experiences. Still figuring out answers for every dimension #2023 #Learnings

→ How you've adapted to industry shifts, and GenAI's meaningful adoption. Possible use cases vs relevant, meaningful production-ready use cases. Example - Newly launched section in Amazon reviews, What customers say. 

→ How you've overcome engineering challenges balancing business goals. New ways to solve old problems with Foundation models. Time vs building a production-grade solution. Example - Moving away from custom NER vs Leveraging LLM Embeddings, Blend of both custom embedding + RAG, New ways of solving.

→ How your skills align with the company's vision, Learning to predict the future. New approaches and papers evolve faster than certifications. A blend of tech + and domain is key. Segment Anything model, Visual QnA, Intructpix2pix have made more vision use cases feasible Tryon, etc..

→ How you bridge the gap between tech and business, Fast yet impactful use cases, Get the basics right. Demos / New offerings vs making it to production need a careful selection of use cases / applying past experiences to get things right in the first iteration. Balance the tradeoff between creativity vs innovation vs build a product strategy vs solve a real need vs fancy demos. #learning #perspectives #solutions #datascience #MachineLearning #AI #DeepLearning

Keep Exploring!!!

Data Science Interview Questions

For hiring 2-5yrs exp, Some basic and intermediate questions

1.  What experience do you have in developing deep-learning models for computer vision applications?
2. What challenges have you encountered while working with deep learning models?
3. How have you optimized deep learning models for speed and accuracy?
4. What techniques have you used to improve the accuracy of deep learning models?
5. What experience do you have in deploying deep learning models in production?
6. How have you evaluated the performance of deep learning models?
7. How have you handled data imbalance in deep learning models?
8. What experience do you have in developing transfer learning models?
9. How have you used reinforcement learning in deep learning models?
10. What experience do you have in developing generative models?
11. How would you design a deep-learning model to detect objects in a video stream?
12. How would you design a deep-learning model to classify images?
13. How would you design a deep learning model to detect anomalies in time-series data?
14. How would you design a deep-learning model to generate text?
15. How would you design a deep-learning model to identify faces in an image?
16. How would you design a deep-learning model to detect fraud in financial transactions?
17. How would you design a deep-learning model to predict stock prices?
18. How would you design a deep learning model to detect objects in a 3D environment?
19. How would you design a deep learning model to detect anomalies in a network?
20. How would you design a deep-learning model to generate music?

Keep Exploring!!!

Model Optimization / Performance Key Notes

• Knowledge distillation is a method in which a small model (student) is trained to mimic a larger model or ensemble of models (teacher). 
• DistilBERT, reduces the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.
• Pruning was a method originally used for decision trees where you remove sections of a tree that are uncritical and redundant for classification
• The more common meaning is to find parameters least useful to predictions and set them to 0
• Quantization reduces a model’s size by using fewer bits to represent its parameters. 
• By default, most software packages use 32 bits to represent a float number (single precision floating point). If a model has 100M parameters, each requires 32 bits to store, it’ll take up 400MB. If we use 16 bits to represent a number, we’ll reduce the memory footprint by half. Using 16 bits to represent a float is called half precision.

Knowledge Distillation

Pruning in Keras example

Ref - Link 

Ref2 - Link 

Large Transformer Model Inference Optimization

• Reduce the memory footprint of the model by using fewer GPU devices and less GPU memory;
• Reduce the desired computation complexity by lowering the number of FLOPs needed;
• Reduce the inference latency and make things run faster.
• Post-Training Quantization (PTQ): A model is first trained to convergence and then we convert its weights to lower precision without more training
• Unstructured pruning is allowed to drop any weight or connection, so it does not retain the original network architecture.
• Structured pruning aims to maintain the dense matrix multiplication form where some elements are zeros

Keep Exploring!!!

Model Training Checklist

Often I end up running a working code but poor segmentation results. Need to save cost / try low res to slowly make changes.

Making a working model on a limited set is key.

• Run for limited set 1000 images
• Run for smaller resolution 128x128
• Run for smaller batches
• Model Compilation
• Batch Compilation
• Adjust learning rates

Keep Exploring!!!

Deep Learning Markets - Use cases

 Combination of Vision, Forecast, Recommendation, Anamoly Detection, Optimization

Ref - Link

Keep Exploring!!!

Infra Costs - Training Large Datasets - Deep Learning

Infra and Costs - Link

Insights - Link

• Infra - GTX 1080 TI GPUs and cuDNN
• Dataset - 220,000 carefully annotated hair images

Infra Providers - Cirrascale, Lambda

Training large models - Link

• 4 days to train GPT-3 on 1,024x NVIDIA A100 GPUs.
• With each A100 GPU priced at $9,900, we’re talking almost $10,000,000 to setup a cluster that large
• you can rent A100 GPUs from public cloud providers like Google Cloud, but at $2.933908 per hour, that still adds up to $2,451,526.58 to run 1,024 A100 GPUs for 34 days
• Each TITAN X, for example, costs roughly $3,000

Keep Exploring!!!

Deep Networks - Tuning Tips - Segmentation - Observations

1. Try to overfit your network on much smaller data and for many epochs without augmenting first (Link)

2. Explore variations of the network (Link)

3. Change layers number and units number.

4. Change batch size.

5. Add dropout layer. (Link)

6. Verify that your code is bug free There's a saying among writers that "All writing is re-writing" -- that is, the greater part of writing is revising. 

7. Keep a Logbook of Experiments

8. Look at individual layers, Build a simpler model first (Link)

9. Use nn.BCEWithLogitsLoss as the criterion instead of nn.BCELoss and pass the raw logits to it by removing the sigmoid. (Link)

10. In colab plus GPU + HighRAM worked better

Even after 100+Epochs doesn't seem promising.

Github code is a good start but needs to relearn, Start again :)

Keep Learning!!!