Transfer Learning Notes

Teach the process/approach for students to get better clarity :). My answers for below question 

 When does Transfer Learning work?

Keep Exploring!!!

Dense to Sparse - AI World

• What we do in CNN - Convert Dense to Sparse with convolution and activations 
• What we do in NLP - Text Preprocessing: Stemming / Lemmatization / Stop-word removal - Vectorization 
• Topic Modelling - Words - Documents - Non-Negative Matrix Factorization 
• ML Feature Engineering / Recommendations - PCA / SVD 

Everywhere we attempt to retain key features/vectors aligned to vision/text/features/topic modeling tasks. Converting Dense to Sparse is the way to get the signal from the noise :)

Keep Exploring!!!

CNN Experiments - Solutions - Building End to End Solutions

CNN Experiments - Solutions - Building End-to-End Solutions

CNN Experiments

• Minimum Exp Without Aug
• Data Aug + CNN Model 
• Data Aug + CNN Model (Deeper Layers) - Few more convolution blocks
• Data Aug + CNN Model (Deeper Layers) - Few more convolution blocks + (Dropouts / Regularizer / Adjusting Learning rate)

To Launch a Product / Build Model things to consider

• Pre-requisites
• Data Collection
• Data Pre-processing and transformation
• Data Imbalances / Data Augmentation 
• Modelling
• Deployment
• Monitoring
• Real-time data training
• Collaborate with Healthcare prof
• Keep updating the model

We have 95% Accuracy, Remaining 5% how do we handle

• Similarity scores
• Ensemble methods
• Human in loop

Keep Exploring!!!

CNN Learning One pagers

Product and Example

• https://tangoeye.ai/
• Retail solutions built on
• Age Detection Models
• Gender Detection Models
• Face Detection
• Re-identification

Models for Training - tensorflow hub

How CNN works - Visualizer

How Features are Learned - 10 class classification

But what is a neural network

The Complete Mathematics of Neural Networks and Deep Learning

Learning Procedure

• Step 1: Take a batch of training data and perform forward propagation to compute the loss.
• Step 2: Backpropagate the loss to get the gradient of the loss with respect to each weight.
• Step 3: Use the gradients to update the weights of the network.

Backprop summary

• Chain rule derivate
• The procedure repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between actual output and desired output
• Ability to create new distinguishing features
• The aim is to find the set of weights that ensure that for each input vector the output vector produced by the network is same as the desired output vector
• The drawback in learning procedure is that the error surface may contain local minima so that gradient descent is not guaranteed to find a global minimum

Activation Functions

• Introduce non-linearity into a model
• We need non-linearity, to capture more complex features and model more complex variations that simple linear models can not capture.
• neural networks use non-linear activation functions, which can help the network learn complex data, compute and learn
• Signmoid, Tanh, Relu

Designing CNN

• The first rule of thumb is that you should not try to design your own architecture from scratch
• If you are working on generic problem, it never hurts to start with ResNet-50. If you are building a mobile-based visual application where there is limited computation resources, try MobileNets

Keep Learning!!!

What does it take to build a good enough CNN model ?

This question came up during class. There is no direct answer. It depends on domain / data / use case / number of classes. Guidelines of recommendations are

• Number of Layers of convolution, Experiment with VGG16 / 19 to get started
• Balacing imbalanced datasets
• Depth of Layers - 32/ 64/ 256 as needed
• Relu, Experiment, Customize on activation functions
• Adjusting learning rates / Loss functions
• Early Stopping / Dropouts for regularization
• Domain Relevant Augmentation
• Transfer Learning Approach

Keep Thinking!!!

CNN Learning Tools - Deep Learning Tools

• Interactive Visualization - Link
• CNN Interactive Explorer - Link
• Tensorflow Playground - Link
• Embedding Projector - Link
• Sketch RNN Demo - Link , Link1

Keep Exploring!!!