Day #177 -Phd Dissertion - Deep Learning of Representations and its Applications to Computer Vision

Key Lessons

• New Feature Learning algorithm with minimal data, RBM Encoding
• New Activation function , Maxout Activation functions looks at two different values and take max of both values (Core)
• Accuracy Vs Depth of Layers in CNN (Depth does help in learning more features and improves accuracy)

Key Lessons

• Probablistic Models
• Deep Botlzman Links
• Maxout - Dropout
• Street Number Transcription

MLE - Describes probability of event happening. Depends on paramaters. MLE Pick parameters that maximize probability.
Gradient Descent - Kind of Cost function to minimize, Looking at derivatives we minimize the function, As long as derivatives +ves me move in opposite direction until it becomes zero
Supervised Learning - Data is features X, Targets y. Learn to map X to Y. Classification - Discrete Y. Regression Continuous Y
Unsupervised Learning for Feature Learning - SVM, Wide Street Margin Approach
Deep Learning - Learn Representations, Visualize features in Convolutional Network
RBM - Generative stochastic (random process) artificial neural network that can learn a probability distribution over its set of inputs.

Spike and Slab Sparse Model

• New Feature Learning algorithm with minimal data, RBM Encoding
• Variational Inference - Lagrange Equations to solve. Require both Analytical and Iterative Optimization
• Spike variables, Slab Variable
• They are paired and multiplied together
• Decomposing Image into set of edges 
• Spike (Edge present or not)
• Edge (Strong Edge or not)
• Fixed Pooling Pattern used to improve the model accuracy
• Worked for few examples

Multi-Prediction Deep Boltzmann machines

• Better way to train DBM
• DBM - Have feedback connections

DBM Training Stages

• Greedy layerwise pretraining - Train RBM Single Layer, Then train other BM, Glue into probablistic model
• Joint Generative training - After taking weights from model, create new classifier
• Discriminative fine tuning
• One unified probabilistic model

Multi-Prediction Training

• Randomly sample different inference problems
• Backprop through the mean field inference graph (Improve log likelihood of estimation variables)

Maxout Activation

• Introduce new activation function
• Input (V) * Weights (W) = Z
• Z fed to activation function
• Activation function element-wise
• Sigmoid activation function (commonly used)
• negative 0, +ve 1
• Relu, Doesn't saturate
• Activation functions looks at two different values and take max of both values (Core)

Street Number Transcription

• Sequences of numbers coming in
• CNN based, Atmost 5 digits (Assumption)
• Log-likelihood for sequence as a whole
• Depth in CNN
• Accuracy Vs Depth of Layers in CNN

Next Lists
CONNECTING IMAGES AND NATURAL LANGUAGE - Andre karpathy
Machine learning in plant breeding - Alencar Xavier's PhD defense
Predicting Poverty with Deep Learning, Neal Jean, PhD Student, Stanford at AAI16
Joachim Dehais - Phd Defense - Computer Vision for Diet Assessment
Evolutionary Design of Deep Neural Networks

I hope to do Phd sometime in near future. I can learn from few dissertions to understand things that happen in AL / ML :). Hoping to publish a quality paper on Deep Learning lessons. Learning is endless, Hope these presentations help in more ideas, perspectives to solve ML problems in my Work.

Happy Learning!!!