Day #89 - Capsule networks

Key lessons

• Instead of adding layers it nests layers inside it
• We apply non-linearity to grouped neuros (capsule)
• Dynamic routing - Replace scalar output feature detector of CNN by routing by agreement based on output

CNN History

• Latest paper on capsule networks
• Offers state of art performance for MNIST dataset
• Convolutional networks - Learn mapping for input data and output label
• Convolution layer - Series of matrix multiplication and summation operation, Output feature map (bunch of learned features from image)
• RELU - Apply non-linearity to it (Network can learn both linear and non-linear functions). Solves vanishing gradient problem. (As gradeient is backpropagating its getting smaller and smaller, RELU prevents it)
• Pooling - Creates sections and take maximum pixel value from each sections
• Each line of code corresponds to layers in networks
• Dropout - Neurons randomly turned on to prevent overfits (Regularization technique)
• For handling rotations - AlexNet added different rotations to generalize to different rotations
• Deeper networks improved classification accuracy
• VGGnet adding more layers
• Googlenet - Convolution with different sizes processed on same input, Several of those together
• Resnet - Instead of stacking layers, Add operation improved vanishing gradient problem

Convolutional Network Challenges

• As we go up the hierarchy each of features learnt will be more complex
• Hierarchy happening with each layers
• Sub-sampling loses spatial relationships
• Spatial correlations are missed in sub-sampling and pooling
• Bad for rotated images (Invariance issues)

Capsule Networks

• Basic idea - Human brain attains transnational invariance in a better way, Instead of adding layers it nests layers inside it
• Nested layer is called capsule, group of neurons
• CNN route by pooling
• Deeper in terms of nesting

Layer based squashing

• Based on output neuron we apply non-linearity
• We apply non-linearity to grouped neuros (capsure)

Dynamic routing

• Replace scalar output by routing by agreement
• Hierarchy tree of nested layers

Key difference - All iterations to compute output, For every capsule nested apply operations
Happy coding and learning!!!