Vector - Matrix - Tensors

Ref - Link

Array is analogous to Tensors

Keep Exploring!!!

Backpropagation Notes - Forward propagation, Backward Propagation, Optimizers Notes

Backpropagation  - The amount of error in the neurons in the output layer is propagated back to the preceeding layers

Optimization algorithms are used to find the optimum parameters/variables of the NNs

• SGD is an algorithm that randomly selects a few samples instead of the whole data
• AdaGrad is a modified SGD that improves convergence performance over standard SGD algorithm
• RMSProp is an optimization algorithm that provides the adaptation of the learning rate for each of the parameters.
• ADAM combines advantages of the RMSProp (works well in online and non-stationary settings) and AdaGrad (works well with sparse gradients)

RNN

• With the BPTT learning method, the error change at any t time is reflected in the input and weights of the previous t times
• The difficulty of training RNN is due to the fact that the RNN structure has a backward dependence over time.

Hyperparameters - The number of hidden layers, the number of units in each layer, regularization techniques, network weight initialization, activation functions, learning rate, momentum values, number of epochs, batch size (minibatch size), decay rate, optimization algorithms

Ref - Link

Keep Exploring!!!

#Life as a #DeepNetwork

At different stages we need to balance #weights #education, #opportunities, #focus and #consistency

Different outputs we need are #Money, #Health, #Family, #Relationship, Security

Similar to #backprop as long as keep adjusting the weights we can get optimal output

Deeper the layers and more focus, Higher the success :)

Keep Exploring!!!

Backpropagation - Different interesting perspectives

After class, students summary of backpropagation concept :)

Perspective #1 - Back Propagation is tuning the weights of a neural network based on the error rate obtained in the previous iteration

Perspective #2 - It is a process of updating the weights & bias at each layer to minimize the error rate

Perspective #3 - Forward propagation is moving forward step by step, backward propagation is adjusting the sails to move ones defined direction...

Perspective #4 - 1. Calculate the output by forwardprop, 2. Calculate the error, 3. Minimize the error by backprop, 4. Update parameter, 5. Repeat till converge

Perspective #5 - Backpropagation:method or algorithm to find the optimal value of weight and bias to minimise the loss function

Perspective #6 - we feed cumulative input to the neuron and apply activation func. compare the output to actual output and update weight and bias. repeat the cycle until correct output

Perspective #7 - basically to reduce the loss, we change the weights using forward and backward feeds

Keep Thinking!!!

Backpropagation

Big thanks to Matt for his post on backpropagation. A big thanks to Upgrad for the teaching opportunity. Many times we need time to connect the dots. From hectic days of model building vs learning basics and teaching is a good opportunity to balance perspectives. 

I was able to work out the example and share it with my students.

Keep Exploring!!!