Day #98 - Advanced Hyperparameter tuning

Neural Network Libraries

• Keras (Easy to learn)
• Tensorflow (For production this is used)
• MxNet
• PyTorch (Popular in community)
• sklearn's MLP

Neural Nets

• Number of neurons per layer
• Number of layers
• Optimizers
• SGD + momentum
• Adam / Adadelta / Adagrad (In practice lead to more overfitting)
• Batch size (Huge batch size leads to overfitting)
• Epochs impact
• Learning rate - not too high not too low, Rate where network converges
• Regularization
• L2/L1 for weights
• Dropout / Dropconnect
• Static dropconnect

Linear Models (Scikit-learn)

• SVC / SVR
• Sklearn wraps libLinear and libSVM
• Compile yourself for multicore support
• LogisticRegression / LinearRegression + regularizers
• SGDClassifier / SGDRegressor
• Vowpal Rabbit
• Regularization parameter (C, alpha, lambda)
• Start with very small value and increase it
• SVC starts to work slower as C increases
• Regularization type
• L1/L2/L1+L2 - try each
• L1 can be used for feature selection

Happy Learning!!!