"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;

January 25, 2018

Day #98 - Advanced Hyperparameter tuning

Neural Network Libraries
  • Keras (Easy to learn)
  • Tensorflow (For production this is used)
  • MxNet
  • PyTorch (Popular in community)
  • sklearn's MLP
Neural Nets
  • Number of neurons per layer
  • Number of layers
  • Optimizers
  • SGD + momentum
  • Adam / Adadelta / Adagrad (In practice lead to more overfitting)
  • Batch size (Huge batch size leads to overfitting)
  • Epochs impact
  • Learning rate - not too high not too low, Rate where network converges
  • Regularization
    • L2/L1 for weights
    • Dropout / Dropconnect
    • Static dropconnect
Linear Models (Scikit-learn)
  • SVC / SVR
  • Sklearn wraps libLinear and libSVM
  • Compile yourself for multicore support
  • LogisticRegression / LinearRegression + regularizers
  • SGDClassifier / SGDRegressor
  • Vowpal Rabbit
  • Regularization parameter (C, alpha, lambda)
  • Start with very small value and increase it
  • SVC starts to work slower as C increases
  • Regularization type
    • L1/L2/L1+L2 - try each
    • L1 can be used for feature selection
Happy Learning!!!
Post a Comment