Day #100 - Ensemble Methods

It took more than a year to reach 100 posts. This is a significant milestone. Hoping to reach 200 soon.

• Combining different machine learning models  for more powerful prediction
• Averaging or blending
• Weighted averaging
• Conditional averaging
• Bagging
• Boosting
• Stacking
• Stacknet

Averaging ensemble methods

• Combine two results with simple averaging
• (model1+model2)/2
• Considerable improvements with averaging can be achieved
• Perform better when combined but not individually
• Weighted average - (model1*0.7 + model2*0.3)
• Conditional average (If < 50 use model1 else model2)

Bagging

• Averaging slightly different versions of same model to improve accuracy
• Example - Random Forest
• Underfitting - Error in bias
• Overfitting - Errors in variance
• Parameters that control bagging - Seed, Subsampling or Bootstrapping, Shuffling, Column Subsampling, Model specific parameters, bags (number of models), More bags better results, parallelism
• BaggingClassifier and BaggingRegressor from sklearn
• Independant of each other

Boosting

• Weight based boosting
• Form of weighted averaging of models where each model is built sequentially via taking into account of past model performance
• Add sequentially how well previous models have done

Weight based boosting

• Number of times certain row appears in data
• Contribution to error / recalculate weights
• Parameters - Learning rate, shrinkage, trust many models, number of estimators
• Parameters - Adaboost (sklearn - python), LogitBoost (Weka - java)

Residual based boosting

• For videos mostly dominant
• Calculate error of predictions / direction of error
• Make Error new target variable
• Parameters - Learning Rate, Shrinkage, ETA
• Number of estimators
• Row sub sampling
• Column sub sampling
• Sub boosting type - Fully gradient based, Dart
• XGboost
• Lightgbm
• H2O GBM (Handle categorical variables out of box)
• Catboost
• Sklearn's GBM

Stacking

• Making several predictions of a number of models in a hold out set and then using a different meta model to train these predictions
• Stacking predictions
• Splitting training set into two disjoint sets
• Train several base learners on the first part
• Make predictions with the base learners on the second (validation) part
• Using predictions from (3) as the input to train a higher level learner
• Train Algo 0 on A and make predictions for B and C and Save to B1, C1
• Train Algo 1 on A and make predictions for B and C and Save to B1, C1
• Train Algo 2 on A and make predictions for B and C and Save to B1, C1
• Train Algorithm3 on B1 and make predictions for C1

Happy Learning!!!

Day #99 - Statistics and distance based features

Stats

• Calculate statistics of derived features from neighborhood analysis
• User_id / Page_id / Ad_price / Ad_position
• Use label encoder
• Treat data points implicitly
• Add lowest and highest price for position of add
• maximum and minimum price values
• Pages user visited
• Standard deviation of prices
• Most visited page
• Many more features
• Introduce new information

Neighbors

• Number of houses in 500m, 1000m
• Average price per sq.m
• Number of schools / supermarkets / parking lots in 500m / 1000m
• Distance to closest substation
• Embrace both group-by and nearest neighbor methods

Matrix Factorizations

• Approach for feature extraction
• User / Items mapping matrix
• User - Attributes matrix
• U X M = R
• Row and column related features
• BOW represent larger parse vector
• Document represented by small dense vector (Dimensionality reduction)
• Matrix Factorizations
• SVD, PCA, TruncatedSVD for sparse matrices
• NMF (Non-Negative Matrix Factorization) - Zero or Positive Number
• NMF makes data suitable for decision trees
• Used for Dimensionality reduction

Example Code

x_all = np.cancatenate([x_train,x_test])

pca.fit(x_all)

x_train_pca = pca.transform(x_train)

x_test_pca = pca.transform(x_test)

Happy Learning!!!

Day #98 - Advanced Hyperparameter tuning

Neural Network Libraries

• Keras (Easy to learn)
• Tensorflow (For production this is used)
• MxNet
• PyTorch (Popular in community)
• sklearn's MLP

Neural Nets

• Number of neurons per layer
• Number of layers
• Optimizers
• SGD + momentum
• Adam / Adadelta / Adagrad (In practice lead to more overfitting)
• Batch size (Huge batch size leads to overfitting)
• Epochs impact
• Learning rate - not too high not too low, Rate where network converges
• Regularization
• L2/L1 for weights
• Dropout / Dropconnect
• Static dropconnect

Linear Models (Scikit-learn)

• SVC / SVR
• Sklearn wraps libLinear and libSVM
• Compile yourself for multicore support
• LogisticRegression / LinearRegression + regularizers
• SGDClassifier / SGDRegressor
• Vowpal Rabbit
• Regularization parameter (C, alpha, lambda)
• Start with very small value and increase it
• SVC starts to work slower as C increases
• Regularization type
• L1/L2/L1+L2 - try each
• L1 can be used for feature selection

Happy Learning!!!

Day #97 - Hyperparameter tuning

How to tune hyper parameters ?

• Which parameters affect most
• Observe impact of change of value of parameter
• Examine and iterate to find change of impacts

Automatic Hyper-parameter tuning libraries

• Hyperopt
• Scikit-optimize
• Spearmint
• GPyOpt
• RoBO
• SMAC3

Hyper parameter tuning

• Tree Based Models (Gradient Boosted Decision Trees - XGBoost, LightGBM, CatBoost)
• RandomForest / ExtraTrees

Neural Nets

• Pytorch, Tensorflow, Keras

Linear Models

• SVM, Logistic Regression
• Vowpal, Wabbitm FTRL

Approach

• Define function that will run our model
• Specify range of hyper parameter
• Adequate range for search

Results

• Underfitting
• Overfitting
• Good Fit and Generalization

Tree based Models

• GBDT - XGBoost, LightGBM, CatBoost
• RandomForest, ExtraTrees - Scikit-learn
• Others - RGF(baidu / fast_rgf)

GBDT

• XGBoost - max_depth, subsample, colsample_bytree, colsample_bylevel, min_child_weight, lambda, alpha, eta num_round, seed
• LightGBM - max_depth / num_leaves, bagging_fraction, feature_fraction, min_data_in_leaf, lambda_l1, lamda_l2, learning_rate num_iterations, seed
• sklearn.RandomForest/ExtraTrees - N_estimators, max_depth, max_features, min_samples_leaf, n_jobs, random_state

Happy Learning!!!

Day #96 - Mean Encoding - Extensions and Generalizations

• Compact transformation of categorical variables
• Powerful basis of feature engineering

Using target variable in different tasks. Regression, Multi-class

• More stats - Percentiles, std, distribution bins
• Introducing new information from one vs all classifiers in multi-class tasks (N Different encodings)

Domains with many-to-many relationships

• User to Apps relationships
• Row for user-app relationship
• Vector for each app`

Time-series

• Presence of mean prev da, prev week, prev day
• Based on data create more complicated features

Encoding interactions and numerical features

• model structure, analyzing trees
• Extract from decision trees (If they are in neighboring nodes)
• xgboost, row features
• Use split points to identify new features
• Manually add more mean encoded interactions
• Involving categorical variables evaluate variable interactions

Correct validation reminder
Local experiments

• Estimate encodings on X_tr
• Map them to X_tr and X_val
• Regularize on X_tr
• Validate mode on X_tr / X_val split

Submission

• Estimate Encoding on whole Train data
• Map them to Train and Test
• Regularize on Train
• Fit on Train

Happy Learning!!!

Day #95 - Work Hacks - Machine Learning Labelling - Quick Work Hacks (Rows comma delimited text to columns)

Happy Learning!!!

Day #94 - Integrate R and Python

Happy Learning!!!

Day #93 - Regularizations

Four methods of Regularization

• Cross Validation inside training data
• 4 to 5 folds of K-Fold Validations
• Split into K non-intersecting subsets
• Leave one out scheme
• Target variable leakage is still present in K Fold Scheme
• Smoothing based on size of category
• Category big lot of data points
• Formula = (mean(target)*nrows+globalmean*alpha)/(nrows+alpha)
• alpha = category size we can trust
• Add Random Noise
• Unstable, Hard to make it work
• Too much noise
• LOO, Leave one out Regularization
• Sorting and calculating mean on some type of data
• Fix sorting order of data
• Use Rows 0 to N-1 to calculate mean for N-1
• Least Leakage

 Happy Learning!!!

Day #92 - Mean Encoding

Mean Coding

• Add new variables based on certain features
• Label encoding is done usually
• Mean encoding is done as variable count / distinct unique variables
• The proportion of label encoding also is included in this step
• Min encoding with label encoding
• Label encoding - No logical order
• Mean encoding - Classes are separable
• We can reach better loss with sorted trees
• Trees need huge number of splits 
• Model tries to treat all categories differently

Constructing Mean Encoding

• Goods - Number of ones in a group
• Bads - Number of zeros

Likelihood = Goods/(Goods + Bads) = mean(target)
Weight of Evidence = In(Goods/Bads)*100
Count = Goods = sum(target)
Diff = Goods-Bads

Happy Learning!!!

Database Sharding and Scalability Basics

Some Key considerations for NOSQL Vs RDBMS

• Performance - Latency tolerance, How slow my queries can run for huge data sets
• Durability - Data loss tolerance when database crashes losing in-memory or Lost transactions tolerance
• Consistency - Weird results tolerance (Dirty data tolerance)
• Availability - Downtime tolerance

Options for Scalability

• Replication - Create copies of database, Application can talk to either database
• Sharding - Sharding choosing a partition key, Key-value store partition based on key
• Caching - Precomputed and stored, Manage cache expiration time and refresh logic

For streaming data we had already discussed Events Hub, Apache kafka. Now we have something called KSQL (Kafka streaming SQL to run on continuous data)

Great Session Talk

 

RDBMS VS NOSQL Considerations, Quick Summary

• Performance - Latency tolerance
• Durability - Data loss tolerance
• Consistency - Weird results tolerance (Dirty data tolerance)
• Availability - Downtime tolerance

Happy Learning!!!