Optimization solver / Anamoly Detection

Hexaly is the world’s fastest optimization solver for Routing, Scheduling, Packing, and more.

Use Cases

• Modeling and Solving the Aircraft Landing Problem
• How Hexaly solves Renault Group’s workspace allocation problem in seconds
• Hexaly establishes new records for the Inventory Routing Problem (IRP)

Anamoly 

• Anomaly Detection
• Anomalies in Log Patterns

Keep Exploring!!!

Forecast + Optimization

• Regression to find optimal values of 'X' values
• Add a constraint to make it an optimization problem
• Optimization with minimum expense for each track

Keep Exploring!!!

GPT to learn some more ML optimisation / causal inference

What is numerical optimisation, How it is different from machine learning. List some usecases problems in numerical optimisation ?

Numerical optimization is the process of finding the values of input parameters that minimize or maximize a given objective function. It is different from machine learning in that it does not involve training a model with data, but instead relies on mathematical algorithms to find the best solution.

Some use cases of numerical optimization include:

• Portfolio optimization
• Supply chain optimization
• Vehicle routing
• Facility location
• Network design
• Scheduling
• Parameter estimation
• Image registration
• Model calibration
• Optimization of control systems

What are top 10  python packages for numerical optimization

• NumPy 
• SciPy 
• Scikit-Optimize 
• PyOpt 
• PyGMO 
• CVXOPT 
• Pulp 
• Pyomo 
• DEAP 
• Heuristic Optimization Algorithms (HOALib)

List some constraint for windmill turbine based on temperature, sunlight, humidity, windspeed with synthetic data python code pulp optimization solution

constraints = [

    # Temperature

    pulp.lpSum([temp_var[i] for i in range(len(temp_var))]) <= max_temp,

    pulp.lpSum([temp_var[i] for i in range(len(temp_var))]) >= min_temp,

    # Sunlight

    pulp.lpSum([sunlight_var[i] for i in range(len(sunlight_var))]) <= max_sunlight,

    pulp.lpSum([sunlight_var[i] for i in range(len(sunlight_var))]) >= min_sunlight,

    # Humidity

    pulp.lpSum([humidity_var[i] for i in range(len(humidity_var))]) <= max_humidity,

    pulp.lpSum([humidity_var[i] for i in range(len(humidity_var))]) >= min_humidity,

    # Windspeed

    pulp.lpSum([windspeed_var[i] for i in range(len(windspeed_var))]) <= max_windspeed,

    pulp.lpSum([windspeed_var[i] for i in range(len(windspeed_var))]) >= min_windspeed

]

List some usecases problems in operations research ?

• Vehicle Routing Problem
• Scheduling Problem
• Network Flow Problem
• Knapsack Problem
• Integer Programming Problem
• Queuing Theory Problem
•  Linear Programming Problem
•  Resource Allocation Problem
•  Facility Location Problem
•  Project Scheduling Problem

List some usecases problems in causal inference ?

• Estimating the effect of a treatment on an outcome of interest
• Estimating the effect of a policy change on a population
• Estimating the effect of a marketing campaign on sales
• Estimating the effect of a new product on customer retention
• Estimating the effect of a change in pricing on customer demand
• Estimating the effect of a change in customer service on customer satisfaction
• Estimating the effect of a change in advertising on brand awareness
• Estimating the effect of a change in the labor market on wages
• Estimating the effect of a change in government regulations on economic growth
• Estimating the effect of a change in the environment on public health

What is casual inference in layman terms

Casual inference is a type of reasoning that allows us to draw conclusions based on our observations. In layman's terms, it is the process of making assumptions or guesses about something based on what we know or have seen. For example, if you see someone walking down the street wearing a suit and carrying a briefcase, you can infer that they are likely on their way to work.

List some usecases problems in Bayesian modelling ?

• Estimating the probability of a medical diagnosis given a set of symptoms.
• Predicting the probability of a customer purchasing a product given their browsing history.
• Estimating the probability of a machine failure given a set of sensor readings.
• Predicting the probability of a customer defaulting on a loan given their financial history.
• Estimating the probability of a terrorist attack given a set of intelligence data.
• Predicting the probability of a stock price movement given market data.
• Estimating the probability of a natural disaster given weather patterns.
• Predicting the probability of a customer churning given their usage data.

What is difference between causal interference vs AB testing

Causal inference is a method of determining the cause-and-effect relationship between two variables. It is used to identify the cause of a particular outcome. It is based on the assumption that there is a causal relationship between the two variables.

AB testing is a method of comparing two versions of a product or service to determine which one performs better. It is used to determine which version of a product or service is more effective. It is based on the assumption that the two versions are statistically different.

List some usecases problems in differentiable programming ?

• Optimizing hyperparameters in neural networks
• Automatically generating neural network architectures
• Automatically generating code from a given specification
• Automatically tuning the parameters of a given model
• Automatically generating data pre-processing pipelines
• Automatically generating feature engineering pipelines
• Automatically generating loss functions
• Automatically generating optimization algorithms
• Automatically generating regularization techniques
• Automatically generating distributed training strategies

Bayesian modeling is generative probability based on historical data, Is it correct ?

Yes, that is correct. Bayesian modeling is a type of generative probability model that uses historical data to make predictions about future events.

When can we use differentiable programming , Does it have advantage over deep learning

Differentiable programming can be used when a problem requires the use of a combination of symbolic and numerical computation. It has the advantage of being able to combine the flexibility of symbolic computation with the power of deep learning. This allows for more complex models to be created and trained, which can lead to better results. Additionally, differentiable programming can be used to create models that are more interpretable and easier to debug.

Keep Exploring!!!

Delivery Optimizaton at Amazon

• Local optimizations were tackled first
• Stop consolidation — a chain of addresses linked by a small road segment
• Same-day delivery consolidation

Strategies

• Customer Order and Network Density OptimizeR (CONDOR).
• Simultaneously determining how orders are split into shipments and the sourcing fulfillment center for each shipment.
• Determine the right tradeoff between the levels of complexity and optimality.
• Single warehouse may assemble the order if it has all the items; if not, the order can be split into multiple shipments.

Ref Link 

Keep Exploring!!!

Model Optimization / Performance Key Notes

• Knowledge distillation is a method in which a small model (student) is trained to mimic a larger model or ensemble of models (teacher). 
• DistilBERT, reduces the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.
• Pruning was a method originally used for decision trees where you remove sections of a tree that are uncritical and redundant for classification
• The more common meaning is to find parameters least useful to predictions and set them to 0
• Quantization reduces a model’s size by using fewer bits to represent its parameters. 
• By default, most software packages use 32 bits to represent a float number (single precision floating point). If a model has 100M parameters, each requires 32 bits to store, it’ll take up 400MB. If we use 16 bits to represent a number, we’ll reduce the memory footprint by half. Using 16 bits to represent a float is called half precision.

Knowledge Distillation

Pruning in Keras example

Ref - Link 

Ref2 - Link 

Large Transformer Model Inference Optimization

• Reduce the memory footprint of the model by using fewer GPU devices and less GPU memory;
• Reduce the desired computation complexity by lowering the number of FLOPs needed;
• Reduce the inference latency and make things run faster.
• Post-Training Quantization (PTQ): A model is first trained to convergence and then we convert its weights to lower precision without more training
• Unstructured pruning is allowed to drop any weight or connection, so it does not retain the original network architecture.
• Structured pruning aims to maintain the dense matrix multiplication form where some elements are zeros

Keep Exploring!!!

Edge Deployment Optimization thoughts

1. Deploy lite weight models. Deploy Quantized models
2. Minimal edge processing, Detailed cloud processing
3. Message loss prevention with Queues and async processing
4. Transfer only selected frames instead of videos
5. Offline video upload to cloud vs Real-time selected image upload for real-time notifications

Keep Thinking!!!

Branch and Bound for Mixed Integer Solutions

Continuous Variables - Variables can attain any value between lower and upper bound

Discrete - Variables with particular values - Size, Binary, Set of values

Ref - Link1  

Branch-and-Bound (BB) method. – the framework for almost all commercial software for solving

mixed integer linear programs. Basis of all modern MILP codes.

Algo ref

Branch and Cut - Pure cutting plane algorithms. Ineffective for large Mixed Integer Linear Programming.

Ref - Link 2

Package

A parallel branch-and-bound engine for Python

TSP using Branch and Bound Algorithm 

Notes

If the given problem is a discrete optimization problem, a branch and bound is a good choice.

Branch and bound work efficiently on the combinatory optimization problems

Ref - Link 

Papers

• Combining regression models and metaheuristics to optimize space allocation in the retail industry
• AN INVESTIGATION OF AUTOMATED PLANOGRAMS USING A SIMULATED
• ANNEALING BASED HYPER-HEURISTIC
• A Heuristic Approach for a Shelf Space Allocation Problem 
• Layout Optimization and Promotional Strategies Design in a Retail Store based on a Market Basket Analysis
• A New Hybrid Optimization Algorithm for the Optimal Allocation of Goods in Shop Shelves
• Shelf Space Allocation: A Critical Review and a Model with Price Changes and Adjustable Shelf Heights
• Optimization Algorithms for the Shelf Space Allocation Problem
• Interaction Between Shelf Layout and Marketing Effectiveness and its Impact on Optimizing
• Shelf Arrangements
• An Implementation of the Branch-and-Price Algorithm Applied to Opportunistic Maintenance PlanningBranch and Cut: An Empirical Study

Linear Programming

Keep Exploring!!!

Essentials of Metaheuristics

Essentials of Metaheuristics

Key Notes

• Stochastic optimization -  employ some degree of randomness to find optimal solutions
• Metaheuristics are applied to I know it when I see it problems
• Hill-climbing is a simple metaheuristic algorithm
• All metaheuristics are essentially elaborate combinations of hill-climbing and the random search

• To Tweak a vector we might (as one of many possibilities) add a small amount of random noise to each number
• Single-State Global Optimization Algorithms

Simulated Annealing

• Simulated Annealing gets its name from annealing, a process of cooling molten metal
• If R is better than S, we’ll always replace S with R as usual. But if R is worse than S, we may still replace S with R with a certain probability P(t, R, S):

Iterated Local Search

• Iterated Local Search (ILS) tries to search through this space of local optima in a more intelligent fashion: it tries to stochastically hill-climb in the space of local optima

The Genetic Algorithm

To breed, we begin with an empty population of children. We then select two parents from the original population, copy them, cross them over with one another, and mutate the results.

More Reads

• Metaheuristic Optimization
• Using meta-heuristics and machine learning for software optimization of parallel computing systems: a systematic literature review
• Application of meta-heuristic algorithm for multi-objective optimization of sustainable supply chain uncertainty
• METAHEURISTIC OPTIMIZATION OF POWER AND ENERGY SYSTEMS: UNDERLYING PRINCIPLES AND MAIN ISSUES OF THE ‘RUSH TO HEURISTICS
• Metaheuristics with Local Search Techniques for Retail Shelf-Space Optimization
• Joint Optimization of Product Price, Display Orientation and Shelf-Space Allocation in Retail Category Management*
• ROBUST OPTIMISATION METAHEURISTICS FOR THE INVENTORY-ALLOCATION PROBLEM 
• Local Search for the Vehicle Routing Problem
• A Public Transport Bus Assignment Problem: Parallel Metaheuristics Assessment
• MAP Clustering under the Gaussian Mixture Model via Mixed Integer Nonlinear Optimization
• Mixed-Integer Nonlinear Optimization
• Learning to Search in Branch-and-Bound Algorithms⇤

Keep Exploring!!!

Linear programming - Optimization Problem - Tools

Linear programming is a simple technique where we depict complex relationships through linear functions and then find the optimum points.

Let us define some terminologies used in Linear Programming using the above example.

Decision Variables: The decision variables are the variables which will decide my output.They represent my ultimate solution. To solve any problem, we first need to identify the decision variables.

Objective Function: It is defined as the objective of making decisions.

Constraints: The constraints are the restrictions or limitations on the decision variables.They usually limit the value of the decision variables.

PuLP – LP Solver Front-End

LpProblem - Defines the LP problem. Holds the constraints and objective function.Interface to the LP Solver (external)

LpVariable - Abstracts an LP variable (with name). Values will be changed by the solver.Float or integer. Defines the permitted variable value range

LpConstraint – Constraint rule. Can be one of: <=, =, >=

Keep Learning!!!

Learning Notes - Convex Optimization in Python with CVXPY

Key Notes
Convex Optimization problem

• Decision variable to solve
• Objective Function
• Inequality Functions
• Linear equality constraints
• All functions are convex
• Convexity - Positive curvature, curve upwards like parabola

Useful in below fields

• Applies in various fields
• All ML algos based on Convex Optimization
• SpaceX landing Convex Optimization

How to Solve ?

• Using Solvers for particular form of problem
• Linear program, quadratic problem, second order con program

Convexity Verification

• Using Disciplined convex programming
• Determine curvature of every node in expression tree

cvxpy

• Developed in python
• Modelling framework
• Parameter assignment
• Simulation visualization
• Energy Management (Load Demand, Battery Charge / Discharge / Price margin / Load)

Keep Thinking!!!

Weekend Learning - Convex Optimization - Stephen Boyd, Professor, Stanford University

Key Notes
Mathematical Optimization

• Choices of a vector/numbers
• Constraint - Legal / Technical / Physics
• Judged by objectives
• Examine on profit/utility

Purpose

• Make good actions
• Reduce risk/ cost is objective / action
• Constraint come from the manufacturing process

Variables

• Vector x could be trades, schedule 
• Resource allocation 
• Optimize signals

AI / Stats / ML

• X - parameters to model
• Constraints (impose requirements)
• Optimization used for worst-case analysis

Optimization-based models

• Aggregate small number of agents
• Simplistic assumptions and formulate the problem
• Predictive ability of models

Convex Optimization

• Minimize objective
• Constraint to hold
• Linear constraints
• Constraints and linear functions will curve up

Why?

• Methods available to solve them

Different application areas

• Spacex landing is effort of optimization
• Optimal trajectory to landing path
• 10 times a second
• Networking / Circuit design

How to use ?

• Formulate as convex problem

Examples

Example #1 - Radiation treatment planning

• Things decided are actions
• linear y = Ax
• options - beam diverges / tissues / hits bone scatters
• Overcharge / Undercharge

Example #2 - Image in painting

• Guess the lost parts
• Minimize function / Convex problem
• Remove 5% of pixels

SVM

• Predict boolean outcome
• spam/ fraud 
• Old school - gradient method
• Convex Optimization - Differentiability irrelevant

Lasso

• Methods for sparse model construction
• With 1/5th measurements analyze

Solving

• Define in High Level language
• Solved by solver
• Helps in rapid prototyping

Large Scale Distributed Optimization

• Grid Updates
• Image / Video processing

Read Ferenc Huszár's answer to Why is Convex Optimization such a big deal in Machine Learning? on Quora Happy Learning!!!