Day #258 - Deep Learning - Non Convex Optimization

Convex optimization there can be only one optimal solution. Non-convex optimization may have multiple locally optimal points. Hence, finding the global minimum is very difficult.

What makes non-convex optimization hard?

• Potentially many local minima
• Saddle points
• Very flat regions
• Widely varying curvature

Examples of non-convex problems

• Matrix completion, principal component analysis
• Low-rank models and tensor decomposition
• Maximum likelihood estimation with hidden variables
• Usually non-convex
• The big one: deep neural networks

How to solve non-convex problems?

• Stochastic gradient descent
• Mini-batching
• SVRG
• Momentum

One good slide from link

Ref Link - 

Ref - Link

Ref -  Link

Backprop - Link

Happy Mastering DL!!!