Motivation for Non-convex Optimization

• Extremely high dimensional spaces
• Web-scale document classification problems
• Imposition of structural constraints on the learning models being estimated from data
• Structural constraints often turn out to be non-convex.
• Non-convex optimization techniques, such as sparse recovery, help discard irrelevant parameters and promote compact and accurate models.

Ref - Link

Ref - Link

Why do neural nets need to be non-convex?

• Neural networks are universal function approximators
• With enough neurons, they can learn to approximate any function arbitrarily well
• To do this, they need to be able to approximate non-convex functions

Non-convex Optimization for Machine Learning

Convex Optimization for Neural Networks

Basically, since weights are permutable across layers there are multiple solutions for any minima that will achieve the same results, and thus the function cannot be convex (or concave either).

Link1, Link2

Keep Exploring!!!