Day #160 - GAN Talk - Ian Goodfellow

Key Summary
Generative Modelling

• Form representation (Probability distribution) from training examples
• GAN generate more samples 

Why Generative Modelling

• Learn Distribution over different training examples
• High Dimensional Probability Distribution
• Simulate futures for Reinforcement Learning
• Realistic Image Generation tasks 

GAN Applications

• Predicting Next Frame in Video - Adverserial loss to train the image
• Super Resolution - Downsample to half resolution, SRRestNet, SRGAN
• GAN for interactive photo-editing
• Image to Image Translation - Conditional GAN for multi-modal output distributions

How Generative Models work?

• Maximum Likelihood (Write down density function that model describes)
• Distribution controlled by parameters Theta
• Density Functions Types, Accomplishing ML through different approaches
• Explicit / Implicit Density Function
• Markov Chain to estimate Density Function / Gradient
• Procedure to draw samples from the probability distribution

Fully Visible Belief Nets

• Explicit formula based on chain
• Wavenet is a Fully Visible Belief Network (Minimise cost function with no approximation)
• VAE (Write Density function where density is intractable)
• Markov Chain Perform poorly in High Dimensional Spaces

GANs

• Use a Latent Code
• Asymptotically consistent
• No Markov Chain Needed

How GAN work ?
Two Models - Adversary of Each Other
Generator - Generates Samples that resemble training distribution
Discriminator - Tool to inspect sample is real / fake (Differentialble function)

Training Procedure

• Sample two different mini batch of data
• Generator minimizes log probability of the discriminator

GAN - Realistic Samples GAN (Designed to fool discriminator)
VAE - High Likelihood use VAE (Designed to maximize likelihood)
DCGAN works very well on faces

Tips and Tricks

• Learning conditional models often gives better samples
• Label Smoothing - Good Regularizer

Happy Mastering DL!!!