Weekend Lectures - Andrej Karpathy: Tesla Autopilot and Multi-Task Learning for Perception and Prediction

Key Lessons
Single Task

• For Lane tasks to do are - cars detection, multiple car types, take complete viewpoint, occluded cars
• Architecture, Loss function, Object detection techniques
• Train - Test - Fix until you get the required accuracy

Multi-Tasks

• For a moving vehicle, the tasks to compute are Static Objects, road signs, overhead signs, Traffic lights, Lane Lines, Road markings, curbs, crosswalks, environment tags
• All these inputs will be worked on simultaneously
• Single tasks, Subtasks of moving objects
• For every vehicle - vehicle type, lights, indicators, blinkers. All are independent predictions

Key Learnings

• Architectural considerations
• Loss function considerations
• Training dynamics
• Team workflow
• Feature sharing at which layer

Architecture choices

• Moving objects
• Static objects
• Signs
• Traffic lights
• Decide frame rate according to tasks
• Feature sharing between tasks
• Tasks to create features

Paper References
Auto DeepLab
Which tasks should be learned together in multi task learning

Integrating multiple views

• 8 cameras inside the vehicle
• Different viewpoints of each camera
• Some of the layers can be shared as edges/shape can be similar
• RNN sharing features from different frames
• Static parts / Moving parts / path prediction
• A lot of domain-driven optimization
• Sampling tasks / sub-sampling networks / Sharing features

Loss functions

• Each task will fill into same loss and backpropagate
• Panoptic feature Pyramid networks 
• Include both Object detection and Semantic segmentation
• Large grid search over task weights to find best mean average precision

Training Dynamics

• Data Distribution
• Over Sampling 
• Within task oversampling (Liked it)
• Semi balanced batches
• Data Engine for each task and dedicated instances
• Single task - Early Stopping when validation loss is lower

Session #2 - Deep learning applications: training a multi task classifier

• Fetch Product, Price, Feature, Time <-> Price mapping, Multiple prices over time
• 90K brands
• Single task vs Multi-task classifier
• Leverage current Keras custom loss generator

Multi-task

• Leverage a few layers common for multiple class
• Classify 2 labels vs n labels
• Right product @ Right price @ Right time = Best price @ available margins @ Right time

Example of Multi-Task

• Vehicle Classification + Vehicle Color Detection
• Vehicle Classification + Type detection

Keep Thinking!!!