Weekend Reads - Tesla AI Andrej Karpathy on Scalability in Autonomous Driving

Key Notes

• 1 million internet connected devices
• Navigate on Autopilot / Right Lane changes
• Summon to parking lot
• Divide into smaller problems, Solve and incrementally provide features
• 360 degree safety features / Safety assist
• Detection to avoid / alert accidents
• Ambition - Full Self Driving
• High Definition Maps Approach
• Similar to human decision making
• No Lidar maps approach

Neural Networks in Production

• Static objects / parking lines, Cross walks
• Hard to achieve predictions
• Stop sign on walls / flash lights / held by person / occlusion
• Situations where it applies right / left
• When to ignore , Largest dataset of stop sign
• Techniques to boost occluded data
• Chinese / Korean / Japanese Signs
• Hydranet, 48 Networks, 1000 predictions
• Shared CNN backbone
• 50 Plus seems people
• Curate dataset, new landmark
• Segmentation / Detection etc..

• Curate dataset, new landmark
• Segmentation / Detection etc..

Summary - Most of them direct detections, shared backbone, transfer learning, a lot of datasets

Tesla Logs Analysis

NTSB Tesla Analysis

Happy Learning!!!

Weekend Reads

• Anti-aging startups - This area is getting good investments - Link
• Hacking with Circuit python - Link
• Dropbox ML Read - How ML saves us $1.7M a year on document previews
• WFH Here to stay - Why employers will have no choice but to embrace remote working 
• Have not worked much on services, API. During discussions with my friend this topic, API naming conventions hit me hard link

The new investments for Anti-Aging, Electric cars, Recycling supply chain are key upcoming trends towards sustainable innovation.

Keep Thinking!!!

Kubeflow Pipelines - Learning Notes #1

To appreciate something we need to why, how, what about the Tool.

Key Notes

Why / Necessity

1. Monitoring of Model
2. Training /Serving - Differences in transformation, handling missing data
3. Frequency to refresh the model

Production System Components

Kubeflow Platform

Develop, Deploy, Manage

Pipelines, Data Management, Serving (Rest End Point) 

Pipeline Component

Commands

Setup cluster, permissions in yaml file

Demo with screenshots

Pipelines

• Domain-Specific Language
• Instantiate Components
• Define Dependency between components
• Compile and Deploy Pipeline

Custom Components

Somehow the gap between ML code vs kubeflow code there is a lot of learning. How much time it takes to port to this infra? I need to experiment to comment. A lot of features are there but we shouldn't end up rewriting ML code to pipeline code. 

Notes #2

Codify ML Workflows

Adopt pipeline mindset

Experiment, Reproduce, Share pipeline

Define Pipeline

• The description on ML Workflow
• Runs on Container
• Execution vs Runtime decoupled
• Components - one step of workflow
• Component - Packaged as Docker image
• Pod for Each Step
• Pipeline SDK

More Reads

Example

pipeline sdk key notes - Link1, Link2, Link3

SDK Summary pointers

KALE (Kubeflow Automated pipeLines Engine) is a project that aims at simplifying the Data Science experience of deploying Kubeflow Pipelines workflows.

An Argo workflow executor is a process that conforms to a specific interface that allows Argo to perform certain actions like monitoring pod logs, collecting artifacts, managing container lifecycles, etc.

Katib is a Kubernetes-native project for automated machine learning (AutoML). Katib supports hyperparameter tuning, early stopping and neural architecture search (NAS). Learn more about AutoML at fast.ai, Google Cloud, Microsoft Azure or Amazon SageMaker

Katib is the project which is agnostic to machine learning (ML) frameworks.

Ray Train, an easy-to-use library for distributed deep learning.

Dask is a flexible library for parallel computing in Python.

ML metadata (MLMD) library by Google. MLMD is an integral part of TensorFlow Extended (TFX) and a stand-alone application

The most important entities created and stored by MLMD are:

• Artifacts that are generated by the pipeline steps (e.g., the trained model).
• Metadata about the executions (e.g., the step itself).
• Metadata about the context (e.g., the whole pipeline).

Keep Thinking!!!

Interesting Research paper Read - Gender and Race Preferences in Hiring in the Age of Diversity Goals: Evidence from Silicon Valley Tech Firms

Paper - Link

Key Insights

• Women are 9-10% more likely to receive a callback compared to men, 
• Whereas Black Hispanic and Asian applicants are 8-13% less likely to receive a callback compared to White applicants

Key Notes

Studying hiring discrimination at the intersection of race and gender, giving primacy to both

How hiring discrimination, in particular, leads to occupational segregation.

Experiment #1 - Send Fictitious resumes with randomized white-sounding and black-sounding names to potential employers for different types of occupations and consistent discrimination against African Americans across occupations (Bertrand and Mullainathan 2004).

Insights

Statistical discrimination - Employer who imperfectly observes an applicant's quality and productivity resorts to group-level averages to make inferences about the individual, which may lead to discrimination

Taste-based - Employers may have a prejudiced taste and animus towards a particular group, leading to discrimination (Becker 1971).

Discriminatory phenomenon -  female discrimination in male-dominated occupations and male discrimination in female-dominated occupations

ML Approach

Keep Thinking!!!

Interesting Learning Notes

Note #1 - What is difference between performance and scalability problem ?

• Performance problem - Fixing performance issue for a website example - Pageload time is very high for homepage of website
• Scalability problem - Scaling website to support 10X user base than current user base

This post was very useful for above answer Performance v Scalability – For Employers

Note #2 - Read / Write Advantages / Disadvantages for Normalized / Denomarmalized Databases ?

• Normalized - Insert in multiple tables, Highly consistent
• Denormalized - Easy Insert, Consistency issue with updates (Multiple  versions of Records may exist)

I loved reading this post again and again - Data storages and read vs write controversy. Post is Very Simple, intuitive and clear.

Session - Advanced techniques for Web functional testing

• Tip #1 - Changing window size during execution set_window_size
• Tip #2 - Screenshots comparison using needle (Python based)

More Reads

Successful Scalability Principles

Bayesian statistics made simple

Software Carpentry: Lessons Learned

Performance Testing and Profiling: A Virtuous Cycle

Happy Learning!!!

Weekend Reading Notes

Session #1 Netflix's Distributed Computing Strategies: Optimistic Design for the Eventual Consistency Model

Good Netflix Case study on Cassandra for High Performance DB's

• In a master / slave configuration there is a interval for data sync
• Early 2000's reads were done on replicated databases 
• Repair option possible in cassandara
• MYSQL users - Facebook, Zappos, Symantec etc..
• FB replays logs across Slave systems 
• Remove Foreign keys to improve performance
• Netflix Cassandra cluster (1 Million writes / Reads worked successfully) - More reads link
• Benchmarking Cassandra Scalability on AWS - Over a million writes per second
• Pessimistic Design - High Consistency = High Latency, Performance issues 
• Optimistic Design - Trust Data Store, For 1% or edge cases have contingency plans
• Example, Amazon (Low Consistency, sometimes sell items not in inventory), Send a polite email, 10% credit for next purchase

Session #2 - How Python Scripts Power Drones

CS Concepts 

• Whats the difference between statically and dynamically typed languages?
• Explain several different CPU scheduling algorithms.

Good Programming Questions

• Given an array of integers, sort the array into a wave like array, namely a1 >= a2 <= a3 >= a4 <= a5

Good Solution Approach - Sort array first and swap pairs of numbers? So 7 6 5 4 3 2 1 becomes - 7 5 6 3 4 1 2

• Given an array of non-repeating, non-sorted, positive and negative integers, find 4 numbers in that array which satisfy the following equation: A1 + A2 + A3 = A4 
• Given a string determine which character appears the most and the number of times that character appeared.

Happy Learning!!!

Weekend Learning - Good Session - Taming Big Data with Berkeley Data Analytics Stack

Good Session - Taming Big Data with Berkeley Data Analytics Stack 

Notes captured from the session

Big Data Use Cases (Making personalized decisions for each customer, Analyse data trends)

Data Processing Goals

• Earlier Trend - Analyse historical data
• Current Trend - Real time data processing
• Goal - Sophisticated data processing (Trend analysis, Anomaly detection)

Open Analytics Stack

• Apps - Data Analysis, Mining, Decision Driven Apps
• Data Processing - HBase, Hive, Hadoop
• Storage - HDFS
• Infrastructure - Cluster

Goals of Open Analytics Stack

• Support batch, interactive and stream processing

Implementation Notes

• Store data in memory (SSD's, 512GB of RAM)
• FB / Yahoo / Bing - Some very large jobs but vast majority are pretty small
• Aggregating inputs for other jobs fit in memory of cluster
• Parallelism of jobs, Failure Recovery, Job Scheduling handled
• Trade-off between accuracy and response time
• Single execution framework for batch, streaming and interactive computations

New layers added are mentioned in ()

• Application
• Data Processing (In Memory Processing)
• Storage (Data Management Layer), (Resource Management)
• Infrastructure
• One cluster for both MPI and Hadoop
• Spark (Batch & Interactive Apps Support)
• Spark and Shark are available in Amazon Elastic Map Reduce
• Tachyon - Storage abstraction

Architecture and Component - Screenshots

Download the components from link
AMP Lab Blog link

Good Session, Happy Learning!!!