"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;

October 12, 2018

The Value of Data

The value of data across the chain as we look at data from different perspectives.

Early Stages of Data Analysis

  • Data Collection
  • Data Reporting
  • Dashboard Reporting

Mid Level Data Analysis

  • Metrics / Measures for Creation (Business Intelligence / Trends)
  • Predictive (Forecasting)

Advanced Level

  • Machine Learning Based Models
  • End to End Automated Workflow chain with all levels of Data Analysis



Pain Points of BI

  • Reactive Mode - It doesn't help us to know the future but rather learn from past
  • Data Distribution - Cannot fit Data and Distributions to understand better, Data can be fit into any form of pattern / distribution. BI does not help us Deep Dive to identify that
  • Tight Schema Bound Architecture - Models, Dimensions are Static, Dynamically cannot change the facts, a measure without changing underlying ETL

Happy Understanding with Data!!!

October 11, 2018

Day #139 - Video Analytics Case Study - Retail Scene Gun Detect Alert Situation

Summary of key points from papers studied, Sample projects, Code base to follow it further. This is a summary and bookmark. Credits to bookmarked papers / authors

Papers Studied
Paper #1 - Real Time Monitoring of CCTV Camera Images Using Object Detectors and Scene Classification for Retail and Surveillance Applications

Key Points
  • AlexNet has 60 million parameters and 650,000 neurons, consists of five convolutional layers. Those layers are followed by max-pooling layers, and three globally-connected layers with a final 1000-way softmax layer
CNN Architecture Overview
  • Kernels are learnable filters
  • Pooling layer (sub-sampling) reduces the dimensionality of feature map
  • Output of Softmax function categorical distribution
Datasets
Application Architecture
Input CCTV Video -> Detection using CNN -> Push Notification to Mobile

Paper #2 - Crime Scene Prediction by Detecting Threatening Objects Using Convolutional Neural Network

From the scene the objects are extracted, CNN is used to identify. Detecting weapons is a bigger challenge than identifying / classifying it.

Paper #3 - Automatic Handgun Detection Alarm in Videos Using Deep Learning
  • Reformulated problem to reducing number of False Positives
Detection Models
  • Sliding Window Approach
  • Large number of candidate windows
  • Runs Classifier on all windows
  • HOG based model
  • Good at 0.07 frames per second / pedestrian
Region Proposals
  • Region based CNN
  • Approach selects accurate candidate regions
  • CNN to extract features, SVM to classify them
  • Good at 7 frames per Second
  • Alarm Activation Time per Interval (AATI)
Paper #4 - Automated Detection of Firearms and Knives in a CCTV Image (Michał Grega*, Andrzej Matiolanski, Piotr Guzik and Mikołaj Leszczuk)

Challenges
  • Blurryness
  • Low Resolution
Knife Detection Algorithm
  • Input Image -> Sliding Window -> Feature Extraction -> SVM Classification -> Decision
Firearm Detection Algorithm
OpenCV Techniques used
  • Background Detection
  • Canny Edge Detection
  • PCA
Neural Network operates on the Image Obtained after PCA (Reduced Dimensions)
OpenCV Post Evaluation
  • Dialation
  • Erosion
  • Difference between two images
Experimentation - Tested this haar cascade based implementation, Works fine for Gun Detection
Results


Other Interesting Projects
Real Time Implementation of Gunshot Detection System
tensorflow-gun-detection
Deep Neural Net Approach To Identify Guns

Happy Learning!!!

October 10, 2018

Video Analytics Use Cases

Real-world Use Cases from Surveillance / Security Cameras (Simple to Medium)
  • Object Detection
  • Face Detection
  • Slip and Fall
  • Loitering
  • Crowd Detection
  • Abandoned Objects
  • People Counting
  • Gender-Based People Counting
  • Attendance / Visitor Tracking
  • Traffic Signal / Lane Detection / License Number Tracking
  • Emotion Analytics
  • Parking Analysis
  • Mob / Crowd / Weapon Detection in Public Places
Advanced Video Analytics Use Cases (Complex) - Deep Learning Use Cases
  • Medical Processing X-Ray / Genome Data 
  • Tumor Identification
  • Autopilot Self Driving
  • Smart Farming - Fruit Ripening / Fruit Classification
  • OCR Recognition / Financial Data Analysis
  • Smart Home management / Intrusion Detection
  • Online Media Moderation - Abuse / Content Rating / Detection
Manual Fruit Classification



Computer Vision Size Classification

Packing Validation




Grape Detection and Segmentation



Smart City Analytics - ML + DL
  • Parking Analytics
  • People Counting
  • Face Recognition
  • Intrusion Detection
  • Device Monitoring And Alerting
Happy Learning!!!

Day #138 - Opencv Based Blob Detection and Image Extraction



Happy Learning!!!

October 08, 2018

October 07, 2018

Day #136 - RNN Session

Notes from session - Andrej Karpathy, Research Scientist, OpenAI - RE•WORK Deep Learning Summit 2016 

RNN
  • Operate on Sequences
  • Image Captioning (Produce Sequence of words)
  • Sentiment Classification / Machine Translation
  • Video Classification
  • RNN has a state which update everytime when vector comes in x
  • RNN can process vectors so it is encoded
  • Use RNN to generate sequences after learning from dataset
  • Interesting examples of RNN code, function, bogus comments, syntactically very few mistakes
  • min-char-rnn.py, char-rnn projects to look out for (https://gist.github.com/karpathy/d4dee566867f8291f086)
  • Detect and describe objects in image in single pass
Happy Learning!!!

October 05, 2018

Deep Dive PySpark Examples - Big Data Setup - Part II

After experimenting a bit of pyspark I feel Its much better to handle with R / Python. Most of things we can achieve are repetitive between R /Python / Spark / SQL.

  • Data Pipeline tasks at DB Level
  • One Hot Encoding also can done with basic TSQL Code
  • While working in NLP it makes sense to use TF-IDF Vectorizers

Happy Learning!!!


October 04, 2018

Day #135 - Research paper - Human-Centered Autonomous Vehicle Systems


Its a long day and only now I got time to post on this interesting paper - Human-Centered Autonomous Vehicle Systems

I love this paper for the practicality and applicability of this kind of systems. Autonomous Systems and Humans are dependent on each other to learn / teach and understand each other for Safer modes of transportation. In such situations systems have to share common responsibility and dependency to make the rides safer.

This paper underlines this important idea and the shared responsibility across multiple levels. Supervised learning, Personalization, better communication for Safer Navigation

Some key lines (copied from paper)
  • Machine learning is primarily used for the scene understanding problem but not for any other aspect of the stack driving scene
  • Perception, motion planning, driver sensing, speech recognition, and speech synthesis are all neural network models
  • Calibrate distraction, fatigue, cognitive load, emotional state, and activity (uses facial landmark configuration and facial motion analysis)
  • Communicate the degree of uncertainty in the neural network prediction, segmentation, or estimation about the state of driving scene
  • Arguing machines framework (detailed in [12]) to provide human supervision over the primary perception-control system
Nice Lines
"both humans and AI systems have flaws, and only when the share autonomy paradigm is considered at the system level do those flaws have a chance to be leveraged to become strengths"

The same principles should apply for chatbots for better communication. 

Happy Learning!!!

Predictions by 2025

A quick write up on my predictions for technology/education and other sectors by 2025. Will revisit it again after a few years.
  • Large scale deployment of Video Analytics in Scale for Home monitoring / Attendance Monitoring / Retail / Gesture-based tracking. The scale of adoption (30~40% in Indian Middle-Class Home market)
  • Smart Cities Reality Check - Bringing together IoT + Analytics + Solar Energy into better / optimum utilization of resources (In metros compliance would be there for all SEZ, Commercial establishments - 50-60%)
  • Redefined Education - Customized Learning based on Individual patterns / Choices  (30~40% in Indian Middle-Class Education Market, Affordable, Free and Customizable)
  • Healthcare Analytics - More Adoption for Automated / Proactive alerting from Video / Data Analytics (30% of diagnosis would be done by Analytics- prediction)
  • More Open Source Quality Education Tools - More new you-tube similar channels targeting education domain / free and open-source quality education for Higher studies (Udemy, Coursera, Khanacademy probably focused on Engineering, Medical, Arts, Science very specialized channels with focused groups across the globe)
  • Drone / Quad-copters / Air Ambulance - Based on Solar + Drone Technology (They will have a decent market share and adoption for delivery / carry the minimum load)
  • Battery / Electric cars vs Fuel Based - This would still be the same unless the cost of product / better innovation takes place to reduce the cost of manufacturing for Battery cars (20~30% Battery cars in Indian Automotive market)
  • Big Data + Analytics - More the data, More the chaos, Real-time meaningful analytics from huge data sources vs computing power in Home devices or Smart Homes will still be a challenge 
#RandomThought

Update May 29/ 2020
Corona crisis has updated a lot of things. Glad to see many startups built on unreasonable practices are gone from the market.

COVID Accelerated the below things.
  • Online Education and free education is becoming a reality. 
  • Work from Anywhere is becoming a reality.
  • Every Retail store is becoming a Digital Store
  • AI in healthcare / screening 
Link
Key Notes
  • Device to custom food with nutrients
  • Autonomous vehicles market share increase by 50%
  • Electric vehicles market share increase by 50%
  • Use of Green energy
  • Other relevant 2025 predictions listed above
  • SpaceX Mars Colonization
  • Global Warming
  • Impact of China on the World



Happy Learning!!!

October 02, 2018

Day #134 - Summary of learning's from stitchfix

Algorithms Summary from stitchfix Blog
  • Collaborative filtering algorithms  for recommendations (User - User / Item - Item Recommendations)
  • cosine similarity, Finding Image based Similarity 
  • NLP on client feedback to create score of feedback (Possibly positive / Negative / Neutral based on words used)
  • Travelling Salesman problem to identify optimal delivery cycle
  • Keep track of every touch point of transaction to predict / pro-active on next move
  • Markov chain model - Demand Modelling, This could be classic machine learning model that includes several feature variables (Seasonality / Trend / Style etc..)
Notable Quotes
  • Engineers Shouldn't Write ETL
  • Data Platform team enables data scientists to carry algorithm development all the way from concept to production
Happy Learning!!!