Lecture - Visual Question Answering Based on Image and Video - Thao Minh Le

Visual Question Answering Application

demo -  Link

Pics and videos are everywhere, words are how humans communicate

• Vision + ML + NLP - Interesection of all fields
• The Flow
• Low level image processing
• Objects and Shapes
• Object recognition, Relationship between objects
• Relationship between object, events
• Pizza, Type of pizza 

Applications

• Visually impaired assistance

• Video Analytics analysis
• Check a piece of information

• Open-ended questions
• Choice-based questions
• Counting type questions

Next Steps

• Perception - Reasoning - Multistep reasoning

• Difficult for a single model to address
• Obtain knowledge - Form Relationships 
• Dataset - 1 Million questions
• bag of words to embed

• BOW + LSTM

• Reasoning - Chaining of relative predicates to arrive at the conclusion

• Objects - RCNN
• Contextual words - Bidirectional LSTM

• Connect all objects in sequence

• Semantic similarities representation

• Relational Reasoning on Visial QA

• Conditional Relation Network Unit

Every weekend makes me feel guilty about vision current state of art vs what I am working on when I will bridge the knowledge gap!!!

Interesting Vision projects

Project #1 - Enabling Those with Low Vision to Run Independently

Key Summary

• Predict runners the sidewalks of road / Visually impaired
• Segmentation model, Lane detection, Mobile deployment model
• Transfer Learning based on mobilenet. Segmentation model built

My Thoughts - Without segmentation with binary threshold, OpenCV as well we can attempt the same. Applying sobel / laplacian filters extract edges and attempt to find

Project #2 - Smart Toilet

Data upload - Link

Project - Link

• Purpose - Insights from human feces
• Classification - patient’s stool form (i.e., loose, normal or constipated) 
• Detect blood in stool

Can't imagine this use case. So many use cases beyond what we think.

Vision Repos

• Awesome Computer Vision
• Vision Repos
• 500+ Projects and code
• ProteinBERT: A universal deep-learning model of protein sequence and function
• LaneNet-Lane-Detection

Keep Thinking!!!

Interesting Vision Projects

Product #1 - Open Vision API - open source computer vision API based on open source models.

source code - Link

Product #2 - Which Frame? - Link

Search a video semantically

Product #3 - Selfie based facial skin assessment

Dermatology dataset - Link

Project #1 - Link 

Project #2 - Skin lesion detection from dermoscopic images using Convolutional Neural Networks

Project #3 - Deep convolutional network to predict 23 classes of skin disease

Papers 

• Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• How image search works at Dropbox
• Openimagesdataset - Link
• A Computer Vision Application for Assessing Facial Acne Severity from Selfie Images
• Estimating Skin Tone and Effects on Classification Performance in Dermatology Dataset

Keep Exploring!!!