Vision Latest Notes - Text-to-Image Generation

Vision Latest Notes - Text-to-Image Generation

Alignment in Text-to-Image Generation

Key components

• Controllable Generation
• Editing
• Better following prompts
• Customization

Techniques

• GAN
• Auto-Regressive
• Diffusion
• Non AR Transformer

One Liners

• GAN - Learn to Fake it until it becomes Real
• AR - Image to patches, Patches mapped to indexes - Tokens, Prediction - Set of tokens one by one
• Non-AR Transformer - Schedule policy to generate tokens at Each step
• Diffusion - Random noise in each step, Subtract noise to end up with the required semantic quality

Diffusion Overview

GLIDE

DALL-E-2

Image Super-Resolution via Iterative Refinement

Keep Exploring!!!

GAN Limitations

 

Good Read - Limitations of GAN

Ref - Link

Keep Exploring!!!

GAN Vision Startups

Ref - Link

Selected Vision Investments

Keep Exploring!!!

ML - Model Shipping Factories

5 years ago, #startups were in areas/segments 

• AI-Driven Sales -Forecast, Recommendations (Data)
• Chatbots - (NLP, Data)
• Autopilot ADAS - Vision - Image, Video, Data
• BPO - Customer support - NLP, Data, OCR

Companies that manufacture large models (Ref)

Vision / Image

Text

Audio

With GAN models, everything converges. The problem is already solved with LargeLanguageModels. Evolve to the next level or perish. #GAN #AI #startups. 

Keep Exploring!!!

GAN - Vision - Booth.ai Startup Analysis

Product Features

Feature #1 - Custom image generation based on text

Feature #2 - Mention what aspects you need to filter

Feature #3 - Product background customization

Feature #4- Customization with shadow enrichment, Ref - Link, 

Feature #5 - Design Studio - Customize and prepare different campaign ads

Ref - Link

Notes from FC on booth AI

• The outputs can be instantly iterated on and are indistinguishable from the results of a live shoot
• Allow customers to build ads from their AI-created photographs, distribute them across digital channels, and monitor performance.

Take this word :) - If you’re building with generative AI, we’d love to hear from you! Feel free to reach out at jkaehms@foundationcap.com or jchen@foundationcap.com.

Keep Exploring!!!!

Computer Vision - Startup Analysis - GAN

Computer Vision - Startup Analysis

Image Recognition - facial recognition, digital image search, image categorization, optical character recognition (OCR), defect detection, media, and security surveillance

Clarifai 

• Clarifai has developed a platform for building AI-powered software solutions via API, mobile SDK Different industries,including retail, manufacturing, media, and transportation, among others
• Low-to-no-code user interface

Products

• Scribe Label - a platform for data labeling
• Spacetime Search - an AI-based visual search engine that looks for similarities
• Enlight Train - a custom model training suite packed with pre-trained AI models
• Flare - an edge AI platform that offers advanced predictive capabilities and accelerates intelligent video applications from streaming, decoding, batching

Medical Imaging

Medical imaging include x-rays, ultrasound imaging, mammography, computed tomography (CT) scans, MRI, and nuclear medicine, among others.

• Potential applications of medical imaging include confirming, assessing, treating, and following the progress of diseases and certain medical conditions
• Deep learning to assist radiologists in making data-driven decisions in breast cancer screening, treatment, and diagnosis
• AI-based breast cancer detection and treatment solutions that evaluate mammograms, leading to improved clinical results, lower healthcare costs, and better-served patients

Products

• cmTriage® is a workflow management and organization tool that sorts a radiologist’s mammogram workflow
• cmAngio® targets the detection of heart disease, leveraging AI to help doctors assess a patient’s age-adjusted risk of heart disease
• cmDensity™ cmDensity™ is a radiology tool that assists in classifying breast density

Video Analytics

CVEDIA develops and deploys AIbased video analytics for both the commercial and defense sectors.

• Deploy AI solutions for object recognition, enhanced safety, optimized efficiency, and exploring new product opportunities. 
• CVEDIA-RT The company’s software offering comprises an AI-powered modular,cross-platform inference engine,providing both high and low-level interfaces.

Precision Agriculture

Precision agriculture is a management strategy incorporating high-tech tools such as sensors, GPS, robots, mapping tools, and data analytics for maximized economic return and minimized environmental impact

Analyze the data collected by hardware-agnostic sensors

Data-driven approach to crop management, empowering clients with all-encompassing analytics of their food production processes

• Yield prediction: Prospera combines machine learning and computer vision algorithms to analyze leaf-level images that help track the growth cycle and detect areas of replanting
• Field irrigation issues detection: The company uses multispectral imaging to analyze images from satellites, drones, and planes to identify areas of overwatering, underwatering, or machinery malfunctions
• Pests and disease detection: Prospera’s technology analyzes thousands of data sets from cameras installed on drones or other machinery to detect the presence and extent of certain pests or diseases.
• Crop nutritional deficiency prevention: Using Prospera’s technology, growers can get an undisrupted in-field overview and in-depth insight into problems that cannot be identified with the naked eye but are critical to crops’ nutritional value

Smart Data Capture

Smart data capture (often referred to as automated or intelligent data capture) is the process of extracting and accessing real-time data from barcodes, text, IDs, and objects by leveraging AI-based software

Scandit is a software company that develops smart device scanning solutions powered by computer vision

• The software can process up to 480 scans per minute
• It is suitable for enterprise-grade applications, featuring market leading scan accuracy and scalability to support large implementations.
• ShelfView is a cloud-based retail shelf management solution delivering data capture and analytics capabilities for greater shelf visibility and intelligent and efficient store management operations

Ref - Link

Ref - Link

Keep Exploring!!!

Vision - Theory vs Practice vs Reality - Startups - Make Money don't debate :)

Leverage all sources of data - Text, Video, Image, Audio

GAN in business.

What we study

What they sell

GAN

Image Related - Crop, Zoom, Super Resolution

Video

Audio

Startup inspiration runawayml

Keep Exploring!!!

GAN - Observations

Reading about GAN is easier but sad not to get any output after 2000 Epochs.

CycleGAN - CycleGAN is a model that aims to solve the image-to-image translation problem

After 2000 Epochs :( :(. Knowing is 10%, Experimenting is 50%, and Mastering is 40%. Always experiment.

MnistGAN, Loss in CycleGAN

How the loss is calculated while training?

Adversarial Loss: We apply Adversarial Loss to both the Generators, where the Generator tries to generate the images of it's domain, while its corresponding discriminator distinguishes between the translated samples and real samples.

Cycle Consistency Loss: It captures the intuition that if we translate the image from one domain to the other and back again we should arrive at where we started. Hence, it calculates the L1 loss between the original image and the final generated image,

CycleGAn Experiments - Implementing CycleGAN

Image to Image Translation using CycleGANs with Keras implementation

Experimented this code - Code Example

Keras-GAN, Conditional GAN

6 GAN Architectures

• Transforming an image from one domain to another (CycleGAN),
• Generating an image from a textual description (text-to-image),
• Generating very high-resolution images (ProgressiveGAN) and many more

Loss Notes, pixelwise MSE loss

• CycleGAN Tutorial from Scratch: Monet-to-Photo
• CycleGAN and pix2pix in PyTorch
• CycleGAN-PyTorch

Keep Exploring!!!

GAN Models Study

ChatGPT is not all you need. A State of the Art Review of large Generative AI models

• DALLE-2 model - text to 3D images,
• ChatGPT - texts to code
• Flamingo model - texts to video
• Phenaki model - texts to audio

• DALL·E 2, created by OpenAI, is able to generate original, genuine and realistic images and art from a prompt consisting on a text description
• Imagen is a text-to-image diffusion model [17] consisting on large transformer language models
• Stable Diffusion : Stable Diffusion is a latent-diffusion model that is opensource and has been developed by the CompVis group at LMU Munich

Models Timeline

Transformer-Based GAN for New Hairstyle Generative Networks

Generator

• Resnet50
• Downsampling - Strided convolution
• Residual blocks - Do not change width or height of activation map
• Downsampling - Dialted Convolution

Discriminator

• Pixel Exact Images
• Multihead self attention
• Multi-Head Attention

UNETGAN

MagGAN

MaskGAN

LOHO

Flickr-Faces-HQ Dataset (FFHQ)

huggan/CelebA-HQ

Keep Exploring!!!

Beauty - Paper - Research Reads / Inspirations

Paper #1 - Modifying Face Image for Ageing Marks using Specialized Filter

• shrinking image - cv2.INTER_AREA
• stretching image - cv2.INTER_CUBIC

Key Notes

• Landmarks points [0-24] represent outer face region.
• Points [25-32] represent left eyebrow region. 
• Points [33-41] represent left eye region. 
• Points [45-51] represent right eyebrow region. 
• Points [52-59] represent right eye region.
• Points [ 68-78] represent lip regions and remaining points from [79-99] covers nose position. 
• Face mask is generated using convex hull Technique [1] using the outer face landmark points.

Transformations

• We use horizontal and vertical sobel filter for detecting the wrinkles in the specific region. 
• The average edge strength in each region is defined as the quantification of wrinkles feature.
• We apply a threshold condition to the edge intensity for getting the correct wrinkles from the image. 
• Main concentration will be on the forehead and eye corners

Paper # - BEHOLDER-GAN: GENERATION AND BEAUTIFICATION OF FACIAL IMAGES WITH CONDITIONING ON THEIR BEAUTY LEVEL

Key Notes

• Progressive Growing of GANs (PGGAN) [13] suggested coping with the challenge of generating high-resolution images by learning first through generation of low-resolution images and progressively growing to higher resolutions
• Another important aspect of GANs is their ability to generate images with conditioning on some attribute
• CycleGAN, StarGAN

Code - Link

• This is more useful for cheek/chin expansion

• To ensure that generated image x indeed corresponds to the correct beauty level Discriminator D predict the beauty level and not just the usual real vs. fake probability

Paper - Facial makeup transfer with GAN for different aging faces

Key Notes

• Firstly removing the eyebrows and eyelashes of the input image to prepare for the eye makeup transfer

• Since the information is transferred from pixel to pixel, it needs to be fully aligned before transfering, and then layer is decomposed by the Edge Preserving Smooth Filter

Paper - Facial Makeup Transfer Combining Illumination Transfer

• Facial makeup, eye shadow and lip makeup are processed by different loss functions, and the three are integrated
• OpenCV Bilateral Filtering Algorithm [12] to achieve facial smoothing

BasicSR (Basic Super Restoration) is an open-source image and video restoration toolbox based on PyTorch

•  Real-ESRGAN: A practical algorithm for general image restoration
•  GFPGAN: A practical algorithm for real-world face restoration
•  facexlib: A collection that provides useful face-relation functions.
•  HandyView: A PyQt5-based image viewer that is handy for view and comparison.
•  HandyFigure: Open source of paper figures

Paper # - Data Article template

Paper - Towards Real-World Blind Face Restoration with Generative Facial Prior

• GFPGAN consists of a degradation removal module and a pretrained face GAN as facial prior
• Facial component loss with local discriminators to further enhance perceptual facial details
• Image Restoration typically includes super-resolution, denoising, deblurring and compression removal
• Channel Split Operation is usually explored to design compact models and improve model representation ability

• latent features Flatent to map the input image to the closest latent code in StyleGAN2
• multi-resolution spatial features Fspatial for modulating the StyleGAN2 features.

Paper - SCUT-FBP: A Benchmark Dataset for Facial Beauty Perception

• We extracted 84 points as sample points containing facial contour information and shape information of the eyebrow, eyes, mouth, and so on
• The machinelearning methods we used include SVM regression (SVR), linear regression, pace regression, and Gaussian regression.

Dataset - Link

Paper - Improving Makeup Face Verification by Exploring Part-Based Representations

• The preprocessing step starts by applying a face detector followed by a 2D facial landmarks estimator, both available in DLib
• These landmarks are used to align, crop and resize the face thirds and facial parts
• left periocular, which includes the eye and eyebrow, right periocular, nose and mouth
• Makeup Face Dataset (EMFD)
• Youtube Makeup (YMU) Dataset [13]

FA-GANs: Facial Attractiveness Enhancement with Generative Adversarial Networks on Frontal Faces

• we prefer to enhance facial attractiveness via adjusting the relative distances among important facial components, such as eyes, nose, lip, and chin. 

BeautyGAN - BeautyGAN: Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network

BeautyGAN

Link1, Link2

Keep Exploring!!!