Segment Anything model - Facebook Offering

1. Google Colab GPU Version

2. Sample Code

3. Sample Image

4. Sample results, Segmentation Time = 30 seconds

Key components - Image encoder, prompt encoder, and mask decoder.

• The image encoder is a pre-trained Masked Auto-Encoder Vision Transformer (MAE-ViT) that extracts an embedding for the image.
• The prompt encoder embeds prompts of different types, including points, bounding boxes, free-form text, or rough masks.
• The mask decoder has layers that use self-attention, cross-attention, and an MLP. They create a more informative image embedding, which is then used by another MLP to produce the final mask. The model also estimates IoU for later use in the process.

Ref - Link

Paper - Link

Demo - Link

Keep Exploring!!!

Segmentation - Loss Functions Notes

 

Ref - Link

Different Loss Types

U-Net and its variants for medical image segmentation: theory and applications

Basic Unet

Attention U-net

• An often-desirable trait in an image processing network is the ability to focus on specific objects that are of importance while ignoring unnecessary areas. The attention U-net achieves this by making use of the attention gate
• Repeated uses of the attention gate after each layer improves segmentation performance significantly without adding too much computational complexity to the model.

Residual U-Net

• ResNet uses skip connections which take the feature map from one layer and add it to another layer deeper in the network. 

Dense U-net

Dense U-nets employ DenseNet [67] blocks in place of regular layers. While the ResNet model

allows for deeper neural networks, it does not eliminate the problem of vanishing gradients. ResNet architecture also eventually degrade in performance with increasing layers.

U-net++ is another powerful form of the U-net architecture inspired from DenseNet [67]. It uses a dense network of skip connections as an intermediary grid between the contracting and expansive paths

An adversarial model is a setup where two networks compete against each other in order to

improve their performance. Generative adversarial networks (GAN) are a novel type of adversarial process used to generate new data

RUNet: A Robust UNet Architecture for Image Super-Resolution

• Perceptual loss function [3] during training. map the predicted SR image I and the target image IHR into a feature space and then measure the distance between the two mapped images in the feature space

Influence of tile size on the quality of recognition and extraction of roads and paths in orthophotographs with deep learning

Keep Exploring!!!

Segmentation Notes

If things work happy else its another next step towards solution :)

Residual Blocks

What is Dice Coeff and Dice Coeff Loss in segmentation explain with python code simple terms

Dice Coefficient (also known as the Sørensen–Dice coefficient) is a measure of similarity between two sets of data. It is commonly used in image segmentation to measure the accuracy of a segmentation algorithm.

The Dice coefficient is calculated as the ratio of the intersection of two sets of data to the union of the two sets. It is expressed as a value between 0 and 1, where 1 indicates a perfect match.

Dice Coefficient Loss is a loss function used in image segmentation tasks. It is used to measure the difference between the predicted segmentation and the ground truth segmentation. The loss is calculated as the negative of the Dice coefficient.

Code Examples Notes

Keep Exploring!!!

UNET Segmentation

Minor fixes in UNET Code

Keep Exploring!!!

Rembg Background Removal Tool

 Interesting background removal tool Rembg

Keep Exploring!!!

Segmentation - U2Net

Paper #1 - U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection

Key Features

• Mixture of receptive fields of different sizes
• Increases the depth of the whole architecture 
• Alexnet, VGG, ResNet, ResNeXt, DenseNet, etc. However, these backbones are all originally designed for image classification
• CNN - They extract features that are representative of semantic meaning
• Segmentation needs local details and global contrast information, which are essential to saliency detection
• Key thought - can we go deeper while maintaining high-resolution feature maps, at low memory and computation cost
• U2-Net is a two-level nested U-structure that is designed for SOD without using any pre-trained backbones from image classification
• Input size of 320×320×3
• obtain a feature vector for describing the saliency of this pixel
• saliency detection requires both local and global information
• stacking two differently configured U-Nets
• Convolution + Residual Blocks
• multiple dilated convolutions 
• Dilated Convolution: It is a technique that expands the kernel (input) by inserting holes between its consecutive elements. In simpler terms, it is the same as convolution but it involves pixel skipping, so as to cover a larger area of the input. 
• The standard keras Conv2D layer supports dilation, you just need to set the dilation_rate to a value bigger than one. For example:
• out = Conv2D(10, (3, 3), dilation_rate=2)(input_tensor)

• novel ReSidual Ublock, RSU

• ”(U×n-Net)”, where n is the number of repeated U-Net modules.
• Dialted Convoluton + Custom Resnet
• Repeated modules

• In the training process, each image is first resized to 320×320 and randomly flipped vertically and cropped to 288×288.

Paper #2 - Highly Accurate Dichotomous Image Segmentation

• new metric called human correction efforts (HCE)
• To obtain more representative features, FCN-based models [60], Encoder-Decoder [3,81], Coarse-to-Fine [96], Predict-Refine [78, 90], Vision Transformer [118] and so on are developed.

Demos & Codes

• BackgroundRemoval
• Background Removal with Deep Learning
• u2net
• Rembg

Keep Exploring!!!

Colab Pro+ Segmentation Experiments

The good thing is - Runs in the background, Close the browser, and re-login after a few hours

Cons - 

• Was not as fast as I expected it to complete the training
• Background execution terminates after 24 hours, Does not support long training
• Sometimes console was busy and unresponsive

For 50$ this is the cheapest option at the moment :)

Experiment #1 - 20K Training Images, 5K Test Images

• GPU, High RAM
• TPU, High RAM
• Colab pro+
• Batch size - 100

Failed 

Experiment #2 - 10K Training Images, 2K Test Images

• GPU, High RAM
• Colab pro+
• Batch size - 75

Failed 

Experiment #3 - 10K Training Images, 2K Test Images

• GPU, High RAM
• Colab pro+
• Batch size - 25

In Progress

Experiment #4 - 10K Training Images, 2K Test Images

• Segmentation 512 x 512
• Batch Size = 15
• GPU High RAM

1- 3 Epochs, Incremental Iterations

20 Epochs Seems ok, Not so bad

Everything seems to balance batch size, incremental training, GPU, and TPU based on the problem statement

Segmentation on 512 x 512 seems to have better performance compared to segmentation on 224 x 224

Continue Experiments!!!

Segmentation Notes

Many thanks to this article. Segmentation worked fine on cars / humans. If we could retrain this for larger image would be great

colab notebook link

Demo code link, article2

Sample output

Dataset prep script - Link1

Model codes - Link1, Link2

Keep Exploring!!!!

Segmentation Notes

Loss functions for image segmentation, Link1

Loss Functions in Segmentation

Image segmentation can be thought of a classification task on the pixel level, and the choice of loss function for the task of segmentation is key in determining both the speed at which a Machine-Learning model converges, as well to some extent, the accuracy of the model.

Ref - Link

Dice loss. This loss is obtained by calculating smooth dice coefficient function. This loss is the most commonly used loss is segmentation problems.

Ref - Link

Keep Thinking!!!

Fashion Segmentation Paper Read

Paper - U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection

Key Notes

• Capture more contextual information from different scales thanks
• Code link https://github.com/xuebinqin/U-2-Net
• Segmenting the most visually attractive objects in an image
• Deep features extracted by existing backbones, such as Alexnet [17], VGG [35], ResNet [12], ResNeXt [44], DenseNet [15]
• Convolution with stride of two followed by a maxpooling with stride of two are utilized to reduce the size of the feature maps to one fourth
• Go deeper while maintaining high resolution feature maps
• ReSidual U-block (RSU), which is able to extract intra-stage multi-scale features 
• Multi-scale feature extraction - A 3 × 3 filter is good for extracting local features at each layer
• Convolution + Feature Extraction + Downsample + Upsample

• multi-scale feature extraction target at designing new modules for extracting both local and global information from features obtained by backbone networks.

RSU mainly consists of three component (ReSidual Ublock, RSU)

• an input convolution layer, which transforms the input feature map
• a U-Net like symmetric encoder-decoder structure which takes the intermediate feature map as input and learns to extract and encode the multi-scale contextual information
• a residual connection which fuses local features and the multi-scale features

Dataset - Link

• Labelled Images Samples
• Ground Truth / Training Images
• After 600k iterations (with a batch size of 12), the training loss converges and the whole training process takes about 120 hours Sample data

Creating this is also the key

Usecases

• Remove background
• Create portrait view

Paper #2 - BASNet: Boundary-Aware Salient Object Detection

Code - Link

Background removal tool - Link

Notes

• Architecture is composed of a densely supervised Encoder-Decoder network and a residual refinement module
• Hybrid loss - Binary Cross Entropy (BCE), Structural SIMilarity (SSIM) and Intersectionover-Union (IoU) losses.
• Code Link
• It assembles a UNet-like [57] deeply supervised [31, 67] Encoder-Decoder network with a novel residual refinement module

Paper #3 - BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation

Notes

• Spatial Path with a small stride to preserve the spatial information and generate high-resolution features
• Context Path with a fast downsampling strategy is employed to obtain sufficient receptive field
• Spatial Path (SP) and Context Path (CP)

Paper #4 - BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation

Notes

This architecture involves: (i) a Detail Branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) a Semantic Branch, with narrow channels and deep layers to obtain high-level semantic context

Keep Exploring!!!

Segmentation To-do List

Labelling List

• Guidelines
• Labelme - Video Annotation
• Labelme Segmentation
• Custom OpenCV Approach

Existing Datasets

The Oxford-IIIT Pet Dataset

CelebAMask-HQ

Useful blogs

Image segmentation with a U-Net-like architecture - Keras Code Examples

Good to reuse models

• Face Segmentation
• Hair Segmentation
• Vision Transformer
• Paper - Real-time deep hair matting on mobile devices
• Image enhancement
• Image Segmentation
• Super Resolution
• Image segmentation with a U-Net-like architecture
• Hair Segmentation Realtime using Keras
• Image Captioning

Keep Updating!!!