Transformer Walkthrough

Session #1

Session #2

Keep Exploring!!!

Transformer Notes

• CNN - all the outputs are self dependent 
• CNN/FF-Nets all the outputs are self dependent Feed-forward nets don’t remember historic input data
• RNN - Hidden state memory, correlation between previous input to the next input, Cell state, Forget Gate
• RNN - learn to keep only relevant information to make predictions and forget non relevant data RNN - Conveyer belt
• RNN Perform well when the input data is interdependent in a sequential pattern correlation between previous input to the next input introduce bias based on your previous output

Transformer

• Positional embeddings - the order and position of words in a sequence 
• Self attention - allows each token to dynamically weigh and integrate information from all other positions 
• The self-attention mechanism is a type of attention mechanism which allows every element of a sequence to interact with every others and find out who they should pay more attention to.
• Multi-head attention runs multiple self-attention processes in parallel, capturing diverse aspects of the data

Keep Exploring!!!

Encoder / Decoder Discussions

GPT

• The GPT-2 is built using transformer decoder blocks
• GPT generates one token at a time just like decoder of transformer and has causal language modeling so it is strictly decoder only model.
• GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encode
• Next word as outputs but it is auto-regressive as each token in the sentence has the context of the previous words

BERT

• BERT, on the other hand, uses transformer encoder blocks
• BERT gained the ability to incorporate the context on both sides of a word to gain better results
• BERT generates same number of tokens as input that can be fed to linear layer and uses masked language modeling so this is strictly encoder only model.
• BERT, by contrast, is not auto-regressive. It uses the entire surrounding context all-at-once.

Decoder -  pay attention to specific segments from the encoder

Ref - Link1, Link2

Keep Exploring!!!

Document Q&A

From OCR, Document Extraction, Understanding, Hugging face has come a long way :)

DocQnA Pipeline very impressive

Results

Keep Exploring!!!

• TesserOCR
• MMOCR
• OCRmypdf
• EasyOCR
• PaddleOCR
• Kraken
• OCRopus
• PyOCR
• Tesseract

OCR, Fintech and Usecase

Keep Learning!!!

Transformers - Lesson 2

One-liners - Summary from link

• Lesson #1 - Sentences numerically represented a 1 and 0 on occurence of work, This is one hot encoding
• Lesson #2 - Sequences, when we setup, every next word would be possible combinations/ words divided by count, is the probability, First-word sequence
• Lesson #3 - If we know multiple words in sequence it would be easier / more confident to nail down on possible words
• Lesson #4 - When you consider word by word its is next word, When you keep sequences it becomes easier, Instead of one word consider it like triplets, bi-gram, tri-gram remember sequences in multiple combinations
• Lesson #5 - Create sequences skipping words
• Lesson #6 - Embeddings to leverage similar words
• Lesson #7 - Positional encoding to consider positions/location in the embedding space

I still need few more iterations but this this is first cut understanding.

Keep Exploring!!!

Attention - Lessons

From Application Developer vs Knowing how it works, Still trying to figure out, Be it backpropagation or Network design or Attention.

Wonderful thread 

"Attention is all you need" implementation from scratch in PyTorch. A Twitter thread:
1/

— abhishek (@abhi1thakur) December 13, 2021

Summarizing my lessons

• Lesson #1 - Encoder takes embeddings and source masks
• Lesson #2 - Decoder takes target embedding and target masks
• Lesson #3 - Encoder has sequence of encoders one connected to each other, Encoder1 -> Output -> Encoder2 -> ... Encoder N
• Lesson #4 - Encoder N will be connected to Decoder 1, Decoder has several Decoder Layers
• Lesson #5 - Encoder contains one sequential layer + attention + feedforward layer
• Lesson #6 - In RNN when we read we remember input gate, foreget gate, history gate, output gate. Something here you have the connection to self called self attention, Something like keeping the sequence history
• Lesson #7 - Multihead attention = Multiple self attention layers
• Lesson #8 - Self attention = attention to remember the same sequences, 1-2-3,1-2-3,Again a percentage of sequence might be picked up as historical info which may influnce the next token prediction
• Lesson #9/#10/#11 - Forward function = Softmax + matrix multiplcation
• Lesson #12/#13/#14 - Decoder has similar attention layer, multihead self attention
• Lesson #15/#16 - Padding, Positional encoding with embedding layer
• Lesson #17/#18 -  linear + softmax to decoder output

Keep Exploring!!!

Transformer - Let's relearn

Transformer - Let's relearn

These topics come and on and off. I was able to catch up with sliding windows, CNN, RNN, LSTM. Then a bit of Transformers and also how does it work in vision too :)

AI / ML won't let us feel guilty you have to still learn the basics.

Paper - Attention Is All You Need

Key Lessons

• Representation of the sequence
• Intra-attention of sequence order
• Encoder-decoder structure
• Encoder - a sequence of continuous representations
• The decoder then generates an output sequence (Positional encoding)
• Multi-Head Attention consists of several attention layers

Unofficial Walkthrough of Vision Transformer

• Image is also pixels, learning pixel representations then the same encoding / decoding can be applied.

Transformers for Image Recognition at Scale

Key Notes

• Input image as a sequence of image patches, similar to the sequence of word embeddings
• The Vision Transformer treats an input image as a sequence of patches
• ViT can learn features hard-coded into CNNs (such as awareness of grid structure)
• Image classification with Vision Transformer
• Image classification with Vision Transformer

AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE

• Split an image into fixed-size patches
• Linearly embed each of them
• Add position embeddings
• Feed the resulting sequence of vectors to a standard Transformer encoder

Do Vision Transformers See Like Convolutional Neural
Networks?
pdf: https://t.co/5Yz5F2PZwO
abs: https://t.co/bpHO2rOYDv

find striking differences between the two architectures, such as ViT having more uniform representations across all layers pic.twitter.com/0KT0KE16f9

— AK (@ak92501) August 20, 2021

Do Vision Transformers See Like Convolutional Neural Networks?

• Lower half of ResNet layers are similar to around the
• lowest quarter of ViT layers
• Highest ViT layers dissimilar to lower and higher ResNet layers.

Keep Thinking!!!

Back to Basics - Fundamentals - RNN - Transformers

 It needs a bit more careful *attention* to understand the crux of transformers. This lecture was useful

Slides - Link

Session - 

Transfer Learning

• Use Neural Network on imagenet and finetune on custom data
• Better performance than anything else

Convert words to vectors

• One hot encoding
• Scales poorly with vocabulary size
• Sparse and high dimensional
• Map one hot to dense vectors (Embedding matrix)
• Finding Embedding matrix - Learn as part of tasks
• Learn the Language model
• Training on large corpus of text - wikipedia
• N-Grams, Sliding Window forming rows
• Binary classification - 0 / 1 - Neighbouring word or not

NLP Imagenet moment - Elmo / ULMfit

• ELMO - bidirectional stack LSTM
• ULMfit

Good Paper Read - SQuAD: 100,000+ Questions for Machine Comprehension of Text

Attention

• Only attention no LSTM
• Self-attention, positional encoding, Layer normalization
• Attention and Fully Connected Layers

Self Attention

• Input sequence of vectors
• Output weighted sum of input sequence

Learn weights

• Compute attention weight for its own output
• Compute every other vector to compute attention weights for its own output y_i (query)
• Compare to every other vector to compute attention weight w_ij for output y_j (key)
• Summed with other vectors to form the results of the attention weighted sum (value)

Multihead attention

• Weight matrices - query, key, value weights
• Multiple heads of attention just mean learning different sets of query, key and value matrices simultaneously

Transformer

• Self attention layer - layer normalization - dense layer

Layer Normalization

• Data scaling, weight initialization
• Rest things between uniform mean and standard deviation

Position Embedding

• Word embedding depends on word
• Position embedding depends on position 
• Combine both and run through transformers
• Both position and content reasoned

Attention is all you need

• Translation
• Encoder - Decoder architecture

GPT - Generative pretrained transformer

• Generating text
• ELMo, ULMFIT
• Preceeding words
• GPT2 1.5 Billion parameters

BERT

• Bidirectional encoder representations from transformers

T5 - Text to Text Transfer Transformer

• Input and output as text streams
• 11 billion parameters

Keep Thinking!!!