"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;

March 03, 2022

Focus vs Non-Critical Interruptions vs Meetings

 


Keep Exploring!!!

Managing a Project vs Managing a Team vs Managing Self

Managing Project

My working style is

  • Prototype as much as possible to plan for tasks
  • Guided knowledge with some prep work
  • Formulate experiments / options / blockers / alternatives
  • I prefer to know the big picture, clarity with some early prep and continuous learning to guide/unblock as needed

Other ways of Managing project

  • Task-based / Time based
  • I prefer to go with Know-how and know things from an implementation perspective
  • Everything is connected knowledge, You cannot be hands-on in every area but knowing / measuring / probing to understand from the end solution point of view matters

Managing Team

  • Where they are currently vs What is their goal
  • Look for a long term perspective, What aligns to their growth path
  • More the team less the time you can dedicate for every one

Managing Self

  • Am I learning things I like to do 
  • Does this align with my long term perspectives
  • What I do to refresh my first principles
  • Do I keep learning / connecting the dots with the big picture?

Career/title/skill everything is a point in time snapshot. Even the strongest will become weaker at some point in time. Keep ego/arrogance aside, life is too short to feel I Am the smartest, sometimes it is more about how you genuinely support/understand and get things going. A big no for services or pure people management roles. Pick and choose what best works for you.

Keep Going!!!

March 02, 2022

Research Paper Reads - Fruit Freshness

Paper #1 - Deep Learning Based Classification System For Recognizing Local Spinach

  • Five types of spinach. (a)Jute Spinach, (b)Malabar Spinach, (c)Red Spinach, (d)Taro Spinach, (e)Water Spinach.
  • Because our dataset contains different sizes of the images. 224x224x3 is the input shape of our model

Paper #2 - Automatic Plant Cover Estimation with Convolutional Neural Networks

  • This dataset contains 7,200 images with 750 training and 150 validation images per class and is therefore also balanced
  • Our network consists of two main components: a feature extractor backbone and a network head
  •  The backbone consists of one of the abovementioned standard classification networks potentially in conjunction with a Feature Pyramid Network to increase the output resolution

Paper #3 - Fruit Freshness Grading Using Deep Learning

  • Fruit colour is derived from natural pigments when ripening, enzymatic and non-enzymatic browning reactions lead to the formation of water-soluble dark colours
  • Visual characteristics, e.g., shape, wholeness, spots, bruises, and blemishes, can reflect the speed of fruit deterioration
  • The consistency of physical shape may indicate the thickness of fruits that may have implications of its capability to defend against diseases
  • Geometric changes are a frequently observed result of fruit degradation
  • Texture is another important measurement of the level how a fruit has decayed
  • Fruit texture, colour and shape are three important visual features for fruit quality grading


  • Added random noises follow the sequential order: Random brightness adjustment, random contrast, and random erasion


  • In total, there are (approximate) 4,000 images collected with each type of fruit about 700
  • The freshness grading is scaled from 0.0 to 10.0 with 0.0 indicating total corruption and 10.0 for total freshness 
  • We define the fruits being harvested as absolute freshness with a numerical level description of 10.0

Paper #4 - Machine Vision based Fruit Classification and Grading - A Review

  • Some extraction methods like Speeded Up Robust Features (SURF), Histogram of Oriented Gradient (HOG) and Local Binary Pattern (LBP) 
  • Features of fruits like color, size, shape and texture
  • Automatic sorting system that can perform fast, save time and reduce manual labor
  • The basic steps of the automatic image-based fruit grading are: fruit image recognition, fruit object recognition



Color features extraction methods broadly fall in two categories:

1. Global methods (global color histogram, histogram intersection, image bitmap)

2. Local methods (local color histogram, color correlogram, color difference histogram)

# Fitting K-NN to the Training set

from sklearn.neighbors import KNeighborsClassifier

classifier = KNeighborsClassifier(n_neighbors = 5, metric = 'minkowski', p = 2)

classifier.fit(X_train, y_train)

# Predicting the Test set results

y_pred = classifier.predict(X_test)



Paper #5 - Deep Learning for Automatic Quality Grading of Mangoes: Methods and Insights

  • Augment the CNN classifier with a convolutional autoencoder
  • The rationale of considering such networks is as follows: 1) the presence of autoencoder
  • forces the network to remember essential information for reconstruction when extracting features for classification, thereby having a regularizing effect; and, 2) the latent features learned by the network could benefit other downstream tasks as they contain the compressed information for reconstruction;

Data Augmentation

  • Horizontal or vertical flip, each with 50% probability;
  • Brightness, –20 to +20%;
  • Contrast, –10 to +10%;
  • Rotation, –20 to 20 degrees;
  • Zoom in/out, 0.8x to 1.25x

The ConvAE-Clfs consist of 3 components:

  • A convolution-based encoder that compresses an image into a latent vector;
  • A convolution-based decoder that reconstructs the image from the latent vector and some intermediate features;
  • A fully-connected classifier that takes the latent vector as input and gives the class prediction.

The proposed convolutional autoencoder-classifiers were shown to have no clear advantage over the single-task CNNs, but the result should be verified with larger datasets and more related tasks

Paper #6 - Deep Learning and Machine Vision for Food Processing: A Survey

  • An MVS includes two main parts to enable objective and non-destructive food evaluation: 1) acquiring and 2) processing


Keep Exploring!!!

Story telling with data - Early Thoughts

I accidentally had to skim through the book storytelling with data. The context of data presentation and some examples sharing it here.

Example #1

Chart #1


Updated Chart #1

My Observation - Subjective outcomes so bar chart would convey highs/lows effectively


Example #2

Chart #1

Updated Chart #1

My Observation - One-liner info added. Plus volumetric growth is projected.

Example #3

Chart #1


Updated Chart #1

My Observation, Line chart is less clutter in terms of low / high patterns

Summary

  • Understand the context
  • Choose an appropriate visual display
  • Eliminate clutter
  • Focus attention where you want it
  • Think like a designer
  • Tell a story
Disclaimer - Subjective topic, lies on individual choices and preferences though :) What you can interpret.

Keep Exploring!!

February 28, 2022

Where is all interesting DS work in Fintech, Auto, Oil and Gas, FMCG, Retail, Healthcare, Insurance

Almost all the startups / core work is around delivery, eCommerce, fashion. Predominantly I could spot research papers from Swiggy, Myntra, Flipkart. They are consistent in applying and sharing their lessons with the broader community. 

Facebook/ Google they have a ton of data, they are at another level in terms of innovation, adoption. Tesla is a pioneer above all playing its own game.

The rest of fortune 100, 500, Other companies in Fintech, Auto, Oil and Gas, FMCG, Retail, Healthcare, Insurance. Hope they also come forward and share, solve problems equally challenging and innovative. Does it mean 80/20%? 80% of core work is done by only 20% of companies.

Very generic JDs, Ton of skills listed without calling out what they intend to do, what kind of future they like to build. 

Keep Thinking!!!

February 26, 2022

Data scientist - 1JD and 10 different Skills

  • Statistics 
  • Machine learning
  • MLOps
  • Cloud
  • Deep learning
  • NLP
  • Chatbots
  • Vision
  • Optimization
  • Pick and choose your expertise
My Perspectives
  • Foundation / First-principles understanding of basics
  • I keep forgetting if I don't revise, The actual project work might involve data/vision specifics depending on your consulting work
  • Doing your work vs preparing for broader understanding for interviews is not worth the time, The gap between reality vs jd is way too far
  • Pick and choose areas where you love to experiment, gain more understanding
  • Full Stack Expert vs Full Stack Exposure know yourself better :)

Keep Thinking!!!

February 21, 2022

Code Perspectives

 


Source - Link

Domain - Ideas - Algos - Prototype - Scale..

Keep Exploring!!!!


February 18, 2022

Hair styles research paper reads

Paper #1 - MichiGAN: Multi-Input-Conditioned Hair Image Generation for Portrait Editing

Code - Link

Key Notes

  • MichiGAN is capable of enabling multiple input conditions for disentangled hair manipulation.
  • Editing appearance (b), structure (c), and shape (d) while keeping the background unchanged
  • Disentangle the information of hair into a quartet of attributes – shape, structure, appearance, and background, and design deliberate representations
  • Appearance is encoded through our mask-transformed feature extracting network
  • Background encoder is placed parallel to the generation branch, which keeps background intact
  • An explicit disentanglement of hair visual attributes, and a
  • set of condition modules that implement the effective condition mechanism for each attribute with respect to its particular visual characteristics;
  • An end-to-end conditional hair generation network that provides complete and orthogonal control over all attributes individually or jointly;
  • An interactive hair editing system that enables straightforward and flexible hair manipulation through intuitive user inputs



  • We represent the hair shape as the 2D binary mask of its occupied image region
  • Backbone generation network to bootstrap the generator with specific appearance styles instead of random noises.
  • Force the GAN to reconstruct the same background content;

Loss Types - 

  • Feature matching loss. To achieve more robust training of GAN, we also adopt the discriminator feature matching loss
  • Perceptual loss. We also measure high-level feature loss with the pre-trained VGG19 model
  • Structural loss. We propose an additional structure loss to enforce the structural supervision

Paper #2 - LOHO: Latent Optimization of Hairstyles via Orthogonalization

Code is available at Link

Notes

  • Our approach decomposes hair into three attributes: perceptual structure, appearance, and style, and includes tailored losses to model each of these attributes independently.
  • Optimizing StyleGANv2’s extended latent space and noise space
  • Novel approach to perform hairstyle transfer on in-the-wild portrait images and compute the Frechet Inception Distance (FID) score. FID is used to evaluate generative models by calculating the distance between Inception [29] features for real and synthesized images in the same domain


  • Pretrained VGG [28] to extract high-level features 

Paper #3 - Applications of Generative Adversarial Networks in Hairstyle Transfer

Notes

  • InterFaceGan, StyleGan

Paper #4 - Learning to Generate and Edit Hairstyles

Notes

  • GAN model termed Hairstyle GAN (H-GAN)
  • Recognition, generation and modication of hairstyles, by using a single model.
  • VAEGAN [14] integrates the Variational Auto-Encoders (VAE) into GAN
  • InfoGAN [5] further models the noise variable z in Eq (1) by decomposing it into a latent representationy and incompressible noise z



Keep Exploring!!!

February 16, 2022

Developer Guidelines

Taking inspiration from post customizing it for developer guidelines

  1. Prioritizing cool things and how to decide what to do from product/time perspective
  2. Trusting an approach without knowing how it works - Don't do
  3. Analyze and build your data-driven perspective - Take time to think
  4. Principles to develop products and live your creative life
  5. What you should “demand” from your peers / PM Team  
  6. Dealing with Top-Down requests and focusing on effectiveness - Work vs Todo Random Requests
  7. What matters when building a product vision
  8. Dealing with Impostor Syndrome and Hero Syndrome 
  9. A framework to make your product live 100x more impactful 
  10. How to work better with peers engineers and designers
  11. Who to follow and what to read to learn more about better career goals
Keep Exploring!!!


February 13, 2022

Perspective of Interview

  • During Interview - Puzzles, Physics, Permutations - Sometimes I felt I need to be a maths teacher
  • On Job - Dirty Data, No clear problem statement, Basics of schema/data engineering we don't know but talk about Feature stores, No features but already on MLops Era
  • Working on #docker #python #api #kubeflow did not make me #fullstack #developer but it was only #fullstack #exposure, to #master need more #time #perspectives

Interview vs Reality vs Job everyone looks from their own lens. Reality is far from what we think

From LinkedIn post

Because in real life, it doesn't matter how well you crack hard algorithmic questions.

What matters is:

  • What projects you've built
  • What you've got to say
  • How you manage stakeholders
  • How you manage stress

These are the driving factors behind your success in the field.

More read - Link



Keep Thinking!!!