"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;

June 28, 2021

CVPR Paper Reads - Large-scale Product Recognition

CVPR Paper Reads - Large-scale Product Recognition

Paper #1 - 1st Place Solution to CVPR 2021 AliProducts Challenge: Large-scale Product Recognition

Key Lessons

  • The final solution employed 11 models including three backbones: efficientnet, efficientnetv2, and nfnet.
  • Small models were trained with less epochs and large models were trained with more epoch

Data Augmentation

  • RandomCrop: 448*448
  • RandomRotation: ±30°
  • RandomHorizontalFlip: p=0.5

Paper #2 - Solution for Large-scale Long-tailed Recognition with Noisy Labels

Key Lessons

  • CNNs and Transformer, including ResNeSt, EfficientNetV2, and DeiT
  • Ensemble three different network architectures with ImageNet pretrained weights, including ResNeSt-101, DeiT-small and EfficientNevV2-m.

Paper #3 - An Effective Ensemble Method for AliProducts Challenge: Large-scale Product Recognition

Key Lessons

  • The AliProducts dataset consists of more than 3M images of nearly 50K different products.
  • All networks are initialized with pre-trained weights on ImageNet and trained with cross entropy loss.
  • As for image augmentation, we use RandomCrop, RandomHorizontalFlip as well as Nomalization

Paper #4 - RETAIL VISION WORKSHOP 2021 - PRODUCT PRICING CHALLENGE(4TH PLACE SOLUTION)

Key Lessons

  • First step involves detecting the prices present on shelves. A single class called "pricing"  (Bounding Box)
  • Second step is to detect and recognize text present inside the pricing. Google Vision API was used for text detection and recognition
  • Price Text Box Extraction: The text box with the max area containing only number was chosen the price box(or integer part of the price). 
  • Price Text Cleaning, Price Rounding off

Summary -  As we can see a mix of techniques custom detection, OCR comes into play for item price area detection, parsing, cleaning, and product match based on both text, price, value. We could also do a similar image / key points match too.

More reads - Link

Keep Thinking!!!

June 27, 2021

Metrics of product building

However, we have sprints/process/domain expertise. 

  • What is the measure of how much we know the product perspective, vision vs implementation
  • After few sprints how do product and business feel about the outcome, Are they in line with what is developed vs envisioned
  • Everyone has a way of conveying/thinking their perspective, How do we call out / communicate all the business flows and ensure we keep everyone on the same page
  • How much of the team believes / inline with the storyline and implementation
  • Tech stack never ends, business domain learning, new trends keep popping up. 

I have less time left, Its better to fight selective battles, I recognize my time is less when I near 40s

Keep Thinking!!!

June 26, 2021

Tech Leader vs Business Leader

A business leader who knows about domain but not about technology cannot sell solution capabilities in terms of technology effectively. To appreciate the technical capabilities you need a certain level of tech acumen. Today the line of tech and business knowledge keep overlapping

What happens here?

  • Afraid of experimenting 
  • Look for expertise outside
  • You will get stuck to evaluate/promote the internal tech team

A tech leader who does not understand the business will not be able to succeed in his role or with his team. If you do not develop from domain perspective you will ultimately burn out with a pile of unsold inventory of tech solutions.

What happens here?

  • Build prototype in all areas
  • Have diverse focus no deep expertise
  • Lacking business insights will lead to aborted projects not meeting customer expectations

I have observed both types of leadership resulting in burnout and missing innovation.

All this will result in impacting the company culture, delivery, and work-life balance. We have abundant tech talent but less collaboration and vision. Working on one idea for 5 years will give you more refinement/clarity/focus vs working on 10 ideas in 5 years. Expertise, Experience, Perspectives come from time. Every course cannot directly give you the knowledge you need unless you experiment/modify and keep adding more insights/lessons. 

Take time, Build your own path!!!

Keep Thinking!!!





Convert avi to mp4 in ffmpeg for streamlit

Streamlit didn't work with avi. ffmpeg tool worked for converting from avi to mp4

What did not work 

ffmpeg -i video_Raw.avi -c:v copy -c:a copy -y video_Raw_New.mp4

What Worked 

ffmpeg -y -i video_Raw.avi -vcodec libx264 video_Raw_New.mp4

Keep Learning!!!


Notes from Azure Synapse Training

Lesson #1 - Tables – Indexes Best Practices

  • Clustered Columnstore index (Default Primary) - Highest level of data compression. Best overall query performance
  • Clustered index (Primary) - Performant for looking up a single to few rows
  • Heap (Primary) - Faster loading and landing temporary data. Best for small lookup tables
  • Nonclustered indexes (Secondary) - Enable ordering of multiple columns in a table. Allows multiple nonclustered on a single table. Can be created on any of the above primary indexes. More performant lookup queries
Queries with the following patterns typically run faster with ordered CCI:
  • The queries have equality, inequality, or range predicates
  • The predicate columns and the ordered CCI columns are the same.
  • The predicate columns are used in the same order as the column ordinal of ordered CCI columns.
  • Caching of results, Enable caching at DB level then query level - Resultcachehit flag returns the value whether it was reused

Fact table primarily CCI as we would run large aggregations based on dimensions so CCI becomes a choice for fact tables. 

Lesson #2 - Distributed table design recommendations

  • Hash Distribution: Large fact tables exceeding several GBs with frequent inserts should use a hash distribution.
  • Round Robin Distribution: Potentially useful tables created from raw input. Temporary staging tables used in data preparation.
  • Replicated Tables: Lookup tables that range in size from 100’s MBs to 1.5 GBs should be replicated. Works best when table size is less than 2 GB compressed.

Lesson #3 - Result-set caching

Cache the results of a query from SQL pool storage. This enables interactive response times for repetitive queries against tables with infrequent data changes. The result-set cache persists even if SQL pool is paused and resumed later. 

Cache Checks

You can tell if a query was executed with a result cache hit or miss by querying sys.pdw_request_steps for commands where value is like ‘%DWResultCacheDb%’

Lesson #4 - SQL Data Classification is a new feature in the Public Preview, that:   

  • Automatically discovers columns containing potentially sensitive data
  • It provides a simple way to review and apply the classification recommendations through the Azure portal.
  • The sensitive data labels are persisted in the database (metadata attributes) and it audits and detects access to the sensitive data.
  • We offer built-in set of labels and information types, however customers can chose to define custom labels across Azure tenant using Azure Security Center

Lesson #5 - Dynamic Data Masking

  • Prevent abuse of sensitive data by hiding it from users
  • Easy configuration in new Azure Portal
  • Policy-driven at table and column level, for a defined set of users
  • Data masking applied in real-time to query results based on policy
  • Multiple masking functions available, such as full or partial, for various sensitive data categories (credit card numbers, SSN, etc.)
Lesson #6 - Spark vs SQL Server (Memory Handling)

Keep in mind spark uses memory much in the same way as sql server uses the buffer pool by storing frequently used objects in memory it reduces overall I/O and improves performance in large joins, sort and aggregates contrast this with a traditional hadoop based architecture which relies heavily on writing data out to disk between steps.

Every concept technical maps as an advancement or some sort of limitation which existed in place. Compared to SQL 2008 where you don't have so much of these feature synapse has beautifully evolved as a good environment for real-time / ML / big data handling capability for reporting / Ml recommendations/lakehouse / real-time BI system. Gone are the days of month-end jobs or Data sync jobs. 

All good lessons :) Fantastic Features!!!

June 21, 2021

My Perspective of Interviews

Few things I keep a tab on from a time/candidate perspective - Listening to candidate answers, asking for quantifiable data, cover all areas during the time.

  • Project discussions - To bring the best out of candidates I ask them to pick their best projects to demonstrate architecture challenges, performance issues, deployment.
  • Introduce certain scenarios/brainstorm to get perspectives from candidates. I look at areas they are able to explore / with constraints the alternatives they bring to the table.

Make it a good experience for the candidate, We all keep learning. Be better than yesterday.  

Keep Thinking!!!

Interesting observations tesseract

While extracting digits from analog meters below two links we use to get the values

Lesson #1 - Setting the path to a folder vs complete executable, Minor thing took a while since not using it often

Ref - Link

Lesson #2 - Very useful for different situations on how it can be interpreted, 11 worked best. 6 was ok


Ref - Link

Keep Exploring!!!

June 14, 2021

Quick Research Paper Reads - Retail - Supply Chain

Price Optimization in Fashion E-commerce

Key Notes

  • Key parameters - product display page, MRP and the discounted price, clickthrough rate (CTR) & conversion
  • To maximize revenue, we need to predict the quantity sold of all products at any given price
  • Another significant challenge is cannibalization among products
  • We overcame this problem by running the model at a category level and creating features at a brand level, which can take into account cannibalization
  • To solve it, the Linear Programming optimization technique


Feature Engineering

    
Linear Programming
Now we need to choose one of these three prices such that the net revenue is maximized.

Online Data Sources
  • Clickstream data: this contained all user activity such as clicks, carts, orders, etc.
  • Product Catalog: this contained details of a product like brand, color, price, and other attributes related to the product.
  • Price data: this contained the price and the quantity sold of a product at hour level granularity.
  • Sort Rank: this contained search rank and the corresponding scores for all the live products on the platform
Key Notes
  • The task of assortment planning is to determine the optimal subset of k products to be stocked in each store so that the assortment is localized to the preferences of the customers shopping in that store.
  • Broadly there are three aspects to assortment planning, (1) the choice of the demand model, (2) estimating the parameters of the chosen demand model and (3) using the demand estimates in
  • an assortment optimization setup.
  • The forecast demand will then be used in a suitable stochastic optimization algorithm to do the assortment planning.
In the age based model for demand forecasting of fashion items, the demand of an article i in store s at time t, is formulated as:

June 13, 2021

Domain + Tech + Impact

My role has always been innovating / initiatives/domain knowledge-driven based use cases for more revenue opportunities / lowering the cost of operations / better customer service. Recollecting some of my milestone projects.

Reverse Logistics

Customer Service Projects

  • Better delivery insights/emails to measure status at each leg
  • Provided more touchpoints for better repair/refurbishment delivery

Warranty rewrite

  • Rewrite warranty with traceability to new rules
  • Data lineage for different warranty rules
  • Tracking between repairs/exchanges

Vision for Retail Innovation

  • RFID, EAS, and legacy devices for people counting, loss prevention vs vision-based solutions
  • Ideate, prototype, demonstrate, patent. I wasn't there to collaborate or see how intel scaled it up but happy for the ideas that went till NRF / products

Startups collaboration

  • Vision for ad effectiveness, Measuring sales impact from digital displays
  • Vision for logo damage assessments - measuring logos to be replaced due to wear and tear in aircrafts
  • Vision for Agriculture - Duplicate vendor detection and alert

Startups pitch and failed attempts. Vision doesn't work alone. It has to be a combination of vision + data to be a successful product.

Keep Thinking!!!

June 12, 2021

AL / ML Work - The three categories to focus on

Research perspective

  • Apply new approaches + solve new problems
  • Differentiation in terms of approach / performance / patenting / renewed opportunities

Practitioners

  • Apply ML Use cases in current projects
  • Bring in the project insights/awareness within the Project team
  • Differentiation in terms of moving towards ML adoption/implementation

Business Perspective

  • Contribute to AI-driven business lens focused use cases in the domain
  • Build industry focused generic solutions 
  • Differentiation in terms of wider community impact / collaboration with business / clients
Mastering the latest tech vs Coding vs ML ops vs Building domain knowledge, Scale as much as you can, or narrow down and pick your battles!!!

Keep Thinking!!!