The Uniqueness of Customer Data and the Importance of Automated Cleanup

In the digital age, managing customer data can be a daunting task. Here are some points to consider:

• Each customer's data will be unique.
• Each customer's data will encounter parsing issues.
• Each customer's data will disrupt pipelines.

No data is pristine; changes should not be spontaneous. When dealing with data, especially on a large scale, it's imperative to have a robust process in place. Having a process to automate cleanup is essential to scaling your solution.

Keep exploring!

Data Privacy vs Cheap value vs Compromises

How your data is bought at Low costs and leveraged to build ML Models !!!

Keep Exploring!!!

Why big companies failed to build LLMs

 

How Alexa dropped the ball on being the top conversational system on the planet
—
A few weeks ago OpenAI released GPT-4o ushering in a new standard for multimodal, conversational experiences with sophisticated reasoning capabilities.

Several days later, my good friends at PolyAI…

— Mihail Eric (@mihail_eric) June 11, 2024

• Data was poorly annotated. Documentation was either nonexistent or stale.
• Experiments had to be run in resource-limited compute environments. 
• That meant for months our internal annotation team had been mislabeling thousands of data points every single day
• antagonistic mid-managers that had little interest in collaborating for the greater good
• Duplicated efforts due to no shared common ground 

Keep Exploring!!!

Data Science & Data

Every project is a learning experience. Data science is based on "Data". Working with no data, less data, or encrypted domain knowledge with minimal data has been challenge over the past 4 years. Yet, even when data is plentiful, there remains a balancing act between leveraging it effectively and mitigating trust issues, as collaboration can sometimes be overshadowed by the scramble for credit. Everyone wants to work on a model, not on data, the old google paper still comes into their eyes :). The current trend is to train large language models (LLMs) on uniform datasets, yet this approach glosses over an important truth: no dataset can capture the full spectrum of reality. Issues such as digital poverty, underrepresentation, and inherent biases are embedded within the data we collect. Without addressing these challenges, solutions can be superficial and short-lived. Moving fast with a lot of guardrails is essentially a band-aid, not a solution. Take a step back and balance data vs model. Build something that lasts forever not for paychecks!!!

Keep Thinking!!!

Data - Use Case - Iterative Thinking - Evolving solutions

Many times learning comes from people around us. For vehicle PPF it gave a lot of insights

• While removing all the door cladding/mirrors, the 0-2yrs exp people were putting all screws together
• While fitting on screws, Same dimension screws were applied for two parts, There was confusion on pending screws
• The PPF person was very focused, and the sun film person was a separate person

Three things to build a solution

Many times learning comes from people around us. For vehicle PPF it gave a lot of insights

Observation #1

• Fresher Lens -  While removing all the door cladding/mirrors, the 0-2yrs exp people were putting all screws together
• Experience Lens  - The person (lead) he asked them group based on the parts
• Lesson - group problems, data logically to debug / build solution

Observation #2

• Fresher Lens - While fitting on screws, Same dimension screws were applied for two parts One screw was perfect silver, and another was perfect black, When all screws were filled, juniors were clueless about where it had to be fitted
• Experience Lens  - The lead was able to provide clarity to fit the connecting dots
• Lesson - Leadership is solving with what you have, not afraid to look back and rework where it is needed

Observation #3

• The PPF person was very focused, the sun film person was a separate person
• Lesson - Build relationships, cannot solve all problems all alone
• Be good at a few, use expertise when you need a great product

What does it imply in the AIML context

• Inexperienced team members initially consolidated all components indiscriminately during disassembly. The team leader guided them to categorize components systematically, akin to structuring data for efficient algorithmic problem-solving.
• Different screws of identical size were used interchangeably during the assembly process, leading to confusion. The leader demonstrated critical thinking, using available resources to retrospectively address the issue, a trait essential for refining machine learning models.
• Task specialization was evident; individuals focused on PPF or sun film application roles. This mirrors the need for specialization and collaboration in AIML, leveraging cross-disciplinary expertise to enhance overall model performance.

Keep Exploring!!!

Buying Decision - Data Analysis

This could be biased but when you have limited budget and have to take a convincing decision :)

 

Keep Exploring!!!

What is difference between Experience vs Expertise

To differentiate we need to understand below bias in our decisions, perspectives, views

Confirmation Bias - This refers to interpreting new information in a way that confirms our pre-existing beliefs. For instance, if you have knowledge of SQL Server, you may believe that the index storage patterns in NoSQL are similar. While there can be similarities, it's crucial to be aware of the differences as well.

Misconceptions of Skills - Often, we mistake awareness of technology as expertise. Merely being able to compile and produce output doesn't necessarily mean understanding how it works or its intricate mechanics.

Halo Effect - This is when you either completely like or dislike everything about a person or thing, with no middle ground. Judging a technology without a thorough understanding of it is an example of this effect. Intellectual humility - Acknowledging what you don't know is the drawing of wisdom

WYSIATI (What You See Is All There Is) - Here, you cannot consider what you do not know. It's about having a balanced view versus a mindset of 'I know it.' 

Consequences of this

• We have a larger number of data scientists who possess a basic understanding of a variety of domains, rather than deep expertise in a few domains. 
• Moreover, we have more data scientists who are algorithm-oriented compared to those who can skillfully blend algorithms, data, and common sense. 
• The absence of awareness in domains and data will only result in solutions similar to those found on platforms like Kaggle, which don't necessarily meet business requirements.
• Preparation - No amount of preparation is enough to face the customer, Every possible scenario needs to be tested. The only thing that matters is the correctness of your analysis

Keep Thinking!!!

Business of ChatGPT

• LLM which can collect more data and the relevant answer will lead the race
• Data is king, relevant information is key
• Creating very special is they key, crawling open web vs limited access to information
• Data race - who owns, who shares, who collaborates

Profits vs Morality. In terms of money vs having a good quality society - Attend important things in life - Link 

Keep Exploring!!!

Data understanding of Database Developer vs Data Science: Understanding the Different Perspectives on Values

Database developers and data scientists have different ways of looking at values. 

For a database developer, values can be seen as a vegetarian, young family, and budget buyer. 

For a data scientist, the mix of values that have something in common is vegetarian + young family or young family + budget buyer. 

To understand the behavior of these two perspectives, it is important to look at the story of the perspectives change between database developers and data scientists.

Database developers look at values from a more traditional perspective, focusing on individual values and how they can be used to create a database. They look at individual values and how they can be used to create a database that is efficient and effective. They also look at how the values can be used to create relationships between different pieces of data.

Data scientists, on the other hand, look at values from a more holistic perspective. They look at the mix of values that have something in common and how they can be used to create a more comprehensive understanding of the data. They look at how the values can be used to create relationships between different pieces of data, as well as how the values can be used to create a more comprehensive understanding of the data.

Keep Exploring!!!

How all Web Analytics data is stored ?

Google Analytics data is stored in a proprietary database and is not accessible to query as a SQL query.

Salesforce Analytics data is stored in a proprietary database called Wave. It is not accessible to query as a SQL query. Instead, it is accessible through the Salesforce Analytics Query Language (SAQL) which is a specialized language for querying and manipulating data stored in Wave.

Tealium analytics data is typically stored in a NoSQL database, such as MongoDB or Cassandra. It is not accessible to query as a SQL query. However, Tealium does provide an API that allows you to access the data programmatically.

Keep Exploring!!!

Interesting Data Strategy - Group By Business / Entity

Interesting Key points 

• Empowering domain experts. In specialized fields, experts should be brought on board.
• Centralized ETL Fails - fail to lay the foundation for current and future use cases that will create value.
• Data Products - Grouping data by Logical business Entities works better to solve use cases end to end

            Ref - Link

• Startups are evolving in consolidating across variety of sources, Ref - Link

Ref - Link

Keep Exploring!!!

Supply Chain Tower - Basics

What is a supply chain tower in retail?

A supply chain tower in retail is a set of integrated processes that enable retailers to manage the flow of goods from the suppliers to the customers. This includes the procurement, storage, distribution, and delivery of products. It also involves the management of inventory, pricing, and customer service. The goal of the supply chain tower is to ensure that the right products are delivered to the right customers at the right time and cost.

What are key components of supply chain tower

• Demand Planning: This involves forecasting and anticipating customer demand, and setting production and inventory levels accordingly.
• Procurement: This involves sourcing, negotiating, and purchasing the necessary materials and components for production.
• Production: This involves the actual manufacturing of the product, and may involve multiple steps and processes.
• Inventory Management: This involves tracking and managing inventory levels, and ensuring that the right products are available when needed.
• Logistics: This involves the transportation and distribution of the finished product to the customer.
• Customer Service: This involves providing customer support and managing customer relationships.

What are different products available for supply chain tower offering?

• Supply Chain Planning: This includes solutions for demand planning, supply planning, inventory optimization, and production planning.
• Supply Chain Execution: This includes solutions for order management, warehouse management, transportation management, and global trade management.
• Supply Chain Analytics: This includes solutions for analytics, business intelligence, and reporting.
• Supply Chain Collaboration: This includes solutions for supplier and customer collaboration, supplier performance management, and product lifecycle management.
• Supply Chain Visibility: This includes solutions for real-time visibility, event management, and tracking.
• Supply Chain Optimization: This includes solutions for supply chain optimization, process automation, and optimization.

What are different commercial products available for supply chain tower offering?

• Oracle Supply Chain Management Cloud: Oracle Supply Chain Management Cloud is a comprehensive suite of cloud-based applications that enable organizations to manage their entire supply chain from end-to-end. It includes modules for demand planning, order management, inventory optimization, and transportation management.
• SAP Ariba Supply Chain Tower: SAP Ariba Supply Chain Tower is a cloud-based platform that helps organizations manage their supply chain processes from end-to-end. It includes modules for demand planning, supply chain analytics, and supplier management.
• JDA Supply Chain Tower: JDA Supply Chain Tower is a comprehensive suite of cloud-based applications that enable organizations to optimize their supply chain operations. It includes modules for demand forecasting, inventory optimization, and transportation management.
• Infor Supply Chain Tower: Infor Supply Chain Tower is a cloud-based platform that helps organizations manage their supply chain processes from end-to-end. It includes modules for demand planning, supply chain analytics, and supplier management.
• Kinaxis RapidResponse: Kinaxis RapidResponse is a cloud-based supply chain management platform that enables organizations to manage their entire supply chain from end-to-end. It includes modules for demand planning, supply chain analytics, and inventory optimization.

What it takes to build supply chain tower open source 

• Develop a clear vision and strategy: The first step in building a supply chain tower open source is to develop a clear vision and strategy for the project. This should include a detailed plan for how the project will be developed, what type of technology will be used, and how the project will be maintained and supported.
• Gather resources: Once the vision and strategy have been established, the next step is to gather the necessary resources to build the supply chain tower open source. This includes gathering the necessary personnel, software, hardware, and other resources needed to develop the project.
• Design the architecture: The third step is to design the architecture of the supply chain tower open source. This includes deciding on the type of technology to be used, the architecture of the software, and the overall structure of the project.
• Develop the software: Once the architecture has been designed, the next step is to develop the software. This includes writing code, testing the software, and ensuring that the software meets the requirements of the project.
• Test the software: Once the software has been developed, the next step is to test the software to ensure that it meets the requirements of the project. This includes running tests, debugging the software, and ensuring that the software is secure and reliable.
• Deploy the software: Once the software has been tested, the next step is to deploy the software. This includes setting up the necessary infrastructure, deploying the software to the production environment, and ensuring that the software is available to users.
• Maintain and support the software: The final step is to maintain and support the software. This includes providing bug fixes and updates, responding to user feedback, and ensuring that the software is secure and reliable.

What is 09 supply chain offering and who are its competitors

09 Supply Chain is an online marketplace for businesses to find and purchase services related to logistics, transportation, and supply chain management. It provides a platform for businesses to compare prices, find the best deals, and connect with logistics providers. Its competitors include Freightos, Flexport, and Shipa Freight.

Top 10 ways to have real time inventory

• Automate inventory tracking with barcode scanners.
• Implement a cloud-based inventory management system.
• Utilize radio-frequency identification (RFID) technology.
• Utilize mobile devices for inventory tracking.
• Utilize inventory management software.
• Utilize automatic data collection (ADC) systems.
• Utilize point-of-sale (POS) systems.
• Utilize cycle counting.
• Utilize automated replenishment systems.
• Utilize automated inventory tracking systems.

Top 10 ways to have real time replienshment planning

• Utilize real-time inventory management systems.
• Analyze customer demand patterns.
• Establish a reorder point.
• Develop a replenishment strategy.
• Automate the replenishment process.
• Monitor supplier performance.
• Utilize forecasting models.
• Utilize safety stock.
• Utilize vendor-managed inventory.
• Monitor inventory levels.

Compare features between Freightos, Flexport, and Shipa Freight, 09 Supply Chain

Freightos:

• Instant online freight quotes and bookings
• Access to global carriers and freight forwarders
• Real-time tracking and visibility
• Automated document generation
• Integrated payments
• Customs clearance
• Insurance
• Freight audit and payment
• Data analytics

Flexport:

• Global freight forwarding
• Customs brokerage
• Warehousing and distribution
• Real-time tracking and visibility
• Data analytics
• Automated document generation
• Integrated payments
• Insurance
• Freight audit and payment

Shipa Freight:

• Global freight forwarding
• Instant online freight quotes and bookings
• Real-time tracking and visibility
• Automated document generation
• Integrated payments
• Customs clearance
• Insurance
• Freight audit and payment
• Data analytics

Keep Exploring!!!

Experience - Exposure - Domain - Data - Diversity Thinking

• My First year - Windows98 Testing
• Second year - C/C++ Mq Adapter Design
• Third year - Application support / migration / ITSM / Deployment / Production Support
• Fourth Year - SQL Migration / Performance Testing
• Fifth Year - Biztalk / SQL Migration / Replication / Developer
• Sixth Year - OLTP SQL Developer 
• Seventh Year - BI + OLTP Developer / Warranty migration 220 million records (Supply chain 4th year :))
• Eighth Year - QA Manager / Prod support / On-call / People Management
• 9th Year - DB / QA Developer / Product Transition
• 10th Year - Forecasting Feature in Product / DB / QA / Automation
• 11th Year - Hardware Integration / Professional Services / SQL Developer
• 12th Year - AWS Scaling / Perf testing on horizontal scaling for Tag counting / Function + Perf / DB
• 13th Year - SQL 2016 Migration Analysis / Big Data Architecture Analysis
• 14th Year - AI / ML - PG / DB Dev / BI
• 15th Year - AI / ML - PG second year / DB Dev / BI / Freelancing
• 16th - AI Freelancing / Training / DB / BI Dev / Patents
• 17th  -  AI Freelancing / Training / DB / BI Dev / Patents
• 18th - Vision / Forecasting / Recommendations  / People Management / Training
• 19th - Vision / Forecasting / Recommendations  / People Management / Training

All the different parts of experiences sum up and help me in knowing the Data journey / AI journey/business challenges 

It was varied roles / multiple domains and products but everything was worth it :)

Thanks to all my previous companies/training providers / past employers/freelance offered by startups...

Multiple lenses - Development - QA - Automation - Performance - Support - BI - Database performance tuning - Computer vision - Forecasting - Recommendations - Many times I have influenced / implemented product key features based on domain expertise.

• Warranty Migration of 220 million SKUs
• 3PL touch point integration
• Spot capabilities in Product with Data & AI (What can we do with what we have :) 

Keep Exploring!!!!

Good Read Modern Data Stack vs Realistic cost friendly working architecture

 

Keep Thinking!!!

Myth of Data

•  I don't like my job
• Data is insufficient
• There are no good data science use cases

Learn all coding questions, Practice and learn all ML maths, Solve all kaggle problems, Land your dream job, the data you will face the same problem

• Data is insufficient

What you see in learning / kaggle is not real-world data issues

Wherever you go, bad data and incomplete data only will be there. 

Keep Thinking!!!

Interesting Reads - Books H1

Some of the books I was able to review in the last 6 months. We need to revise again and again and experiment.

• Bird's eye view
• 30 K Perspective
• 20K Perspective

I am still learning. I take time to build my perspective. Experience is a mix of learning, doing, knowing, connecting with industry experts. Always be open to learning - unlearn - relearn.

Books List for future reference

• O'Reilly - A Practical Introduction to Supply Chain
• O'Reilly - Agile Conversations
• O'Reilly - AI Blueprints
• O'Reilly - Architecture Patterns with Python
• O'Reilly - Beautiful Code
• O'Reilly - Bioinformatics Programming Using Python
• O'Reilly - Breaking Out: How to Build Influence in a World of Competing Ideas
• O'Reilly - Building Evolutionary Architectures
• O'Reilly - Change Your Life with CBT
• O'Reilly - Cloud Analytics with Microsoft Azure - Second Edition
• O'Reilly - Communicate to Influence: How to Inspire Your Audience to Action
• O'Reilly - Data Governance: The Definitive Guide
• O'Reilly - Data Lake Analytics on Microsoft Azure: A Practitioner's Guide to Big Data Engineering
• O'Reilly - Design Thinking for Training and Development
• O'Reilly - Designing Data-Intensive Applications
• O'Reilly - Digital Supply Networks: Transform Your Supply Chain and Gain Competitive Advantage with Disruptive Technology and Reimagined Processes
• O'Reilly - Empathy (HBR Emotional Intelligence Series)
• O'Reilly - Exam Ref AZ-303 Microsoft Azure Architect Technologies
• O'Reilly - Fluent Python, 2nd Edition
• O'Reilly - Focus (HBR Emotional Intelligence Series)
• O'Reilly - Fundamentals of Supply Chain Theory, 2nd Edition
• O'Reilly - Graph Algorithms
• O'Reilly - Hands-On Vision and Behavior for Self-Driving Cars
• O'Reilly - How Stitch Fix uses human-in-the-loop machine learning for personalization
• O'Reilly - How to Persuade and Influence People: Powerful techniques to get your own way more often
• O'Reilly - Influence and Persuasion (HBR Emotional Intelligence Series)
• O'Reilly - Influence in Action: How to Build Your Conversational Capacity, Do Meaningful Work, and Make a Powerful Difference
• O'Reilly - Kubernetes in Action
• O'Reilly - Learning Python, 4th Edition
• O'Reilly - Linear Programming and Resource Allocation Modeling
• O'Reilly - Logistics Management
• O'Reilly - Machine Learning Design Patterns
• O'Reilly - Metaheuristics for Logistics
• O'Reilly - Nature-Inspired Optimization Algorithms
• O'Reilly - Practical Git: Confident Git Through Practice
• O'Reilly - Practical Machine Learning for Computer Vision
• O'Reilly - Practical MLOps
• O'Reilly - Practical Statistics for Data Scientists, 2nd Edition
• O'Reilly - Purpose, Meaning, and Passion (HBR Emotional Intelligence Series)
• O'Reilly - Python: Master the Art of Design Patterns
• O'Reilly - Resilience (HBR Emotional Intelligence Series)
• O'Reilly - Self-Awareness (HBR Emotional Intelligence Series)
• O'Reilly - Success in Programming: How to Gain Recognition, Power, and Influence through Personal Branding
• O'Reilly - Supply Chain and Logistics Management Made Easy: Methods and Applications for Planning, Operations, Integration, Control and Improvement, and Network Design
• O'Reilly - Supply Chain Management and its Applications in Computer Science
• O'Reilly - Supply Chain Management For Dummies
• O'Reilly - Supply Chain Optimization through Segmentation and Analytics
• O'Reilly - The Azure Cloud Native Architecture Mapbook
• O'Reilly - The Cloud-Based Demand-Driven Supply Chain
• O'Reilly - The Science of Influence: How to Get Anyone to Say "Yes" in 8 Minutes or Less!, Second Edition
• O'Reilly - Visual CBT: Using pictures to help you apply Cognitive Behaviour Therapy to change your life

Bookmarks for future reference!!!

Keep Thinking!!!

One Liners, Concepts, Slowly Changing Dimensions

SCD Summary

Sometimes one link is good enough to summarize 

• Type 1 - Overwrite previous value
• Type 2 - Add new row, Deactive old record, activate new one
• Type 3 - Add new attribute - Activation Data / Effective Date
• Type 4 - Add History Table

Docker - Docker is a tool designed to make it easier to create, deploy, and run applications by using containers

Kubernetes - Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services

Docker vs VM

• In Docker, the containers running share the host OS kernel
• A Virtual Machine, on the other hand, is not based on container technology. They are made up of user space plus kernel space of an operating system

More Reads

Kubernetes cheatsheet

Keep Simplifying Concepts!!!

Personal Datawarehouses

We need to have the ability to claim our personal data / use it to trade without PII. Everything now is paid Gmail / google photos/youtube. The end-user data is used without any benefits for End-user.

• Reclaim your Google Data
• Reclaim your Social Media Data
• Reclaim your Amazon Data
• Reclaim your Location Data

Hope there is some value for user data.

Keep Thinking!!!