"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;
Showing posts with label DataScience. Show all posts
Showing posts with label DataScience. Show all posts

November 25, 2024

🚀 Navigating the Complex Landscape of AI Adoption in Business 🚀

In the rapidly evolving world of artificial intelligence, businesses face a multifaceted challenge when it comes to AI adoption. The decision to build or buy, to hire directly or outsource, and to choose the right use cases are critical and can significantly impact the success of AI integration within any organization.

🔍 Key Considerations:

1. Cloud Partnerships: Aligning with a cloud provider can dictate the models and technologies available to you. It's essential to leverage these partnerships effectively to maximize your AI capabilities.   

2. Use Case and Data Availability: Choosing the right use case is just the beginning. The availability and adequacy of data for model training or fine-tuning are paramount. Without sufficient data, even the most promising AI projects can falter.

3. Model Development Timeline: Whether it's benchmarking, extended testing cycles, or A/B testing, understanding the time required to develop and refine AI models is crucial for planning and execution.

4. Costs and Talent: The infrastructure and talent costs can often lead businesses to outsource AI and machine learning tasks. However, this brings its own set of challenges and dependencies.

5. Accuracy and Maintenance: Developing AI models that not only perform well initially but also maintain high accuracy over time requires continuous updates and skilled personnel.

6. Ethical AI: Adopting AI responsibly ensures that the technology not only serves the business goals but also aligns with broader ethical standards.

🌟 Solution Spotlight:

Innovative solutions like vector search, keyword search, semantic search, or rule-based search can address specific needs, but success fundamentally depends on the right blend of talent, technology, and timing.

As we continue to embrace AI, let's discuss how we can overcome these challenges through innovative strategies and collaborative efforts. How is your organization navigating these complexities in AI adoption? 

Share your insights! 

#AI #BusinessStrategy #Innovation #DataScience #CloudComputing #EthicalAI

April 18, 2024

Data Science & Data

Every project is a learning experience. Data science is based on "Data". Working with no data, less data, or encrypted domain knowledge with minimal data has been challenge over the past 4 years. Yet, even when data is plentiful, there remains a balancing act between leveraging it effectively and mitigating trust issues, as collaboration can sometimes be overshadowed by the scramble for credit. Everyone wants to work on a model, not on data, the old google paper still comes into their eyes :). The current trend is to train large language models (LLMs) on uniform datasets, yet this approach glosses over an important truth: no dataset can capture the full spectrum of reality. Issues such as digital poverty, underrepresentation, and inherent biases are embedded within the data we collect. Without addressing these challenges, solutions can be superficial and short-lived. Moving fast with a lot of guardrails is essentially a band-aid, not a solution. Take a step back and balance data vs model. Build something that lasts forever not for paychecks!!!

Keep Thinking!!!


March 23, 2024

How to get correct in the First Attempt with AI

Experience in AI = Ability to ask the right questions even if you don't have answers and provide AI awareness, complexity, ROI, and helping them manage costs vs Selling vision + charging $$$$ hefty for all types of costs build/buy/explore/expand. 

Build targeted products :)

Keep Exploring!!!

March 22, 2024

Failures in AI/ML/GenAI Adoption

Success in #AI/ML/GenAI projects has a lot of challenges. Some projects' data availability / some projects handling bias / Some projects balance features vs bugs / Knowing 80% features vs 20% future releases. This needs a lot of iteration and team mix to make it work. Success goes in LinkedIn posts. Failures end up haunting us searching for the next success.

Keep Exploring!!!

February 24, 2023

ML - Model Shipping Factories

5 years ago, #startups were in areas/segments 

  • AI-Driven Sales -Forecast, Recommendations (Data)
  • Chatbots - (NLP, Data)
  • Autopilot ADAS - Vision - Image, Video, Data
  • BPO - Customer support - NLP, Data, OCR
Companies that manufacture large models (Ref)


Vision / Image



Text




Audio



With GAN models, everything converges. The problem is already solved with LargeLanguageModels. Evolve to the next level or perish. #GAN #AI #startups. 

Keep Exploring!!!

February 02, 2023

FUTURE of AGRICULTURE

FUTURE of AGRICULTURE

Quick Summary

  • Data Related Usecases - #Forecast yield, water/nitrogen needs
  • #Vision Related - Predict disease, patterns/weather info
  • Mix of Skills - Agriscience, Data, AI, Transfer Learning, Customizing to new markets / similar
  • Recommendations - Derive insights/spot risks plus options to mitigate / prescriptive options
  • Model climate adaptation changing weather to recommend suitable crops
  • Impact - 13 countries, 500 crops, 10k varieties
  • Interesting - Model to predict a new variety of crop yield
  • Intelligent cloud for sustainable agriculture.
  • Using Images, Data, Text, historical data everything to build intelligence

Detailed Insights

  • Massive dataset / Best position to build knowledge graph
  • Apps for Farmers
  • Scaling digital solution to 500 crops, 10k varieties
  • AI models for 22 commodities, 13 countries
  • Provide Data and Infra
  • Breadth and Depth in Data from data collected
  • Challenging applying model in unseen regions
  • General AI capability and Knowledge Graph
  • Wheat in India vs Nigeria vs Canada
  • Iterating on Capability
  • Agriscience, Capability Science
  • 13 countries, 2.2 billion hectare scale
  • Country scale error predictions Nigeria (Wheat prediction)
  • Model predicted disease options
  • Modell hypertuned for different farms / conditions / transfer learning
  • predict yeild model in crops
  • water / nitrogen update
  • predict disease / optimize water
  • Model an asset
  • Sustainability score
  • Predict disease - model climate side of risk
  • Spread and breadth of dataset
  • Domain knowledge Vision, Agriscience
  • Models to detect Cloud Detection vs Cloud Shadow
  • Model climate adaptation, risks, recommendations
  • Predict with minimum dataset
  • Model to predict new variety of crop yeild
  • Partnering with cloud providers
  • Cloud for intelligent agriculture
  • Partner with industry and solve problems
  • Apps / Platform

Ref - Link

Knowledge graphs can incorporate both structured (for example, coming from a spreadsheet, or precision agriculture equipment) and unstructured data (a twitter feed, images, YouTube video, bulletin board information, books etc.) Knowledge graphs can be successful and valuable if they can uncover new insights by automatically incorporating new data sources, understanding the context, finding new connections, and continuously evolving and learning.

Keep Exploring!!!

Data Science Interview Questions

For hiring 2-5yrs exp, Some basic and intermediate questions

  1.  What experience do you have in developing deep-learning models for computer vision applications?
  2. What challenges have you encountered while working with deep learning models?
  3. How have you optimized deep learning models for speed and accuracy?
  4. What techniques have you used to improve the accuracy of deep learning models?
  5. What experience do you have in deploying deep learning models in production?
  6. How have you evaluated the performance of deep learning models?
  7. How have you handled data imbalance in deep learning models?
  8. What experience do you have in developing transfer learning models?
  9. How have you used reinforcement learning in deep learning models?
  10. What experience do you have in developing generative models?
  11. How would you design a deep-learning model to detect objects in a video stream?
  12. How would you design a deep-learning model to classify images?
  13. How would you design a deep learning model to detect anomalies in time-series data?
  14. How would you design a deep-learning model to generate text?
  15. How would you design a deep-learning model to identify faces in an image?
  16. How would you design a deep-learning model to detect fraud in financial transactions?
  17. How would you design a deep-learning model to predict stock prices?
  18. How would you design a deep learning model to detect objects in a 3D environment?
  19. How would you design a deep learning model to detect anomalies in a network?
  20. How would you design a deep-learning model to generate music?
Keep Exploring!!!


January 30, 2023

Multi-Channel Analytics

Multi-Channel Analytics, Patient Pathways, Omni-Channel Segmentation, Territory Design, Customer Targeting, Attribution Modeling, & Predictive Commercial Mix

Multi-Channel Analytics key use cases and features to use

Multi-Channel Analytics can be used to improve customer engagement and understanding across channels, including digital, social, and mobile. Some key use cases for Multi-Channel Analytics include:

1. Understanding customer sentiment across channels - Omni-Channel Segmentation key features

  • Segmentation of users into homogeneous groups based on their behavior or interests
  • Creation of target audiences for specific products or services
  • Optimization of marketing campaigns and content to reach the right audience

2. Identifying customer needs and preferences across channels

Top ways to Identify customer needs and preferences across channels

There are a number of ways to identify customer needs and preferences across channels. Some of the most common methods include customer surveys, focus groups, and interviews.

3. Measuring customer engagement across channels

4. Generating customer insights from multiple channels

5. What are top customer insights from multiple channels?

There are a number of customer insights that can be gleaned from multiple channels, including social media, email, and customer service. Some of the key insights that can be gleaned from social media include customer sentiment, customer feedback, and customer demographics. Email can provide insights into customer engagement, customer service issues, and customer buying behavior. Customer service can also provide insights into customer sentiment, customer service issues, and customer buying behavior.

6. Optimizing customer experiences across channels

There are a few key things that you can do to optimize customer experiences across channels:

  • Use customer feedback to improve your product - Use customer feedback to improve your product. This can be done through surveys, focus groups, or customer feedback forums.
  • Use social media to connect with customers - Use social media to connect with customers. This can be done through Facebook, Twitter, and other social media platforms.
  • Use customer service to resolve issues - Use customer service to resolve issues. This can be done through phone, email, or live chat.
  • Use marketing to attract new customers - Use marketing to attract new customers. This can be done through advertising, social media, and other marketing channels.

Keep Exploring!!!

January 27, 2023

GPT to learn some more ML optimisation / causal inference

What is numerical optimisation, How it is different from machine learning. List some usecases problems in numerical optimisation ?

Numerical optimization is the process of finding the values of input parameters that minimize or maximize a given objective function. It is different from machine learning in that it does not involve training a model with data, but instead relies on mathematical algorithms to find the best solution.

Some use cases of numerical optimization include:

  • Portfolio optimization
  • Supply chain optimization
  • Vehicle routing
  • Facility location
  • Network design
  • Scheduling
  • Parameter estimation
  • Image registration
  • Model calibration
  • Optimization of control systems

What are top 10  python packages for numerical optimization

  • NumPy 
  • SciPy 
  • Scikit-Optimize 
  • PyOpt 
  • PyGMO 
  • CVXOPT 
  • Pulp 
  • Pyomo 
  • DEAP 
  • Heuristic Optimization Algorithms (HOALib)

List some constraint for windmill turbine based on temperature, sunlight, humidity, windspeed with synthetic data python code pulp optimization solution

constraints = [

    # Temperature

    pulp.lpSum([temp_var[i] for i in range(len(temp_var))]) <= max_temp,

    pulp.lpSum([temp_var[i] for i in range(len(temp_var))]) >= min_temp,

    # Sunlight

    pulp.lpSum([sunlight_var[i] for i in range(len(sunlight_var))]) <= max_sunlight,

    pulp.lpSum([sunlight_var[i] for i in range(len(sunlight_var))]) >= min_sunlight,

    # Humidity

    pulp.lpSum([humidity_var[i] for i in range(len(humidity_var))]) <= max_humidity,

    pulp.lpSum([humidity_var[i] for i in range(len(humidity_var))]) >= min_humidity,

    # Windspeed

    pulp.lpSum([windspeed_var[i] for i in range(len(windspeed_var))]) <= max_windspeed,

    pulp.lpSum([windspeed_var[i] for i in range(len(windspeed_var))]) >= min_windspeed

]

List some usecases problems in operations research ?

  • Vehicle Routing Problem
  • Scheduling Problem
  • Network Flow Problem
  • Knapsack Problem
  • Integer Programming Problem
  • Queuing Theory Problem
  •  Linear Programming Problem
  •  Resource Allocation Problem
  •  Facility Location Problem
  •  Project Scheduling Problem

List some usecases problems in causal inference ?

  • Estimating the effect of a treatment on an outcome of interest
  • Estimating the effect of a policy change on a population
  • Estimating the effect of a marketing campaign on sales
  • Estimating the effect of a new product on customer retention
  • Estimating the effect of a change in pricing on customer demand
  • Estimating the effect of a change in customer service on customer satisfaction
  • Estimating the effect of a change in advertising on brand awareness
  • Estimating the effect of a change in the labor market on wages
  • Estimating the effect of a change in government regulations on economic growth
  • Estimating the effect of a change in the environment on public health

What is casual inference in layman terms

Casual inference is a type of reasoning that allows us to draw conclusions based on our observations. In layman's terms, it is the process of making assumptions or guesses about something based on what we know or have seen. For example, if you see someone walking down the street wearing a suit and carrying a briefcase, you can infer that they are likely on their way to work.

List some usecases problems in Bayesian modelling ?

  • Estimating the probability of a medical diagnosis given a set of symptoms.
  • Predicting the probability of a customer purchasing a product given their browsing history.
  • Estimating the probability of a machine failure given a set of sensor readings.
  • Predicting the probability of a customer defaulting on a loan given their financial history.
  • Estimating the probability of a terrorist attack given a set of intelligence data.
  • Predicting the probability of a stock price movement given market data.
  • Estimating the probability of a natural disaster given weather patterns.
  • Predicting the probability of a customer churning given their usage data.

What is difference between causal interference vs AB testing

Causal inference is a method of determining the cause-and-effect relationship between two variables. It is used to identify the cause of a particular outcome. It is based on the assumption that there is a causal relationship between the two variables.

AB testing is a method of comparing two versions of a product or service to determine which one performs better. It is used to determine which version of a product or service is more effective. It is based on the assumption that the two versions are statistically different.

List some usecases problems in differentiable programming ?

  • Optimizing hyperparameters in neural networks
  • Automatically generating neural network architectures
  • Automatically generating code from a given specification
  • Automatically tuning the parameters of a given model
  • Automatically generating data pre-processing pipelines
  • Automatically generating feature engineering pipelines
  • Automatically generating loss functions
  • Automatically generating optimization algorithms
  • Automatically generating regularization techniques
  • Automatically generating distributed training strategies

Bayesian modeling is generative probability based on historical data, Is it correct ?

Yes, that is correct. Bayesian modeling is a type of generative probability model that uses historical data to make predictions about future events.

When can we use differentiable programming , Does it have advantage over deep learning

Differentiable programming can be used when a problem requires the use of a combination of symbolic and numerical computation. It has the advantage of being able to combine the flexibility of symbolic computation with the power of deep learning. This allows for more complex models to be created and trained, which can lead to better results. Additionally, differentiable programming can be used to create models that are more interpretable and easier to debug.

Keep Exploring!!!

January 07, 2023

Models at Different Stages of Customer Lifecycle

 


Stage I: Reach
Reach is the initial stage of the buyer’s journey. The key is to attract more visitors and provide an engaging experience that will lead to a purchase.

1. Smart Content Curation 
2. Programmatic Media Buying
3. AI Generated Content 
4. Voice Search 

Stage II: Act
The second stage of the consumer journey is intended to draw the consumer in and to make them aware of your products and services.

5. Propensity Modeling 
6. Ad Targeting 
7. Predictive Analytics 
8. Lead Scoring 

Stage III: Convert
This is the important stage of moving a consumer from being an interested prospect to being a customer or member.

9. Dynamic Pricing 
10. Re-Targeting 
11. Web & App Personalization 
12. Chatbots 

Stage IV: Engage
Once a purchase is made, it is important to continuously build engagement and loyalty with the intention of expanding the relationship and potentially generating referral business.

13. Predictive Customer Service 
14. Marketing Automation 
15. 1:1 Dynamic Emails 

Keep Exploring!!!

January 04, 2023

Delivery Optimizaton at Amazon

  • Local optimizations were tackled first
  • Stop consolidation — a chain of addresses linked by a small road segment
  • Same-day delivery consolidation

Strategies

  • Customer Order and Network Density OptimizeR (CONDOR).
  • Simultaneously determining how orders are split into shipments and the sourcing fulfillment center for each shipment.
  • Determine the right tradeoff between the levels of complexity and optimality.
  • Single warehouse may assemble the order if it has all the items; if not, the order can be split into multiple shipments.

Ref Link 

Keep Exploring!!!

January 02, 2023

December 31, 2022

Building Vision Models - Myth vs Reality

Challenge / Perception - Customer feels we need millions of images 

Reality - We do not need to wait for perfect data, Data collection, or Synthetic data creation everything is a going process

Challenge / Perception  - Data collection is effortless, It can be done by google search / kaggle

Reality - The real world and the kaggle dataset are miles apart. Real-world challenges are dependent on light/angle/hardware used. Buying data is even more expensive :). Data cost is more costly than model training time

Challenge / Perception  - I need the start of the art model with 99% accuracy / Can we get a performance like the state of the art? 

Reality - We need to be realistic with the data we have, and an incremental model that we can develop. 

Challenge / Perception - Model development is a one-time effort. Collect / Build / Deploy / Move on 

Reality - Base model / retrain / field test and next version is incremental effort. ML is an iterative incremental effort. It has a set of parallel ongoing efforts like below


When the customer wants state of art but has no strategy on how they need to incrementally build upon becomes an effective challenge to provide the vision/clarity.

The building is easy sometimes vs Selling is hard many times. 

Keep Thinking!!!


December 29, 2022

Computer Vision Landscape 2022 Report

Why Computer Vision?


Hardware used
  • High-resolution types
  • Compact types
  • High-speed types
  • Standard types



Illumination techniques

  • Backlighting - Detect the presence or absence of gaps or holes, identify bubbles, cracks, or scratches
  • Dark field lighting - Features like edges, scratches, or notches
  • Bar lighting
  • Dome lighting

Key Works


Change Detection Techniques
  • Frame differencing.
  • Background subtraction.
  • Motion segmentation.

Tracking Techniques

  • ByteTrack
  • Simple Online And Realtime Tracking (SORT)
  • DeepSORT
  • TransMOT
  • FairMOT

From Post

Keep Exploring!!!


December 28, 2022

Vision for Product Catalog Image Correction

Do you know for each category near / far view has an impact on click-to-view conversion

Challenge - How do we know similar products?

Solution - Build vectors of text for each category/features/images. measuring distances between vector representations of products in an embedding space. Features for image vector - product image and the product title

Challenge - How do you change/customize catalog?

Solution - The title provides context that helps the model focus on the relevant regions of the image. Based on it remove the background / zoom  it


Challenge - How does the solution look like? How many models?

Solution -  Global network takes the whole image as input and based on the product title, it determines which portion of the image to focus on. That information is used to crop the input image, and the cropped image passes to the local branch.


From Ref - Using computer vision to weed out product catalogue errors

Keep Exploring!!!

Solving #Vision Problem - #Amazon Way - Object Detection

To detect objects in Conveyer, How does Amazon Solve it?

Challenge - Field Of View, Dataset, and Occlusion is always a challenge to detecting objects

Solution - The first step was simply to take pictures of products as they moved along conveyor belts in fulfillment centers, building up a library of images. This gives a consistent environment

Challenge - Different speeds of products moved in the Conveyer belt?

Solution - On a conveyor belt, the lighting and the speed of the item are relatively controlled and constant.

Challenge - What features we can leverage to match?

Solution - Product Dimension, Visual features, text

Summary from post

#Datascience #production #computervision 

Sometimes a combination of techniques, simplifying/restricting the target environment to a certain position/view/conditions is good enough to get a good first-cut working solution.

Keep Exploring!!!

December 26, 2022

Five things About GPT

Five Things about GPT

#1. GPT-3, or the third generation Generative Pre-trained Transformer, is a neural network machine learning model trained using internet data to generate any type of text

ChatGPT is a chatbot technology developed by 𝐎𝐩𝐞𝐧𝐀𝐈. It is designed to assist with a variety of tasks and functions, including answering questions, providing information, and completing tasks. 

#2. Number of layers and parameters



#3. Parameters

  • GPT-2 was released in February 2019 with 1.5 billion parameters 
  • GPT-3 was released in June 2020 with 175 billion parameters (~120x improvement)
  • GPT-4 will be released soon and is expected to have 100 trillion parameters (~500x improvement)
#4. How intelligent can it be?


#5. GPT-4 is built on the Transformer architecture, which has been effective for a variety of machine-learning tasks, including computer vision. This means that GPT-4 might be used for tasks such as image and video generation

#6. Training Approach


Keep Exploring!!!

December 19, 2022

Vision In Manufacturing

 Another interesting use cases from link

Key Features

  • Targeted for visual inspection tasks in manufacturing environments
  • Docker container
  • Segmentation and localization
  • Active learning - identifies suspect examples quickly for human review and labeling
  • Auto-align images from camera streams


Visual Auto Inspection - With human in loop will be robust :)

Keep Exploring!!!



November 25, 2022

AI Usecases - Oil and Gas

The predictive model is trained to identify parts that are likely to fail so that

  • Plan for preventive maintenance
  • Avoid well downtime
  • Make sure replacement parts are available else order them in advance
  • Identify failure trends
  • Geo-locate failed wells


Keep Exploring!!!



AI Usecases - Telecom

  • Gender based churn
  • Senior Citizen Distribution
  • Partner Distribution
  • Dependent Distribution
  • Phone Service Distribution
  • Multiple Lines Distribution in Customer Attrition
  • Internet Service Distribution in Customer Attrition

AI Telecom Usecases.

Keep Exploring!!!