Analysis of ML Landscape

 Analysis of ML Landscape - Link 

Keep Exploring!!!

Saas - software as a service to Services in SaaS Model

Hoping to see more

• VaaS - vision as a service - Pay per use - Configure - Feed data Get insights 
• DaaS - data analytics as a service - Data to insights, Configure, Explore and get insights
• IaaS - Insights as a service - Generate required KPI customized for domain
• FaaS - Forecasting as a service - Apply Algos, Tune and Pick and run best algos
• RaaS - Recommendations as a service- Apply Algos, Tune and Pick and run best algos

#Saas - software as a service - Let 2022 see more, #VaaS - vision as a service, #DaaS - data analytics as a service, #FaaS - Forecasting as a service, #RaaS - Recommendations as a service 

Vision Opportunities Products for Indian Markets - #Computervision #EdgeAnalytics #Products

Making it affordable, large scale adoption is key for success

Keep Exploring!!!

Follow your passion - Decrypt it

Follow your passion means

• Work on your interest outside work hours
• Build domain perspectives by following companies/trends
• Differentiate between your work vs your interests
• Build prototypes for your passion outside work
• Teach / Mentor / Learn / Unlearn as much as possible
• Be unique in your own ways of perspectives / ideas

It's not money/titles, Sometimes you need to carve your own path. Keep going!!!

Building Products / Career without titles

Sometimes when you work on building a product/solution

• You look at business opportunities, market landsscape, You take product manager lens
• You read/scan through options to build minimum prototype with development manager lens
• You connect with VCs for feedback / evaluate with sales lens
• Everything is time bound, You have the lens of program manager

We demonstrate multiple roles in different situations. This is all what it means experience :)

Keep Learning!!!

Blockchain Basics

#Blockchain = #cryptography powered token-based environment for #Anonymous #decentralized environment to avoid visibility of real #Beneficiaries

Paper - Blockchain Technology: Bitcoins, Cryptocurrency and Applications 

Key Notes

• Blockchain is a decentralized ledger used to securely exchange digital currency
• Distributed database that maintains continuously growing tamper-proof data structure

Algos

• The Blockchain metadata is stored in Google’s LevelDB by Bitcoin Core client
• The individual blocks are identified by a hash which is generated using secure hash algorithm (SHA-256) 
• Decentralized and no central authority has full control
• Transaction is broadcasted over Bitcoin network to inform everyone that new owner of these coins

Business Process

• Buy Bitcoins
• Trade based on bitcoins
• Avoid visibility to institutions
• Set your own values for bitcoins
• Finally based on current rates reverse back or buy something property / material
• Avoid any tracking / Value of transaction

How it may work

• Distributed databases
• Broadcasted to all nodes
• Decentralized
• Everyone has common access across ledgers  which are consistemt

Security

• Once we get into the system / datanases
• Keys to decrypt
• Every transaction may have its own hashkeys / token
• Hard to crack without getting complete details as token may play a key role to deter breaking into ledgers

Paper - Bitcoin, Currencies, and Fragility

Key Notes

• peer-to-peer version of electronic cash without going through financial institution.

Paper - Trends in Development of Databases and Blockchain

• DCS (Decentralization, Consistency, Scalability) theorem for the blockchain systems
• Consistency - Any read in the distributed system gives the latest write on the nodes.
• Availability - A Client always receives a response at any point of time irrespective of whether the read is the latest write.
• Partition Tolerance - In case of partition between nodes in the distributed system, the system should still be functioning.
• Decentralization - There is no trusted entity controlling the network, hence no single point of failure
• Consistency - The blockchain nodes will read the same data at the same time.
• Scalability - The performance of blockchain should increase with the increase in the number of peers

Blockchain and database both can achieve many functionalities and features by coping with each other.

More Reads

• Blockchain Technology Overview
• Cryptocurrencies and the Future of Money
• Editorial: Understanding Cryptocurrencies*
• Understanding the Evolution of Blockchain Ecosystems: A Longitudinal Measurement Study of
• Bitcoin, Ethereum, and EOSIO
• A Survey on Blockchain for Big Data: Approaches, Opportunities, and Future Directions
• Non-Fungible Token (NFT): Overview, Evaluation, Opportunities and Challenges

Papers - Dec30th

• Evaluation of Distributed Databases in Hybrid Clouds and Edge Computing: Energy, Bandwidth, and Storage Consumption
• Building Blocks of Sharding Blockchain Systems: Concepts, Approaches, and Open Problem 
• What is a Blockchain? A Definition to Clarify the Role of the Blockchain in the Internet of Things
• Digital Building Twins and Blockchain for Performance-Based (Smart) Contracts
• A GENERALISED LOGICAL LAYERED ARCHITECTURE FOR BLOCKCHAIN TECHNOLOGY
• A Proof of Useful Work for Artificial Intelligence on the Blockchain
• Connecting AI Learning and Blockchain Mining in 6G Systems
• Machine learning in/for blockchain: Future and challenges
• Blockchain as a Service: A Decentralized and Secure Computing Paradigm
• BAASH: Enabling Blockchain-as-a-Service on High-Performance Computing Systems
• uBaaS: A Unified Blockchain as a Service Platform
• Integration of Blockchain and Cloud of Things: Architecture, Applications and Challenges
• A Survey of Blockchain Applications in Different Domains
• A Survey on Blockchain & Cloud Integration
• DATA STORAGE IN THE DECENTRALIZED WORLD: BLOCKCHAIN AND DERIVATIVES
• BLOCKCHAINS VS. DISTRIBUTED DATABASES: DICHOTOMY AND FUSION
• Evaluation of Distributed Databases in Hybrid Clouds and Edge Computing: Energy,
• Bandwidth, and Storage Consumption

Keep Exploring!!!

Design Lessons

This link Design Primer is very useful.

I like the complete well-discussed solution, Focused on the big picture, not just syntax :)

• Design the Twitter timeline and search
• Design a web crawler
• Design the data structures for a social network
• Design Amazon's sales rank by category feature
• Design an online chat
• Design an LRU cache
• Design a call center
• Design a parking lot
• Coding Challenges

Keep Exploring!!

Retail Trends 2022 Report

Key Notes

• Data collection from devices
• Insights shared for Retail Locations in US
• Positive Trends end of year
• Positive growth in physical retail

• Traffic up since october for holiday traffic
• 2022 will see more in Home improvement category
• Heavy investment in online, supply chain technology
• Physical stores also expanding for online brands
• Touch and feel,Product Experience
• Services for better relationships / get customers back
• Healthcare is growing
• Smaller format stores (Sephora / Kohls / Target)
• Grocery did well in 2021
• Hyperlocal has picked up

Keep Exploring!!!

AR vs VR

Virtual Reality

• Virtual Reality (VR) is a computer-generated environment with scenes and objects that appear to be real
• Virtual reality (VR) is a simulated experience 
• Three-dimensional, computer-generated environment
• VR requires a headset device

Augmented Reality

• Augmented reality (AR) is an interactive experience of a real-world environment
• Augmented reality (AR) overlays digital content and information onto the physical world
• AR uses a real-world setting while VR is completely virtual
• AR can be accessed with a smartphone
• AR - Exploring Hotels in Google maps, Exploring things inside a car, Keeping things in Home and visualizing how it looks

Keep Exploring!!!

Metaverse - Lets decrypt it

Paper #1 - Virtual World, Defined from a Technological Perspective, and Applied to Video Games, Mixed Reality and the Metaverse [v-0.16]

Key Notes

• Multiple definitions ‘virtual world’ in various communities'
• worldliness’ of a virtual world
• a social networking site has persistence but no sense of synchronous environment
• “a computer-generated display that allows or compels the user (or users) to have a sense of being present in an environment other than the one they are actually in, and to interact with that environment"
• Multiuser sharing, interaction and online
• Virtual world, Virtual money, Virtual transfer Virtual (Simulated) Synthetic Environment
• Different applications and how they can simulate avatars, virtual environments, real time collaboration

• Virtual Reality / Mixed Reality

Paper #2 - Metaverse for Social Good: A University Campus Prototype

• Human-centered computing → Interactive systems and tools;
• Metaverse is a combination of “meta" (meaning beyond) and the stem “verse" from "universe"
• VR and augmented reality (AR) technologies

• Massive Multiplayer Online Video Games. Massive multiplayer online (MMO)

Paper #3 - Metaverse Shape of Your Life for Future: A bibliometric snapshot

A Survey on Metaverse: the State-of-the-art, Technologies, Applications, and Challenges

• Hyper Spatiotemporality refers to the Metaverse, a virtual world parallel to the real world.

When Creators Meet the Metaverse: A Survey on Computational Arts

All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda

Shopping in the #metaverse with $CEEK
Concept VR store presented to @hm by #CEEK Creating mainstream use cases for $CEEK + scaling #Virtualreality beyond games. #VRAPP #CEEKVR #NFT #VR #CEEKVenues
👉🏽 More at https://t.co/oAvCTgp2Bk pic.twitter.com/OI4BFkyUAw

— Ceek (@CEEK) December 7, 2021

In the future our lives will be infused by #6G-powered #MixedReality, at a very high speed (up to 200 Gbps) #AR #VR #MR #Metaverse #5G #tech #AI #ML #China #CES2022 v/ @FrRonconi @HaroldSinnott @SpirosMargaris @Nicochan33 @ipfconline1 @enilev @psb_dc @TerenceLeungSF pic.twitter.com/h3ZokyRpFY

— Dr. Marcell Vollmer #StaySafe #CES2022 Carpe Diem (@mvollmer1) January 8, 2022

These glasses let you race against Hologram
by @pascal_bornet #AI #ArtificialIntelligence #VR #AR #VirtualReality #AugmentedReality

Cc: @rethinkrobotics @davidsmith4324 @maxjcm @ronald_vanloon @xbrlstandard pic.twitter.com/hASevbCmUC

— Ronald van Loon (@Ronald_vanLoon) January 9, 2022

Keep Exploring!!!

Revisiting my predictions

 

Link for the previous post, After two years at end of 2021, I have scored decently I guess, still 3 more years to go. 

Revisiting my predictions

• Large scale deployment of Video Analytics in Scale for Home monitoring / Attendance Monitoring / Retail / Gesture-based tracking. - Vision has seen good traction wesense, inflect, tangoeye. The adoption is slow but there are startups focused on it - (1/8)
• Smart Cities Reality Check - Yes 2021 there is a lot of adoption in Tier2 cities / move towards adopting battery cars. (2/8)
• Redefined Education - This has become a good commercial market. Byjus, Vedantu, and many online educators. (3/8)
• Healthcare Analytics - Cure.ai and a few other vendors have successfully implemented vision-based diagnosis (4/8)
• More Open Source Quality Education Tools - Yea khan academy and a lot of doctors on youtube, many institutions teaching in social media (5/8)
• Drone / Quad-copters / Air Ambulance - This was a lot of adoption, many drone-based commute pilots, delivery drones (6/8)
• Battery / Electric cars vs Fuel Based - 20-30%, Ather, Revolt, Ola yes it is good. Seems to pick up (7/8)
• Big Data + Analytics - More the data, More the chaos - AL / ML adoption is good in media, ecom domain, Outside than its not mass adoption though. (8/8)

Decent predictions and on track I guess :)

Perceptron vs Neuron

Neuron

• Smallest unit of brain cell
• 86 billion neurons in our brain 
• Information captured in Retina, Passed to a projection (neurons) - Signals / Responses computed and decision taken
• Receiving information through Dendrites
• A computation or cell body does the processing
• Outgoing data is sent via Axons
• The learning activity of brain is Async

Perceptron

• Aritifical simulation of neuron
• Accepts inputs, weights and bias
• Similar to the cell body here also collect all the information
• Here information is processed with activation / decision function before passing to future layers
• The learning activity is synchronous, Forward propagation, Error Computation, Backward propagation, and then the Next loop of forward propagation

A ton of things we don't know even about human body, how our body works and each organs coordinate with each other. 100% we do not have a failover mechanism for the heart. Live with abundant optimism and hope. 

It's one life. 

Stay Kind, Make it beautiful and peaceful. 

Keep Learning :)

Know how questions ? Logical perspectives ? Non-Linearity ?

How do I understand what is activated in each layer of neural network ?

Let's take an image, For object detection, the basics are edges, corners, contours, circles. Every layer certain key properties would get highlighted and summing up all the nodes and the number of layers it would have preserved the activation spots. The output of each hidden layer would be based on activation function - (0,1)-Sigmoid, (-1,1)- Tanh, (0,X) - Relu, (.1x,x for Relu). As the results of each activation in each hidden layer contribute to get features, we do a forward propagation from N to N+10 layers. Then we do a partial derivative of

• Differentiation of Expected output with respect to Actual Output
• Differentiation of ActualOutput / Activation Function
• Differentiation of Input of N+10 Node with respect to Weights passed to it.

You do a forward propagation loop F1

Then compute partial derivative error and Backprop Fron N+10,N+9..Up to N layers. Since this is done from right to left. Last to first layers. The previous computed derivatives gets used in every layer to left.

Sigmoid could not pass/preserve weights for some cases as it's of range (0,1). Relu came for the rescue. You can see in previous posts how relu is able to meet just by half of the epochs it took sigmoid to learn.

Why not to assign all weights same ?

This is nothing but an increased proportion of inputs, This will not introduce any non-linearity. This is same as input with some multiplier ratio

Curse of Dimensionality ?

Images, Videos, Text are multi-dimensional. These cannot be handled as linear problems with separating boundaries. They are non-convex, They would have optimal solutions. We use partial derivatives in neural networks to arrive at optimal value. There could be multiple optimal values. To arrive at local minima vs global minima we take care of learning rates / partial derivatives.

Keep Exploring!!!

Activation Functions - Observations

Same experiment, different activation functions, different observations

• The number of epochs to arrive at similar accuracy - Tanh / Relu seem to perform significantly better than sigmoid. In the case of huge datasets, It makes sense to pick those activation functions which perform better and learn faster.
• Boundary types between each activation function  - Although every activation function solves it you can spot the boundary circular / boxed based on their behavior. 
• The training accuracy with 880 Epochs in Sigmoid = 90 Epochs of Tanh = 78 Epochs of Relu. Every activation function will converge but the number of epochs depends on choosing the right activation function to save compute, faster training

Keep Exploring!!!

What's the future going to be

Outside regular 9-5 jobs these jargons keep appearing in reads recent days. Hope to sit down next week and look thru this from a common man perspective

• Metaverse
• Web3
• Bitcoin
• Crypto
• Blockchain

Usually, I am anti-social media, anti-paid edutech sectors. Will need to understand and see how practically this fits in the lens of impact as a whole on the society.

Metaverse - Virtual representation of self in social media/collaboration / Digital avatars

How it works?

• Similar to gaming environments where you use Augmented reality / virtual reality environment you will be with the gadgets to live in your virtual avatar

Which crowd it may pull?

• Teens, Gamers, Kids could be the earliest targets

Which companies may invest

• Gaming, Fashion would gain most in terms of making it more consumer interactive environment

From Tech point of view we have analyzed blockchain earlier. The core of it is

• Decentralization - There is no trusted entity controlling the network, hence no single point of failure. 
• Consistency - The blockchain nodes will read the same data at the same time. 
• Scalability - The performance of blockchain should increase with the increase in the number of peers and the number of allocated computational resources. 

Keep Exploring!!!

Backpropagation

Big thanks to Matt for his post on backpropagation. A big thanks to Upgrad for the teaching opportunity. Many times we need time to connect the dots. From hectic days of model building vs learning basics and teaching is a good opportunity to balance perspectives. 

I was able to work out the example and share it with my students.

Keep Exploring!!!

A/B Testing products

Vue.AI

• Repeated Testing, Sampling, No Random experiments
• Expose and Run test at different durations / peak days
• Split Traffic across tests

taplytics

• Audience Targeting

More Products

• mattmazur testcalculator
• neilpatel testcalculator

More Read - Link

Keep Exploring!!!

All Data Matters

Data - BI - AI - All Data Matters

Everything is connected. When you look at Data, You need to see

• What is value addition from BI - Dashboards
• What is value addition from Real-time - Dashboards
• If this does not make sense from Realtime / BI - Is it needed for AI / ML Models

The disconnected view will get into cycles. Have a balance of all views and customer angles to get the final goal :)

All data is useful if you have the right lens. Everything has its own value not everyone sees it.

A decade back. I remember in all my transaction tables, the Archival job was to clean up records older than a year and keep database performance updated. Now historical data is the essence of building ML models. Now Historical data is as much important as transactional data :)

Keep Exploring!!!

Relationship between Linear and Logistics Regression

 

From linkedin post

In linear regression, our main aim is to estimate the values of Y-intercept and weights, minimize the cost function, and predict the output variable Y.

In logistic regression, we perform the exact same thing but with one small addition. We pass the result through a special function known as the Sigmoid Function to predict the output Y.

 

So, Logistic regression uses the same basic formula as linear regression but it is regressing for the probability of a categorical outcome.

Linear regression gives a continuous value of output y for a given input X. Whereas, logistic regression gives a continuous value of P(Y=1) for a given input X, which is later converted to Y=0 or Y=1 based on a threshold value.

That’s the reason, logistic regression has “Regression” in its name.

Keep Connecting Dots!!!

Optimism - Persistence - Focus - Collaboration

Some thoughts for both professional / financial goals and financial freedom

• Optimistic viewpoint / Courage to explore ideas
• Concept of Abundance mindset / Optimism
• Zero to one - Belief in Groundbreaking ideas - Don't side with Majority
• Invest in Skills
• Punter Investing - Split across multiple companies (Even if something survives you still gain :))

Work on customer solutions, not just technical aspects.

Keep Going!!!

2021 Retrospection

• There is pleasure in learning, exploring new territories. Automotive / Telematics was interesting this year
• Getting things into production, making ideas work - Good exposure to Retail / Forecasting implementation
• How Analytics can be used for Application performance optimization - I loved Air Crash Investigation type Analysis :)
• Expanding more on vision in Agriculture, Fashion - Fun to work with data unavailability, connecting many missing dots
• Always learning tech seems a bit away, Time to code vs Time to focus on domain vs Time to pick up leftover areas balancing - Docker, Kubernetes, AWS, GCP, ArgoCD, So many things can't master all. Sometimes I feel am I a software technician who knows to operate or someone who knows things and how it works
• Work-Life balance It happens sometimes that is the flavor in consulting
• I could not travel as much as I wanted to, Always things keep running in mind, to-do, how I want to plan out
• A bit more need to diversify learning, financial independence, making smart investments
• Need to cut down on food, healthy lifestyle, Hope to retire early soon
• Personal front things remain as is, Life sometimes goes faster not being able to balance all aspects

In the end, You learn, you grow, you unlearn, and Keep going!!!

Some memories, some regrets, some lessons!!!

Keep Going!!!

When you see your ideas work. Validating its inline with Markets

Many times I was only getting rejected in all my explorations of ideas. I am happy to see some of ideas are pursued actively by others. In a way it validates vision.

• Telematics - Autoinsurance - doofuscorporation 
• Video Surveillance - CAMDOG INC 
• Marketing analytics for brick and mortar stores - privy  

Keep Exploring!!!

A dream / An idea / Still thinking about rental / affordable - Video Analytics as a service

• Are you worried to travel and leaving things at Home
• Do you want a rental security vision service to alert
• Do you just want it to be fitted in custom locations
• Do you want Fire, Intrusion, Sound all capabilities?
• As low as 1$ per hour, we provide working setup in 20 mins
• Your privacy maintained, Retain your data 
• Everything set up with Remote monitoring, plug and play service

For any help please contact us at +......

Need to collaborate/explore more in this space..

Keep Going!!!

Why Questions ? Why docker is Faster / Light weight ?

Docker is based on Linux Containers

Linux Containers (LXC) share OS kernel of the host. OS provides resource isolation. Fast provisioning, bare-metal like performance, lightweight 

Ref - Link

Keep Exploring!!!

Technician or Solution Developer

Who is a Technician?

I can compile/build solutions. I can host. I do not know how it works or internals but I can compile/deploy/customize

Who is a Solution Developer?

Knows how it works, The underlying details, What is the thought process behind implementation

Think Big perspective

• Solution is not just technologies, frameworks, libraries, "patterns" or "paradigms"
• Why you do it, How it works, How does it connect to what I learnt earlier ?

I still feel lot to learn but I believe unlearn, relearn is part of whole process, Enjoy the journey, stay humble. This is also called Cargo cult programming. Doing with minimum know-how.

Keep Going!!!

Good Reads - Building Usecases - NLP & ML

Sometimes we need to have the right set of tools, patterns to build our idea. Bookmarking some interesting links

• Advanced NLP with spacy
• Haystack is an open source NLP framework that leverages Transformer models. It enables developers to implement production-ready neural search, question answering, semantic document search and summarization for a wide range of applications.
• Data Science Topics
• Ray provides a simple, universal API for building distributed applications.
• FORECASTQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data

Keep Thinking!!!

Daily observations on Model building and Feature engineering

 Reality is far from the well-structured features listed. Some things that we discuss on the daily basis

1. Feature not available in the transaction table, asking business users to provide data
2. Currently, it's in excel need to have a process to streamline
3. Feature Data is currently not captured but will be done in future
4. The model overfits because this feature data is currently not captured, Can we leverage/infer through any existing features?
5. The reason for the spike in sales is due to a certain reason which is not captured currently

The question of #Why? #Find? #Add feature data and iterate it is the real crux of learning. Before we talk about feature stores we need to know/collect/understand get all features under one roof.

#datascience #features #perspectives #learning

Domain + Data + Algo,  Connecting, Collecting all of them in a consistent repeatable way with the right data every month only can get consistent results :)

What do I do in my work?

Do I code full day? No

When do I code? When I pick up on building ideas, building a prototype, analyzing issues/data related observations

What do I do in my work? Clearing red flags, Discuss, Review, Recommend based on literature reads, ML ideas, techniques relevant to the context

What are my strengths? Data, Domain, and then ML. Seeing everything with a blended lens matters where we need to map both customers vs solutions vs timelines

What things do I read? Yes, you need a lot of ideas to give quality review comments, competitive products, algorithms at work, arxiv papers, domain-related reads, tech blogs. You need to build an idea repository

Sometimes I feel I am busy but not productive. Sometimes, Weekends provide the window to learn things. Product perspective, tech perspective, customer perspective come with empathy, understanding, and technical acumen. 

Keep going!!!

Times person of year 2021

Successful in one industry vs Success in every industry

Strategy, Guts, Hardwork, Consistency, Vision = Elon

.@elonmusk's ventures have spanned everything from internet startups to sustainable energy to artificial intelligence. Here's how the world's richest person built his fortune #TIMEPOY https://t.co/mlpoC6vjUq pic.twitter.com/vUfmwnLghs

— TIME (@TIME) December 13, 2021

Career Stages, Perspectives, Growing together

• Sprints are very time-bound and coming up with MVP needs quick experiments, spotting the blocks of vision to implementation, and working as a team to get connected from Day 1 to MVP to building blocks. 
• This needs a good mix of experienced folks to connect domain, and data and communicate the technical vision with the development team. 
• This is a mix of aspects balancing engineering aspects of functionality, and scalability in early design vs making it working/operations in minimal time. 
• It is easier said and done but that is where the crux of experience lies, working with clarity in a chaotic situation. 
• Communication becomes the essence of both stakeholders and the team. 
• Bringing the best, being able to contribute, clearing the red flags within the team, and effective collaboration with clients make the mark of a mature leader.

From link

Key points I like

• Don't fall for the hype without first conducting a production-grade proof of concept
• There should be no single point of failure in an app; always have a fallback and thoroughly test it
• Seniority is defined by collaboration, not technical expertise.
• Measure everything - each and everything
• Create your own dashboard for operational and technological excellence.
• Make sure tech is building with business metrics in mind.

##EngineeringLeadership

Data Science Skills

• Domain - When we had the best algos but accuracy was poor we figured out domain aspects we missed
• Algos - Some advanced algos Transformer-based models did give us a good baseline, They do help with their latest improvements. Best Algo plus domain knowledge to find relevant variables is key for success. 
• Data Engineering - Time to incorporate, design, manage features has to be future driven not one at a time
• Communication - Explaining this feature is needed and connects this way to the big picture matters

Data science use case has multiple stakeholders Every aspect of listed points helps to bring everyone/address clarification that arises while implementation.

Keep Thinking!!!

Everything is not same - Perspectives and Clarity matters

20 years of __________________________________________

• 20 years of experience = 20 years of the same project / different projects?
• 20 years of experience = 20 years of same role / multiple roles?
• 20 years of experience = 20 years of services/ product building
• 20 years of experience = 20 years of 9-6 or 9-12 ?
• 20 years of experience = How many endless weekends / production go lives
• 20 years of experience = How many learning migration on skills / domain / data
• Titles vs experience vs Expertise vs Being aware of true self matters

With experience

• Balance both journey and current tasks
• Code to convince someone this is what I meant
• Code to unblock/find next steps
• Code to validate this idea works
• Prototype to share this is feasible

Young folks need time to trust. More than experience connecting with them with all skills/code/experience matters. 

Keep Thinking!!

Zillow Machine Learning Fallout

Good read - Link

Machine learning is no silver bullet if you do not consider domain, data, changing environmental factors. A classic case of missing domain knowledge is flagged in this story.

• Zillow does Real estate - selling, buying, renting, and financing
• Zillow home value estimation models failed.
• Assumption - assumption that housing prices would continue to climb without interruption at a stable rate
• The domain experts warned of issues with the predictions.
• The business went ahead anyway. Finally, it bombed

Lessons

• Domain expert warnings considered as Go / No-go for production, not just model accuracy
• Learn / Incorporate Data Changes to understand changing trends
• Performing A/B Experiments to understand customer behaviors and leverage optimal values based on outcomes
• Better model/feature management / keep improving on features / incorporate external factors based on domain expert perspectives #machinelearning #technology #datascience #domainknowledge

Another good read Zillow, Prophet, Time Series, & Prices

i cannot get over that the zillow data scientist job posting "strongly prefers" you have experience with a python library that is designed to make it as piss easy as possible for little babies to do y_t = f(t) time trend / curve fitting forecasts. pic.twitter.com/YTUcgascCi

— Senior Data Masseuse (@ryxcommar) November 3, 2021

WHY IS INTERMEDIATING HOUSES SO DIFFICULT? EVIDENCE FROM IBUYERS

• Predict that households’ wiliness to pay for liquidity is highest in those markets
• Sophisticated algorithmic pricing

My Perspectives

• I love the housing.com approach to rank an area based on amenities, wellness, connectivity
• Plus a pricing range based on amenities and facilities provided
• Plus growth potential / Availability
• Demand vs Supply

A combination of this would suggest a recommended price that a domain expert could adjust based on other external factors. ML is a guideline, not a blind predictor

Keep Thinking!!!

Leaf Classification

Leaf Classification

Paper #1 - Plant identification using deep neural networks via optimization of transfer learning parameters

Key Notes

• 1.2 million labeled images of 1,000 different categories from the ImageNet = one thousand two hundred per class
• LifeCLEF 2015 - 91,758 labeled images of different plant organs (e.g. flowers, fruits, leaves, and stems), from 1,000 - 91 per class

Parts of Plant

• Branch 
• Entire 
• Flower 
• Fruit 
• Leaf 
• LeafScan 
• Stem 
• Overall

• Increasing the batch size from 20 to 60 improves the overall accuracy
• 80 patches for data augmentation

Paper #2 - Multi-Organ Plant Classification Based on Convolutional and Recurrent Neural Networks

Key Notes

• Feature engineering approaches such as Scale-invariant
• feature transform (SIFT), Bag of Word (Bow), Speeded-Up
• Robust Features (SURF), Gabor, Local Binary Pattern (LBP).
• Most generally used features to distinguish leaves of different species
• Hybrid generic-organ convolutional neural network, abbreviated HGO-CNN
• Three different sizes: 256, 384 and 512
• Crop 256 × 256 center pixels
• Multi-Scale Plant Images Generation
• During network training, 224 × 224 pixels are randomly cropped from the rescaled images and fed into the network

Keep Exploring!!!

Face Swapping - Research Reads

Paper #1 - FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping

Key Notes

• Early replacement-based works simply replace the pixels of inner face region
• GAN-based works  have illustrated impressive results
• GAN-based network, named Adaptive Embedding Integration Network (AEI-Net)
• Adaptive Embedding Integration Network (AEINet) to generate a high fidelity face swapping result

• DeepFakes, and FSGAN all follow the strategy that first synthesizing the inner face region then blending it into the target face

Paper #2 - Face Swapping: Automatically Replacing Faces in Photographs

Paper #3 - Face Detection, Extraction, and Swapping on Mobile Devices

The Face Swap algorithm consists of five main steps:

• Viola-Jones face detection using Haar-like features [1], Active Shape Model fitting [4], face rotation, skin-tone matching, and smoothing using Laplacian Pyramids [2]. The Viola-Jones face detection uses an OpenCV library [5] to detect faces from a frontal view. 
• Laplacian Pyramid for face 1
• Laplacian Pyramid for face 2
• Laplacian Pyramid after Swapping
• Final Collapsed Pyramid
• Image blending Example
• faceswap-GAN
• FaceSwap
• Faceswap Dev
• Deepfake Faceswap
• DeepFake Tools

More Reads

• DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection
• Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward
• Deep Learning for Deepfakes Creation and Detection: A Survey
• DeepFake, CihaNet, SimSwap, FaceSwap-MarekKowal, IPGAN, FaceShifter, FaceSwap-Nirkin, FSGAN, SimSwap
• DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection

Keep Exploring!!!

Remembering Facts vs Evaluating Ideas

I find it hard to remember configuration parameters, default settings, metrics. These are key to many certifications. Often we focus on the problem at hand, not specific functions or code to check.

Every definition is custom to each cloud provider and the set of theoretical FAQ questions, syntax specific to language. We neither measure problem solving or domain knowledge but rely on syntax and remembering facts. This is a stark difference between product vs service companies. 

Certification does not necessarily mean you have the skills to build a solution. They merely imply familiarity with a tool/infra. As long as you map your current skills to new skills find the gaps and address you can build the required solution.

Learning is a collection of observations, experiments, experiences, applying your relevant past lessons. It is a compound effect. Building a solution is easy, but thinking from a futuristic perspective marks the difference between a newbie and an experienced techie.

20 years of experience is not working on the same project. The wider you explore bigger the perspective. The more you fail, the more you are aware of different domains/roles. In the end, let it be a collective memory of different experiences. Win or lose enjoy the journey.

I keep coding my logic with a mix of syntax I recollect across SQL, C, Python, R, C#. First, pseudo logic comes to mind. Later the logic is corrected based on StackOverflow answers. Every language has its own way of defining constructs and separators. Am I a bad programmer, mmm maybe... Always there is more to learn :)

Anyways value addition needs to be quantified so you need to pass this too :)

AI in Finance

Paper #1 - AI in Finance: Challenges, Techniques and Opportunities

Key Notes

• Key Areas are capital markets, trading, banking, insurance, leading/loan, investment, asset/wealth management, risk management, marketing, compliance and regulation, payment, contracting, auditing, accounting, financial infrastructure, blockchain, financial operations, financial services, financial security, and financial ethics
• Classic techniques including logic, planning, knowledge representation, statistical modeling, mathematical modeling, optimization, autonomous systems, multiagent systems, expert systems
• Modern techniques such as recent advances in representation learning, machine learning, optimization, data analytics, data mining and knowledge discovery, computational intelligence, event analysis, behavior informatics, social media/network analysis
• Specific business problems, such as market trend forecasting, stock price prediction, credit scoring, fraud detection, financial report analysis, pricing and hedging, marketing, consumer behavior analysis, algorithmic trading, social commerce, and Internet finance.
• Portfolio planning and optimization: including designing, planning, optimizing and recommending investment portfolios and strategies in a market
• Forecasting and prediction: including the regression, classification, estimation and prediction of trend (up or down), movement (direction and scale, etc.), value (e.g., price or volatility)
• Business profiling: including describing, segmenting, characterizing and classifying markets, products, customers, and services.
• Sentiment and intention modeling: including characterizing, representing, modeling, analyzing and evaluating the polarity, diversity, propensity and their dynamics of customer sentiment and intention 
• Anomaly detection: such as characterizing, quantifying, detecting, classifying and predicting abnormal, exceptional and changing behaviors, products, patterns, performance

Paper #2 - Enhancing Financial Inclusion using Mobile Phone Data and Social Network Analytics

Key Notes

• Datasets - call-detail records, credit and debit account information of customers is used to create scorecards for credit card applicants
• Call-detail records are used to build call networks and advanced social network analytics techniques are applied to propagate influence from prior defaulters throughout the network to produce influence scores
• predictive model for a target measure of interest (e.g., churn, fraud, default) 
• sociodemographic  information, such as age, marital status and postcode; debit account activity, including timing and amount of payments; and credit card activity
• sociodemographic features such as age, marital status and residency as reported at the time of the credit card application are extracted.

Paper - P2P LOAN ACCEPTANCE AND DEFAULT PREDICTION WITH ARTIFICIAL INTELLIGENCE

Key Notes

Features for the first phase are: 

• debt to Income ratio (of the applicant); 
• employment length (of the applicant); 
• loan amount (of the loan currently requested); 
• purpose for which the loan is taken
• loan amount (of the loan currently requested); 
• term (of the loan currently requested); 
• instalment (of the loan currently requested); 
• employment length (of the applicant);
• home ownership (of the applicant. Rented, owned or owned with a mortgage on the property); 
• verification status of the income or income source (of the applicant. If this was verified by the Lending Club); 
• purpose for which the loan is taken; 
• Debt to Income ratio (of the applicant); 
• earliest credit line in the record (of the applicant); 
• number of open credit lines (in applicant’s credit file); 
• number of derogatory public records (of the applicant);
• revolving line utilisation rate (the amount of credit the borrower is using relative to all available revolving credit);
• total number of credit lines (in applicant’s credit file); 
• number of mortgage credit lines (in applicant’s credit file); 
• number of bankruptcies (in the applicant’s public record); 
• logarithm of the applicant’s annual income (the logarithm was taken for scaling purposes); 
• FICO score (of the applicant); 
• logarithm of total credit revolving balance (of the applicant).

Paper #3 - Behavior Revealed in Mobile Phone Usage Predicts Credit Repayment

Key Notes

• Mobile phone transaction history prior to the extension of credit, and whether the credit was repaid on time
• Transition to a postpaid plan
• Call and SMS metadata

Paper #4 - Data Science in Economics

Key Notes

More Reads - 

• A Deep Causal Inference Approach to Measuring the Effects of Forming Group Loans in Online Non-profit Microfinance Platform
• Evaluating the Effect of Credit Collection Policy on Portfolio Quality of Micro-Finance Bank
• The Effect of Client Appraisal on the Efficiency of Micro Finance Bank
• Cloud Computing For Microfinances

Keep Exploring!!!

Solving the right problem at the Right time matters

2013 I was part of the Team that worked on Traffic Forecasting for Retail Stores

• Multiple stores across geographies
• Multiple DB’s for each local store

The forecasting system used to run at Enterprise, Synchronize data to local stores with their own internal synchronization jobs. 

• These jobs were configured to run according to time zones of stores
• The algorithms were mostly around a weighted moving average, trend + moving average 
• The forecast job runs leveraging previous data and projects forecasts by the hour for next day, hourly basis patterns
• The actuals are captured the following day and measured against it
• In case of data not present sister stores (similar stores) data was leveraged for calculation

Whatever we say as of today measure model drift, missing data features, work at scale, coexist along with existing transaction system was built as server components, custom-built. 

What we missed are

• Instead of Traffic forecast if we had done a sales forecast it would have helped to apply solutions for both eCommerce and retail giant
• We had inherent details of out of stock, replenishment alerts. The same could have been used for out of stock forecast per zone, replenishment forecast per zone
• These real-time reports from RFID could have served as effective forecast opportunities on the same

Sometimes we may have the right technology and architecture but not the right use cases. Now I see the same things ML attempts to do with #kubeflow, #pipelines, #scale but the same problem which was solved with models available at that point in time would take a different set of skills to solve today 😊

Keep Exploring!!!

AI - Education Opportunities

Paper #1 - Strengthening e-Education in India using Machine Learning

Key Notes

Applying different data mining algorithms on the data of the person and suggesting which course is appropriate for him based on his background knowledge

Paper #2 - Personalized Education in the AI Era: What to Expect Next?

Key Notes

Content summarization and question generation Multi-modal content understanding: Human-in-the-loop content design

More Reads

Teaching Machine Learning in K–12 Computing Education: Potential and Pitfalls

Estimating returns to special education: combining machine learning and text analysis to address confounding

Keep Exploring!!!