My Observations in Data Science Work

• Code Credit - I have observed people take code from git/forums and change headers/rewrite methods without providing due credit. I call out explicitly the reference/approach from paper/presentation/tech talks. The original work would have taken months to arrive at it. It is better to reference it for anyone to understand the reasons behind the design

• Domain Knowledge - Ignoring the value of domain knowledge - To find a possible loss item, there is a mix of techniques in Retail employed. Implementing the same as events/actions may be the dumbest way for someone who undervalues domain knowledge :)

• Code Management - Treating ML code as Enterprise Software - Models are deployed without source code on edge devices. We use disparate sources of data. My recommendation is to build a high accuracy model and then consider merging the code into a common code / common class.

• Resellers - Do not get carried away with third-party tools, frameworks. Try with open source, build your datasets, better models. Buying from the third party AI software and re-selling it we cannot call it as 'Analytics Company' :)

• ROI - Set realistic expectations, target the low hanging fruits, understand ML as 'Preparedness'. ML cannot be quantified in terms of revenue unless it is a clear business case like Market Basket Analysis, Sold together, Bought together

Keep Thinking!!!

Day #330 - Feature Selection Techniques

After a long time started reviewing Stanford ML Project reports. This report Feature Selection for predictive models is the study report for the day.

Code Examples

Happy Learning!!!

Data Science Hiring Thoughts

New age Resume

• Demonstrate business knowledge
• Demonstrate working prototypes
• Demonstrate technology adoption and value realization

My perspectives run in a different direction. I learn the required technology pieces to get my idea of working. I am not really focused on mastering technology but leveraging it and learning to solve my implementation.

For me over years focus shifted from technology to business, business to futuristic needs. You can only build a future when you can envision the future. The future is built on optimism, creativity, technology, and making it more accessible and affordable.

Build your portfolio and perspective not just your technical skills. - Siva 

More Reads

Link1

Key Lessons

• Numbers are much more powerful than the same bullet point without numbers.
• Increased user engagement 27% by refactoring our front-end experience in React.

Link2

Being able to feel empathy and to take in the other person's perspective

You Probably Shouldn’t Work at a Startup

How to write a resume that converts

Happy Learning!!!

Career Lessons

I have gone through cycles of ideas, initiatives rejected, put down as it flows through leadership levels. Many times it got delayed but there were few memorable successes in #rewrite #warranty to XBOX, filing #retail patents, pushing ideas. I have observed myself going through frustration, rejection cycle. I was able to achieve those ideas in my next role the same/next company. It takes time to prove/demonstrate our ideas have potential. Sometimes we need to wait/ further sharpen our skills/ till the next role / next company to make the idea successful.

• Sometimes your best ideas will have no future, they will get killed. Keep going
• Keep versioning all your ideas and add techniques to improve upon it
• Build domain knowledge + AI to solve it optimally
• When people kill ideas, find a place to grow if you believe in your ideas
• 'Go' where you 'Grow', 'Grow' where you 'Go'

More Reads - Link

Accidental Leaders - (July 16th, 2020)

IT has a lot of accidental leaders. Years of experience may not reflect competency. Some categories of leaders

Idea Killers - Any idea you bring up to the table. The intention is to play a safe game. They view IT projects as sailing in smooth weather. Very less interest in innovation and pure 9-6 safe side players.

Jargon Gurus - Highly qualified, great connection with their ladder up. Any idea you take up they will provide a counter idea for it. Ultimately their goal is to prove the idea is not good enough to pursue

Enthusiast Leaders - They don't know about technical aspects but get carried away with wow factors. Their shortsightedness will not give them long term perspective

Passionate Leaders - Talk on your face, encourage the idea. Warn you when things fail but really back you when it fails. They are hard to work with but they take the organization to the next level.

The IT industry is a very hard industry to spot true leaders. All leaders who speak may not be good at execution. All leaders who deliver may not be good presenters. Keep going. Build the true leader in your inner self - Siva

P and L in Career

Yes, you guessed it right, It's Profit and Loss. Naaaa. My perspective is Passion and Learning.

Profit and Loss Perspective

• Goal is reach title / position / salary
• What is in it for me
• What's my next role
• Sell better

Passion and Learning

• Build expertise
• Continuously find new ways to implement ideas
• My vision will succeed one day
• Share credit and learning
• Customer needs to be happy
• It's okay to fail 

#HappyLearning
#Keep Thinking!!!!

Evolution of Data Storage, Analysis, Analytics - Database - DW - DataLake

2000 - 2010

• Stage 1 - Papers, Ledgers
• Stage 2 - Excel, Access
• Stage 3 - Databases for OLTP / DW for OLAP (ACID Properties)

Knowledge -  I know how my business is performing as of today. I know the past 6 months of historical data and performance

2010 - 2015

• Stage 4 - Hadoop for large scale DW (Velocity, Volume, Veracity)
• Stage 5 - NoSQL (CAP Theorem)

Knowledge -  I store all possible data without strict data type validations, can query in large scale data for Adhoc queries. Schema on Read, Finding insights from unstructured data (logs, text, records, events)

2015 - 2020

• Stage 6 - AI to extract  interpretable data from Image, Video, Sound (Numbers, Object classification, Count, etc)
• Stage 7 - lakehouse = (Hadoop + RDBMS + NoSQL + AI for data extraction from unstructured sources)

Knowledge - I have insights from structured, unstructured, vision, audio and every type of data. Handling all types of data and finding meaningful insights.

CAP Properties and Databases

More Reads
Spanner Reads

How Does Spanner Avoid Single Point of Failures in Writes?

Databases and cost aspects

Happy Learning!!!

IoT Architecture

• Edge - Lightweight protocols for Machine to Cloud communication - MQTT (Lightweight pub/sub) - Immutable data - Read-Only - Data cannot be modified
• Cloud Entry - Large scale data ingestion to consume data - Kafka - (Distributed Log processing)- Immutable data - Data cannot be modified
• Cloud Streaming - Real-time data analysis to report and alert - Spark - (RDDs / ML ) - Immutable data - Data cannot be modified
• Store and Analyze - Reports on data, Completed transactions - Postgres, RDBMS - Do the Remaining CRUD
• ML on Edge, ML on Streaming data, ML on Stored data (completed transaction)

The confluence of blockchain and IoT in Supply Chain can improve the efficiency of the operations, as per @EqualOcean's research. Link > https://t.co/5nWv9EIqNh @antgrasso via @LindaGrass0 #SupplyChain #blockchain #SmartContracts #IoT #Tech #DigitalTransformation pic.twitter.com/bDmPz7Njz0

— C-Suite Tech Point (@CsuiteTechPoint) May 25, 2020

IoT ecosystems that stitch together sensor-based data with legacy systems and newer technologies hold tremendous promise for those engaged in moving goods. Link > https://t.co/Uk0IFVrVqs @DeloitteInsight via @antgrasso @antgrasso_IT #CloudComputing #AI #IoT #IIoT pic.twitter.com/FpUDIR9kgN

— Antonio Grasso (@antgrasso) May 21, 2020

mt @Fisher85M
cc: @MikeQuindazzi @antgrasso#SupplyChain 4.0 & #4IR

[@JacBurns_Comext]#MachineLearning #IoT #IIoT #sensors #CyberSecurity #bigdata #5G #Industry40 #mobile #disruption pic.twitter.com/BCH8SnEEkF

— Digital Retweet (@DigitalRetweet) May 9, 2020

Happy Learning!!!

Queues vs Logs

Queues - Someone will publish a message in the broker, Consumers can read from the queue. Long back I worked on this on SQL Server Message Broker
Logs - All the information/transactions in SQL is implemented as WAL (Write ahead logging). At some point when a checkpoint is reached the transactions are written to disk. Commands are applied and data changes saved to disk.

How logs can be used?

• Read the transaction and replay it elsewhere (Allows multiple consumers without blocking each other)
• Keep logs read-only and let everyone read it (persist it as long as needed)
• Read information in logs in sequence (maintain sequence to replay it in order)

So, Logs can be read across multiple readers and it enables scaling :)
Tools have evolved but the fundamentals are the same. Kafka is similar to a log playback system (distributed log processing) which helps to scale, publish and consume data.

Happy Learning!!!

Telecom ML Use cases

• Data - massive amount of network performance data 
• Broad Areas - Network optimization, Preventive maintenance, Virtual Assistants and Customer Experience
• AI Focus Areas - increase in data traffic, identifying potential problems in the network, best customer experience, AI bot, virtual assistants, Smart Home customer experience, current health of our network, predict a battery failure in telecommunications equipment 
• AI for security management - detect the spread of viruses, the activation of unknown attacks as well as data and information exfiltration.

Machine Learning for Networking: Workflow, Advances and Opportunities

Problem-Adapted Artificial Intelligence for Online Network Optimization

Happy Learning!!!

MQTT vs Kafka Notes

MQTT (Message Queue Telemetry Transport)

• MQ Telemetry Transport
• The choice for wireless networks
• Publish / Subscribe system

Key Concepts

• MQTT Session - Connection, Authentication, Communication and Termination
• Client Operations - Publish, Subscribe, Unsubscribe, ping
• Multiple implementations of client libraries and brokers (Mosquitto, JoramMQ...) exist and are virtually compatible
• MQTT just specifies the transport, and vaguely the application part (i.e. how data is handled and possibly stored, how clients are authorized...)
• Standard pub/sub protocol (with multiple implementations)
• MQTT as a communication protocol between several applications. It was designed to be extremely low light to fit into IoT and resource-constrained environment

Competing Tools

• Constrained Application Protocol (CoAP) 
• Simple Media Control Protocol (SMCP) 

MQTT Recommendations

• Machine-to-Machine (M2M) communication
• MQTT is designed for low-power devices
• MQTT purpose is to hold a communication channel alive on client-side without draining battery and to have a reliable messaging
• The edge devices speak MQTT protocol (for the benefits it has in edge environments). 
• Very easy to configure and use with open source tools, Lightweight with a relatively small data footprint, Varying levels of Quality of Service to fit a range of

Kafka

• The main motive behind Kafka is scalability.
• Apache Kafka is a message broker based on an internal "commit log": its focus is storing massive amounts of data on disk, and allowing consumption in real-time or later (as long as data is still available on disk)
• It's designed to be deployable as cluster of multiple nodes, with good scalability properties. Kafka uses its own network protocol.
• Kafka has no built in msg priority, poor security, heavy protocol
• Apache Kafka may deal with high-velocity data ingestion
• Kafka depends on Zookeeper in order to work properly
• Kafka is better suited for microservices
• Kafka is a messaging broker with transient store which consumers can subscribe and listen to. It's an append only log, which consumers can pull from.
• Specific message storing/distributing software, vaguley of the same family with its own protocol.
• Kafka is broker that can store large volume of data and for long time (or for ever). It was designed to be scalable and provide the best performances. 
• High-throughput, Distributed, Scalable, High-Performance, Durable, Publish-Subscribe, Simple-to-use
• Advantage of Kafka's strengths (replayability, based on an even sourcing architecture)

IoT environments combine both MQTT and Apache Kafka.

IoT Integration with MQTT and Apache Kafka from Kai Wähner

Apache Kafka is the New Black at the Edge in Industrial IoT, Logistics and Retailing
MQTT Overview
IoT Data Platform
MQTT vs Kafka Stackoverflow
MQTT vs Kafka Cloudera
MQTT vs Kafka Stack Share

Happy Learning!!!

AI Magic - AI Use Cases

The stories and magic are the AI use cases we witness today

• Recommendations we provide in eCommerce sites (Bought together, Sold together)
• Chatbot assistance we provide with NLP, Sentiment Analysis of Product Reviews
• Forecasting to stock up requested inventory levels
• Analytics provides insights on who shops in which stores, which brings efficiency to the supply chain and improves customer service

Where Industries Will Put Practical #AI To Work
by @PwC @MikeQuindazzi#ArtificialIntelligence #Tech #Technology #IT

Cc: @thetechprophecy @maxjcm @ronald_vanloon @xbrlstandard @pascal_bornet pic.twitter.com/OpR5HNWOq1

— Ronald van Loon (@Ronald_vanLoon) June 1, 2020

Keep Thinking!!!