Technology - Consulting - Coding - Domain Learning

At a senior role, what are things we can accomplish. I agree with the perspective and the work that is called out Link 

Technical Work

• Review for technically design/architecture
• Analyze for with security/scalability of design 
• Collaborate with other technical teams to agree on interfaces and common APIs

People Work

• 1-1s on a weekly basis
• Regular feedback

Plus my own additions

• Patenting / Knowledge Sharing
• Building your point of view
• Be on top of tech - Code as you need

Ongoing

• Teaching, mentoring and coaching
• Technical conversations and reviewing designs

Plus a perspective on mastering technology vs domain I like this article

Adding my top reasons to solve problems and not to master tech - Work On Interesting problems not Technologies

• Ideas take time and need refinement
• As you keep coding, you keep building perspectives
• Working prototype creates more interest/excitement and keep improving
• Your interest will not die down as you are solving newly known challenges
• Scope, features you will balanace when you spot the unknowns
• Its your idea you will not kill it :)

On WLB - Link

• We collectively create the culture we live in, changes comes from healthy WLB
• 20% of your work produces 80% of your value. Prioritize over priorities

Myth of super performers

I loved the below lines, I have seen this specific behavior. People who deliver but do not share, collaborate within the team. Adding my own perspectives

• X is the only developer who gets anything done
• Do not actively share knowledge with his peers
• Good at communication but bad at collaboration
• Explain simple things in a complicated way
• Good, connect at Leadership Level. Over-communication at the leadership level, limited collaboration at ground level
• Instead, make more people productive will reap the greatest benefits
• Turn our attention from individuals to groups of people
• Don’t mistake humility for ignorance - There are a lot of software engineers out there who won’t express opinions unless asked

Agile principles alternative definition

• Empathy for customer needs
• Actually getting stuff done
• A bird’s-eye view of the product vs market 
• Able to balance birds-eye view to product view vs component view

More Reads

20 Things I’ve Learned in my 20 Years as a Software Engineer

Keep Thinking!!!

SoftBank funded startups and Covid effects

Interesting read - Link

We would have used/heard most of these startups WeWork, Oyo, Uber, Ola, etc. Everything is backed by SoftBank with good investments. With COVID effects their valuation will come down with the 'Shared' concept becoming 'Healthcare / Safety' threat for the near future. In the list of companies, there are Zero Companies in Healthcare, Research, AI, Education sector. 90% of them are in Travel, Hospitality. Vision is more of Short term gain and long term loss. It would be hard to see Tesla, SpaceX, SolarCity, Hyperloop any such game-changing companies coming out of SoftBank portfolio.

A bit more on the 'Shared' Concept. Bunch of startups came up with the strategy for shared mobility, low cost of ownership, etc

• Zoomcar
• OLA
• Uber
• Bounce
• Vogo
• Ridesharing apps

Often in OLA, Uber except for prime segment, very few %% of cabs are well maintained. The concept of personal hygiene, good vehicle quality, cleaning frequently comes with a cost. Will this crisis will teach everyone on more precautions for personal hygiene, better maintenance of commercial vehicles?

Behind tech layoffs lay systemic cash flow negative companies by @gonesgr https://t.co/nJgbqVYDjJ

— Siva (@sivaram2k10) June 1, 2020

Keep Thinking!!!

Process vs Talent Vs Accountability

• The process is a guideline to measure, deliver. Scrum / Agile every process is a method to measure and improve
• Talent - I believe good talent will excel process. Knowledge is built on learning/gathering experiences. We learn from mistakes, errors, experimentation. With good skills, you can deliver superior results. Good Talent will drive the better process without micromanagement
• Accountability - Combination of good talent and accountability will deliver exceptional results. When you own it and have the autonomy, freedom to experiment and push the best you will deliver beating the expectations

Code is not just 'copy-paste' reuse of someone else logic. The solution is an outcome understanding of business, end goals, measurable data, quick turn-key solutions, and picking your best ideas from all your past lessons.

Get Good Talent, provide autonomy, empower them you will get exceptional results!!!

Improving Women Safety

To reduce crime against women more than strengthening laws we need to get to the root cause of issues. We need to analyze the crime data and fix the source of the problem.

We need to analyze the crime patterns based on different aspects to find the underlying patterns.

Pattern vs Solutions

• Correlation with alcohol - How to reduce/limit alcohol consumption 
• Correlation with education - How to reduce dropouts and improve education
• Correlation with income category - Sustainable jobs
• Correlation with marital status - Family aspects
• Correlation to caste - Driven by caste / Unemployment / Dropouts
• Correlation to age group - Social media, porn impact
• Correlation to social behavior - Drugs / partying / Addiction
• Correlation to job type - Government vs Private jobs vs Daily vs Organized Crimes

Education is not limited to a few years. Education is not about degrees. The real purpose of education is to unlearn and relearn things from morality/humanity perspective.

It needs complete society change, not just laws. Let's prepare a safer tomorrow by making the required changes. 

Keep Questioning!!!

Learning vs Compensation vs Titles

This question usually comes to me at different cycles. The experiences we gather from different roles and domains get accumulated and gives us perspectives unique to the domain.

After a decade I had options to choose

• Go forward in same Database space
• Setting up teams from scratch 
• Domain-based focus and build expertise
• Professional services, Automation, QA
• Learn something new

Every challenge once you solve becomes boring. Everything is difficult until you find a way to do it. Passion without skills won't get you where you want to go.

Five years back I decided to reboot myself in Data science space. Every other previous experience gave new perspectives to explore. Building expertise in one area, solving problems in a domain perspective, Working on creating unique IP, Learning other related tasks to delivery.

Many times in career I always get discussions and proposals to go back to previous roles. From data science now I get to hear Big Data + Data science skills. Even within data science to build expertise in video, numbers, text aspects, it needs deep dive and consistent focus for a certain period.

Close to two decades in a few years now question remains how long to remain as Individual contributor, Finding a role that uses all the previous experience, keeping up with titles?

The primary aspect of my satisfaction is learning. After a few years, there could be something that would replace today's data science.

As I grow older, I have to pick and choose my areas of interest, areas to focus on. I never met happiness focusing alone on titles or compensation. They also have a priority after satisfaction though. A job that gives you learning, exciting but not overwhelming challenges, manageable work-life, and compensation would be the best thing to look for and have satisfaction till the day of death. Sometimes you need titles to execute your strategy.

Compensation helps you meet your financial goals. The search is to find a role that gives you opportunities that meets your priorities and leverages your strengths. Career is a long term thing. Keep learning until you find your dream role. Be prepared for the role. Don't wait for titles to learn the role.

Interesting Read - An incomplete list of skills senior engineers need, beyond coding

My Perspectives

How to write a design doc, take feedback, and drive it to resolution, in a reasonable period of time

• Competitive product
• Papers Referenced
• Potential Architecture
• Prototype 
• Demo
• Next Steps

How to mentor an early-career teammate, a mid-career engineer, a new manager who needs technical advice

• What would I do if I am in your situation? This is how I look up and advise. We cannot learn everything. We have to pick and focus on few things which are important for us.

How to influence another team to use your solution instead of writing their own

• View it as a joint success. Sometimes you have to give credit to make things work. 

How to get other engineers to listen to your ideas without making them feel threatened

• You are not selling your idea, You are conveying how it is done across other industries/companies. You need to have a working demo, deep dive on the benefits of it. It may not work in the first go but over a period of time trust will develop

How to craft a project proposal, socialize it, and get buy-in to execute it

• Everything we pick someone who has already experimented / could be something ongoing. Give the context, potential value and take it to next level.

Keep going!!!

2.0 Lifestyle Skills

To survive we need a newer set of skills and a better awareness about yourself

• Building Culture of Learning
• Training and Experimenting Mindset
• Emotional and Communication Skills
• Fail and Learn Mindset
• Balance Attitude dealing with Depression, Life Struggles

Happy Finding Yourself!!!

Finding Great Candidates

• Communicate at the simplistic level
• Create end to end experiments than certifications
• Rely on passion, Consistent learning and good team players
• Look for people who intend to make a change, consistent performance matters 
• Move out of puzzles, programs. Project or a prototype that requires reasonable design, code, use cases, an end to end implementation matters
• Puzzles and program can find a good coder but doing end to end projects requires more skills than just coding
• People who share what they learn can impact a change in culture than people who work in silos
• Great Skills takes years, Passionate about technology to see how it evolves matters

Happy Learning!!!

Experience != Expertise

Software Engineer - Learning to code
Senior Software Engineer - Learning from failures, Coding for a few years
Lead Engineer - Failed in experimenting with different prototypes for a decade
Principal Engineer - Mastered Failure but Master Experimenter, Learns to look code from Failure perspective, Seen more failures to guide the safest way for rest of team

Working for years does not build expertise until you experiment and learn from failure.

Keep Learning!!!

My Career Journey

2003 - Testing windows OS is great
2004 - Coding C++ MQ adapter is interesting
2005 - Setting up an Application support team and Swiss onsite
2006 - Finding my place in Microsoft, Learning the Domain, Supply Chain
2007 - Performance, SQL Migration, Biztalk and Automation
2008 - DB Developer
2009  - BI / OLTP performance tuning / TSQL developer
2010 -  BI / OLTP performance tuning / TSQL developer
2010 - Need more $$ and challenges
2011 - Setting up Team in Amazon
2012 - Better become Individual Contributor, Again Database and QA, Setting up Team
2013 - Performance, Automation, Database development
2014 - Big data jump and getting into it
2015 - Start from scratch Data science
2016 - Data Science year 2 All 2 year Masters deep dive
2017 - Data Science year 3 Image and Data Analytics projects
2018 - Data Science year 4 Machine Learning projects
2019 - Data Science year 5 Deep learning Projects
2020 - Vision Expert and Deploying Solutions in Scale

To sum up - "Data Guy, Empirical Learner"

The days I spent weekends learning is more than my weekday efforts. I have witnessed the transformation of technology and the tools that evolved all these years. Experimenting and working on different roles, starting things from zero and re-learning has been a rewarding experience. All these years I have learned, relearnt, and still learning. Outside this life also gave me lessons and blessings.

Keep learning, Keep growing!!!

Big Data and SQL Journey

I have been a silent observer of Big Data Journey.

• In 2010 - Ran SQL Queries on RDBMS for Committed Data
• Entity Framework came into the picture, heated debate Stored procedures are no longer needed
• Hadoop came, I queried on HBase, Hive for the Real-time / Analytics queries for Current Data and Historical data
• After a few years now I see the queries in Spark SQL and KSQL for Streaming Passing Data
• SQL queries are shifting all the way from Database - Real-time - Real-time Stream Querying
• SQL Skills are coming to every layer of the data processing stack

Querying now goes for - incoming data, passing data, current data, historical data

• Kafka SQL - SQL for Incoming data
• Spark SQL - SQL for Passing data
• HBASE - SQL for Real-time data
• HIVE - SQL for historical data

All Hail SQL!!!

Happy Learning!!!

Build a world class personalized Education System

This post is my personal opinion. There are three types of knowledge in my perspective

1. Fundamentals - The basics of maths, physics, chemistry that underlines every concept we relate to
2. Technology - The innovations/discovery that runs in the technology that is built on the fundamentals
3. Experimentation - The gaps that connect the Fundamentals and Technology with meaningful experiments

I do not believe in Multiple Choice Questions, paper-based expertise. They are good but every method of teaching has their own drawbacks. Outside our regular education, we need to have a constant learning setup, the mindset to experiment, stay curious and generate more ideas. It takes a lot of learning, gathering new perspectives to be creative and passionate about things we do. A personalised education means able to access student is good in fundamentals, theoretical concepts and build the necessary bridge to make the next steps concrete that allows him to build his expertise. We have a disconnected system where we are not encouraged to connect the dots and build a larger system. Education ends when the grades are out.

With such a large pool of students why we are not able to get innovation at grassroots. The system that encourages questions, experimenting and creativity only can lead to innovation. With AI I believe those days are not very far. Access to quality educational content, the ability to provide personalised education is just a few years away.

I believe the intelligence, passion and curiosity to solve will be 10X more with all these resources in place to build a more intelligent, passionate and curious learning mindset.

Keep Learning!!!

Ten years of blogging

Reflecting on this

• Consistent learning past ten years
• Learning gradually expanded across multiple domains
• Working on AI ML since 2015
• Core DB, BI, AL, ML past ten years
• Working on core digital senses text,video, image, data
• Keep sharing knowledge. I have referred back to my learning at some point
• Self-learning is the best way to keep updated in the knowledge ladder

This year

• Improve the content quality
• Network more
• Read 30+ PhD thesis
• Read 100+ research papers carefully
• Code everyday
• Prototype ideas MVP end to end 10+ ideas

Knowledge is awareness of events happening in the field, exposure to several ideas in designing and solving AI problems. Connecting the dots and gathering more perspectives from tech talks, blogs, sessions. As the perspectives add up creativity comes in solving problems. Everything is an opportunity in life when we are open to learn and understand. I may not have a phd but nothing stops me to deep dive, gather the dust and understand things at nuts and bolts level.

Grateful to many online resources / papers / live sessions / cs231 / MIT classes. The doors are wide open when you are passionate to accomplish your dreams. Keep Learning, Keep Sharing.

Keep learning, keep sharing knowledge...

New Year Surprise - Featured in Top 100 Data Science Blogs

2018 gave this surprise today. I was wondering if my blog has improved in rankings. The latest list of  Top 100 Data Science Blogs, Websites & Newsletters To Follow in 2019

My blog is #59th Spot :). From #76 (Honorary mention), It is moved up to #59 this year.

Keep Learning, Keep Sharing Knowledge.

Wish you all a blessed, Successful, Happy and prosperous 2019

Youtube - Please Improve your recommendations

With a lot of social media, new data. It's a lot of data duplication in Youtube recommendations, The recommendations are not contextual, time relevant, emphasis on emotional wellness

• Data Duplication - News from one channel followed by same news reported from other mediums. The data is the same but why would we need to revisit the same new again

• Repeat of News with Same Situations Involved (Aircraft Accident follows a list of other accident news in history). News with Negative emotions or More news and noise could be controlled or promote tips for Air Safety  

• Promote Heterogenous mix - (We listen to songs / tech talks / motivation videos / news-trends) - Recommendations depend on day of week / time of day / place. It has to be more contextual than just what type of videos I saw recently

• Impact on children - I have observed my niece spending tons on time on toys, cars etc. Learning has to be both online and offline. Time moves without any alert. Rather being glued its better we reduce/encourage better content that promotes creativity, questioning, thinking. Children don't know what's best to them, parents often feel giving smartphones makes them calm down

• Emotional Wellness - There has to be a human touch/impact in every aspect of life. We are all tracked by our android phones. They know more about our patterns/actions. It can better understand our highs / low points of life than just making us download/watch more bits and bytes. Recommendations based on our lifestyle/commute / recommending start times / proactive to alert based on our previous activities. 

After two years. Today read this article . Key lessons listed below

• ‘successful’ recommendations is watch time, providing compelling recommendations
• Great for a company trying to sell ads but waste of time for consumer
• YouTube’s interest to keep us watching for as long as possible.

Happy Thinking!!!

Predictions by 2025

A quick write up on my predictions for technology/education and other sectors by 2025. Will revisit it again after a few years.

• Large scale deployment of Video Analytics in Scale for Home monitoring / Attendance Monitoring / Retail / Gesture-based tracking. The scale of adoption (30~40% in Indian Middle-Class Home market)
• Smart Cities Reality Check - Bringing together IoT + Analytics + Solar Energy into better / optimum utilization of resources (In metros compliance would be there for all SEZ, Commercial establishments - 50-60%)
• Redefined Education - Customized Learning based on Individual patterns / Choices  (30~40% in Indian Middle-Class Education Market, Affordable, Free and Customizable)
• Healthcare Analytics - More Adoption for Automated / Proactive alerting from Video / Data Analytics (30% of diagnosis would be done by Analytics- prediction)
• More Open Source Quality Education Tools - More new you-tube similar channels targeting education domain / free and open-source quality education for Higher studies (Udemy, Coursera, Khanacademy probably focused on Engineering, Medical, Arts, Science very specialized channels with focused groups across the globe)
• Drone / Quad-copters / Air Ambulance - Based on Solar + Drone Technology (They will have a decent market share and adoption for delivery / carry the minimum load)
• Battery / Electric cars vs Fuel Based - This would still be the same unless the cost of product / better innovation takes place to reduce the cost of manufacturing for Battery cars (20~30% Battery cars in Indian Automotive market)
• Big Data + Analytics - More the data, More the chaos, Real-time meaningful analytics from huge data sources vs computing power in Home devices or Smart Homes will still be a challenge 

#RandomThought

Update May 29/ 2020
Corona crisis has updated a lot of things. Glad to see many startups built on unreasonable practices are gone from the market.

COVID Accelerated the below things.

• Online Education and free education is becoming a reality. 
• Work from Anywhere is becoming a reality.
• Every Retail store is becoming a Digital Store
• AI in healthcare / screening 

Link
Key Notes

• Device to custom food with nutrients
• Autonomous vehicles market share increase by 50%
• Electric vehicles market share increase by 50%
• Use of Green energy
• Other relevant 2025 predictions listed above
• SpaceX Mars Colonization
• Global Warming
• Impact of China on the World

THE WORLD IN 2040

Interesting Read world in 2040

You are not born educated,
You are not born learned,
You are not born experienced,
You are not born matured.

Education trigger learning,
Learning trigger experience,
Experience trigger maturity.

Maturity becomes your habit & foundation.

— LearnLifeWealthTravel (@AnyBodyCanFly) July 4, 2020

Happy Learning!!!

Sunday Perspective - Data Scientist

Happy Learning!!!

Good Read - Managing your IT career in 30's

Great Read from Quora. Usually I share my readings / learnings. This post was compelling, practical to manage work in 30s.

Read Quote of Michael O. Church's answer to What do software developers age 30 and over know now that they wish they had known in their 20s? on Quora


Read Quote of Michael O. Church's answer to Since programmers on average produce $500,000 for the tech companies they work for, why aren't they paid more? on Quora


Happy Reading!!!

Automation, Tools, QA

This is in continuation to previous post on Automation. I wanted to write this post after reading article Testing Trends…or not?

When I had to test ecommerce portal across locales automation was useful to validate happy path, search, order, returns etc...For specific fixes (perf improvements, tab size adjustments, zoom in adjustments, alignments I had to rely on manual validation. Automation can cover positive functional flows. The pain point was ids were changed frequently with every release. There was always a backlog between automation implementation vs current production code. If you refer back to article Stop Writing Automation, author clearly lists the failure points which will not be covered in automation. 

Automation can be broadly classified into

• BVT Tests
• Regression Tests
• Functional Tests
• In-house Tools for perf tests, set-up

There has been a lot of focus on exploratory testing, context driven testing etc.. My perspective is core of it lies in product centric knowledge. The more you explore / learn about the system, higher the chances to identify critical bugs. Three things that are essential for a QA role are

• Product Centric Knowledge (Willingness to explore and master the domain / product)
• Technical Acumen (Know the system functional know, Learn whenever possible)
• Look at the big picture, Failure points (Here exploratory testing, context driven testing, mind maps will help)

I had worked in DB Dev / Test, UI - Test roles. DB has been more interesting and fascinating than UI :). 

I prefer SDET (profile) mix of both code level / functional tests than purely relying on black box tests. Rotational Software engineer program in Microsoft is a very good example. Fresh Graduates would spend 6 months each in DEV / Test / Support / Product management. Depending on their interest they can finally pick a role of their choice. This model would provide complete picture of Release Cycle. For every role you need to work with respect to the context.  Also, flexibility to adjust to different profile would give you broad range of skills and a better perspective of products / functions.

5 misconceptions / intelligent mistakes of #testautomation by @dorothygraham #btdconf pic.twitter.com/SWEDXLYW2T
— Richard Bradshaw (@FriendlyTester) March 20, 2014

More Reads

• MindMaps
• Why is it so hard to design automated GUI-driven functional tests?

Happy Reading!!!

Testing, Testing !! Learn both Coding & Testing

I have a strong opinion on automation and its benefits. Automation and Functional testing compliment each other. Majority of automation framework I have come across are for UI Automation, ETL, Web Services etc.

More than Automation I have observed below areas addressed very well with custom tools / custom coding efforts.

• Environment set-up / clean up tools
• DB meta data set-up scripts, Extensive SQL Queries for Analysing and validate data 
• Using Tools - SOAP UI / Custom test code for quick testing
• Debugging / checking DB states, APP server logs

These efforts will help to reduce test cycle time, good insight into application, certain level of white box testing will always boost your confidence on technical implementation. Automation evolves from these tools / scripts. Gradually these efforts will provide clarity toward building a good automation framework.

I don't recommend complete black box testing / white box testing. It has to be best of both the worlds. A good amount DEV-Test-DB understanding will always help to know the application better in terms of functionality / performance / certain extent white box testing.

By having a rotational setup. QA can code certain modules for certain releases, Dev can take up some automation / minimum testing to understand the application / domain better. Every function need to compliment each other. Cross functional skills and working experience is always a plus.

More Reads

• Has automated UI regression testing helped you in finding critical defects in system?
• Manual tester wanting to get into automation
• Stop Writing Automation
• Testing Trends…or not?

Happy Reading!!!