"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;

November 19, 2023

Knowing, learning, applying, and predicting

Each skill contributes to the depth of a person's experience or expertise.

  • Knowing, as a skill, involves the ability to recall and understand information accurately. 
  • Learning is a skill that entails not only the acquisition of new information but also the ability to integrate it with what is already known, the ability to adapt to new situations or problems.
  • Applying is a skill that encompasses the practical use of knowledge and learning in real-world situations.
  • Predicting the future is a skill that draws from an understanding of patterns, trends, and the extrapolation of current knowledge to forecast what may happen.

Keep Exploring!!!

November 17, 2023

3PL - Touch points - Logistics - Visibility

3PL, How much visibility do you have over your Logistics? As a consumer / 3PL company track and trace inventory across the chain



The key to the supply chain is visibility.

A look at different touch points collected by FedEx and the level of visibility :)

Touchpoints

  • Item picked up
  • Left the facility
  • On the way
  • At the destination sort facility
  • Arrived at Hub
Keep Exploring!!!

November 14, 2023

Implement OAuth with GCP

This learning is more around adding OAuth for streamlit App. This video and code shared was a good reference to customize.

Creating a custom GCP project, providing OAuth Consent, Adding user info, onboarding is the process.

Step 1 - Fill Consent Screen and Add Test Users







Step 2 - Create Credentials



Sample code




Keep Exploring!!!

Realworld vs Helloworld

  • A Developer without domain knowledge will not be able to have clarity on usage / scalability design thoughts
  • UX Developer will look at how seamless the usage is all about based on target customers
  • API Developer will look at how to scale and make latency as low as possible
  • The product manager understands what things to put as features, targets 

We need different views to understand

  • Can it make money / How to?
  • Who needs it / Does anyone need it?
  • Can it be done?

Key Notes on Failure of AI Projects

Interesting article, summarizing my views / agreed views

  • Treating AI like legacy software
  • Using methodologies that do not apply 99% Accuracy unrealistic ask
  • Cross-functional POCs - Business + Tech Collaboration
  • Miss users at the center and delivered without clarity
  • Direct approach is to find (a) existing use cases with (b) existing budget 
  • No clear strategy for USER ADOPTION

It is a coding user journey.
It is coding arts.

Don’t forget to iterate between sprinting and user testing.
  • Be a programmer who understands users. 
  • Be a programmer who has product sense.
  • Be a programmer who strives for simplicity.
  • Be a programmer who creates arts.
Keep Exploring!!!

LLMOps, Guardails

LLMOps

LLM monitoring in real-time for factual / sentiments/hallucinations. Loved the visualization.


Giskard is an open-source framework for testing all ML models

A Survey on Bias and Fairness in Machine Learning

Evaluating LLM Response

  • Response Evaluation Criteria
  • Factual accuracy of Response
  • Coherence, Grammar
  • Efficiency
  • Safety
  • Emotion, Bias




Ref - Link

Keep Exploring!!!

November 12, 2023

Vector database - Research Paper Read

Vector database

Key Summary

  • Support for high dimensionality and sparsity
  • Describe rich data such as texts, images and video in various domains such as recommender systems, similarity search, and chatbots.


  • Vector data is in geospatial applications 
  • Two dimensional points such as the location of the end-user and points-of-interest may be represented as vectors
  • High-dimensional vectors can be used to represent more complex data such as text, image, audio and video features
  • VDBMSs typically support similarity search through indexing methods that enable rapid and accurate searching of similar vectors
  • Search for vectors that closely resemble a given query vector based on specific distance metrics such as Euclidean distance or cosine similarity.
  • In natural language processing, words and phrases are vectorized into vectors in such a way that similar words have similar vector representations.
  • Word2vec [7], FastText, and Doc2vec [8] are examples of techniques that create vector embeddings for words in natural language
  • From a developer perspective, queries in VDBMSs are more closely related to simple document or keyvalue store queries than to complex queries in relational databases
  • Vectors are retrieved using one or several query vectors
Use-cases
  • Similarity search in general
  • Image and video similarity search
  • Voice recognition
  • Chatbots and long-term memory
Current challenges
  • Balancing between speed and accuracy
  • Growing dimensionality and sparsity
  • Information security
Keep Exploring!!!

Vertex AI Vision

Some key steps to experiment in coming weeks. This low-code vision platform has been in my to-do list. Bookmarking some references

Stream registration

Open the Streams tab of the Vertex AI Vision dashboard.

  1. Go to the Streams tab
  2. Click addRegister.
  3. Enter the stream name and select a region. You can click Add Row to register multiple streams at the same time.
  4. Click the Register button to create one or more streams.

# This command streams a video file to a stream. Streaming ends when the video ends.
vaictl -p PROJECT_ID \
         -l LOCATION_ID \
         -c application-cluster-0 \
         --service-endpoint visionai.googleapis.com \
send video-file to streams STREAM_ID --file-path LOCAL_FILE.EXT

# This command streams a video file to a stream. Video is looped into the stream until you stop the command.
vaictl -p PROJECT_ID \
         -l LOCATION_ID \
         -c application-cluster-0 \
         --service-endpoint visionai.googleapis.com \
send video-file to streams STREAM_ID --file-path LOCAL_FILE.EXT --loop

export SOURCE=gs://cloud-samples-data/vertex-ai-vision/street_vehicles_people.mp4
gsutil cp $SOURCE .

export PROJECT_ID=<Your Google Cloud project ID>
export LOCATION_ID=us-central1
export LOCAL_FILE=street_vehicles_people.mp4

nohup vaictl -p $PROJECT_ID \
    -l $LOCATION_ID \
    -c application-cluster-0 \
    --service-endpoint visionai.googleapis.com \
send video-file to streams 'traffic-stream' --file-path $LOCAL_FILE --loop &


Keep Exploring!!!


November 09, 2023

Good Read - Things I've learned as a Sr Engineer

 Things I've learned as a Sr Engineer

Key things I liked

  • All fields have about 10-20 core principles 
  • Be authentic at work.
  • Good code is code that can be understood by a junior engineer. Great code can be understood by a first-year CS freshman. The best code is no code at all.
  • Writing good proposals for changes is a great skill
  • We should hire more interns, they're awesome
  • What did you do and what did you accomplish. That's all people care about
  • Be kind to everyone. Not because it'll help your career (it will), but because being kind is rewarding by itself.
  • Being a good engineer means knowing best practices. Being a senior engineer means knowing when to break best practices.
  • Walk me through a project: what you think, what you ask, what you do, what tools you use and WHY! What you don’t use and WHY? That’s more valuable for me.
  • Make me think! Hands-on also needs to be brains-on
  • Working with humans is a complex process. We are not logical creatures.

  • Capacity of criticism
  • Opinions and reasons to back them up
  • Willing to learn outside the field
  • Hire for attitude, teachable people
  • Eliminate repetitive strain of sprint planning
  • Scheduled regular check-ins
  • More time to delve into customer issues and develop well-defined proposals
Keep Exploring!!!

November 08, 2023

Why Legacy companies Fail to Innovate :)

 



Well summarized - Vision needs implementation strategy
#Tesla - build one #product with a #vision
Legacy companies - Modules developed by component providers without integration/vision
Lesson - Write Software yourself for 2.0 products

Keep Exploring!!!

November 07, 2023

REST APIs











Code Review Checklist

The rules are:

𝟭. 𝗔𝘃𝗼𝗶𝗱 𝗖𝗼𝗺𝗽𝗹𝗲𝘅 𝗙𝗹𝗼𝘄: Steer clear of tricky control structures; stick to simple loops and conditionals.

𝟮. 𝗕𝗼𝘂𝗻𝗱 𝗟𝗼𝗼𝗽𝘀: Ensure loops have a clear exit point to prevent endless looping.

𝟯. 𝗔𝘃𝗼𝗶𝗱 𝗛𝗲𝗮𝗽 𝗔𝗹𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻: Favor stack or static memory allocation to dodge memory leaks.

𝟰. 𝗨𝘀𝗲 𝗦𝗵𝗼𝗿𝘁 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀: Keep functions concise, handling a single task. This goes along well with Clean Code practices (Signe responsibility principle).

𝟱. 𝗥𝘂𝗻𝘁𝗶𝗺𝗲 𝗔𝘀𝘀𝗲𝗿𝘁𝗶𝗼𝗻𝘀: Utilize assertions to catch unexpected conditions.

𝟲. 𝗟𝗶𝗺𝗶𝘁𝗲𝗱 𝗗𝗮𝘁𝗮 𝗦𝗰𝗼𝗽𝗲: Keep the scope narrow to maintain clarity. Use the smallest scope for your variables (e.g., private or protected in C#).

𝟳. 𝗖𝗵𝗲𝗰𝗸 𝗥𝗲𝘁𝘂𝗿𝗻 𝗩𝗮𝗹𝘂𝗲𝘀: Always check the return values of functions, handling any errors.

𝟴. 𝗦𝗽𝗮𝗿𝘀𝗲 𝗣𝗿𝗲𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗼𝗿 𝗨𝘀𝗲: Minimize preprocessor directives for readability.

𝟵. 𝗟𝗶𝗺𝗶𝘁 𝗣𝗼𝗶𝗻𝘁𝗲𝗿 𝗨𝘀𝗲: Simplify pointer use and avoid function pointers for clearer code.

𝟭𝟬. 𝗖𝗼𝗺𝗽𝗶𝗹𝗲 𝗪𝗶𝘁𝗵 𝗔𝗹𝗹 𝗪𝗮𝗿𝗻𝗶𝗻𝗴𝘀 𝗘𝗻𝗮𝗯𝗹𝗲𝗱: Address all compiler warnings to catch potential issues early. This is often neglected in many projects!

Coding Guidelines

How I spend my time as a developer:

  • 10% writing code
  • 20% refactoring
  • 70% reading code

This is why I optimize my code for readability while still in the writing stage.

It always pays off in the long run.

And I know the next engineer reading that code will be thankful.

So think about this next time you're hurrying to finish a task.

Here's a checklist in no particular order:

  • Variables defined close to where they are used
  • Fluent syntax structured vertically
  • Names are descriptive
  • 80 characters per line
  • Early return principle

Keep Exploring!!!