"No one is harder on a talented person than the person themselves" - Linda Wilkinson ; "Trust your guts and don't follow the herd" ; "Validate direction not destination" ;
Showing posts with label Microsoft. Show all posts
Showing posts with label Microsoft. Show all posts

April 02, 2022

My Perspectives of IT and Business Team Collaboration

If all my ideas have worked I would have climbed a few steps more. For the things it worked, Why it works? Why does it take so long to sell an idea? Does collaboration really work in action vs wishes of leadership expectations? To understand I wanted to look back at some paths in my history.

Sometimes when we look back we know why certain initiatives took so long, Why it works the way it works. This goes back to my Microsoft days XBOX supply chain Team. I was working with Program Management, Product Management, Support, and QA Team. The Product Management Team works closely with the Business Team.

Every Team has its own priorities.

  • Product management - Support Release Xbox, Xbox360 new type consoles with required code changes
  • Business - Setup new plants, identify new vendors, reduce Red Light Repair issues, Increase Warranty
  • Support Team - Reduce the number of customer issues
  • IT Team - Support all the priorities for business / qa / features etc

One issue of the customer writing an email about the warranty in the system is not correct. After a few executive-level escalations. It boils down, We spot there are issues with the way we store. To sort out the issue we provide a free warranty.

When I moved to Amazon, the Initial few days I was listening to another Team debating a similar workflow of repair order flow. I was thinking that everywhere edge cases are the big discussion items.

As we go along keep fixing, and adding new features. This system was one good enough system that tracks cradle to death of Xbox. Manufacturing, shipment, repair, fulfillment, sales, warranty, refurbishment, scrap. Everything was there about the console.

Every time when I hear these in JDs / Ecom Supply chain / Repair functional roles. I feel long back we had done all this as an in-house product. 

The core of it is warranty tied to a customer is a much better approach than a warranty mapped to a console. The order of transactions when they are out of sequence is created out of sequence status and we could keep track of update sequence vs delayed delivery of transactions.

I was able to work with my mentor Roji to build a prototype. It took so long release over release to recommend it to put in production. It had data migration, and a core engine but overall the changes were worth it. After a year when there were no priority projects, An initiative like that picked up this item. At least I didn't see any other announcement of a free warranty extension :) post-implementation.

I was wondering why did it take so long, Idea is necessary, we know dirty records in the system due to the sequence of transactions. There is effort and impact. The priorities across each function, business vs selling / pushing new ideas across functions take time.

What other options could have made it work?

  • If each function thinks about its own priorities it is not effective collaboration.
  • Sometimes when we solve a problem we need to think beyond fixing the current fire.
  • Everything is driven by a cost center. Fix now, Move on let's check later. The perspective of how the next customer should not go through the same issue needs a collective lens, not a function-based
  • Innovation in a way is looking beyond daily functions and seeing what matters more from a customer perspective
  • Keeping every function busy, planning back to back new items vs slow and steady everything has its own tradeoff/results
  • Sometimes even pushing idea visibility with title / connecting with leadership helps
  • If every team thinks from a customer point of view there would be more consumer-focused collaborative efforts than thinking based on individual function priorities.

The true essence of collaboration needs closer coordination, understanding, and planning across functions with customers as the epicenter of focus. 


Keep Thinking!!!


June 26, 2021

Notes from Azure Synapse Training

Lesson #1 - Tables – Indexes Best Practices

  • Clustered Columnstore index (Default Primary) - Highest level of data compression. Best overall query performance
  • Clustered index (Primary) - Performant for looking up a single to few rows
  • Heap (Primary) - Faster loading and landing temporary data. Best for small lookup tables
  • Nonclustered indexes (Secondary) - Enable ordering of multiple columns in a table. Allows multiple nonclustered on a single table. Can be created on any of the above primary indexes. More performant lookup queries
Queries with the following patterns typically run faster with ordered CCI:
  • The queries have equality, inequality, or range predicates
  • The predicate columns and the ordered CCI columns are the same.
  • The predicate columns are used in the same order as the column ordinal of ordered CCI columns.
  • Caching of results, Enable caching at DB level then query level - Resultcachehit flag returns the value whether it was reused

Fact table primarily CCI as we would run large aggregations based on dimensions so CCI becomes a choice for fact tables. 

Lesson #2 - Distributed table design recommendations

  • Hash Distribution: Large fact tables exceeding several GBs with frequent inserts should use a hash distribution.
  • Round Robin Distribution: Potentially useful tables created from raw input. Temporary staging tables used in data preparation.
  • Replicated Tables: Lookup tables that range in size from 100’s MBs to 1.5 GBs should be replicated. Works best when table size is less than 2 GB compressed.

Lesson #3 - Result-set caching

Cache the results of a query from SQL pool storage. This enables interactive response times for repetitive queries against tables with infrequent data changes. The result-set cache persists even if SQL pool is paused and resumed later. 

Cache Checks

You can tell if a query was executed with a result cache hit or miss by querying sys.pdw_request_steps for commands where value is like ‘%DWResultCacheDb%’

Lesson #4 - SQL Data Classification is a new feature in the Public Preview, that:   

  • Automatically discovers columns containing potentially sensitive data
  • It provides a simple way to review and apply the classification recommendations through the Azure portal.
  • The sensitive data labels are persisted in the database (metadata attributes) and it audits and detects access to the sensitive data.
  • We offer built-in set of labels and information types, however customers can chose to define custom labels across Azure tenant using Azure Security Center

Lesson #5 - Dynamic Data Masking

  • Prevent abuse of sensitive data by hiding it from users
  • Easy configuration in new Azure Portal
  • Policy-driven at table and column level, for a defined set of users
  • Data masking applied in real-time to query results based on policy
  • Multiple masking functions available, such as full or partial, for various sensitive data categories (credit card numbers, SSN, etc.)
Lesson #6 - Spark vs SQL Server (Memory Handling)

Keep in mind spark uses memory much in the same way as sql server uses the buffer pool by storing frequently used objects in memory it reduces overall I/O and improves performance in large joins, sort and aggregates contrast this with a traditional hadoop based architecture which relies heavily on writing data out to disk between steps.

Every concept technical maps as an advancement or some sort of limitation which existed in place. Compared to SQL 2008 where you don't have so much of these feature synapse has beautifully evolved as a good environment for real-time / ML / big data handling capability for reporting / Ml recommendations/lakehouse / real-time BI system. Gone are the days of month-end jobs or Data sync jobs. 

All good lessons :) Fantastic Features!!!