Day #2 - Data Management Course Notes

Link
Module #4 - Master Data Management
Key Notes

• Master data, agreed and shared across Enterprise (Customer, Employee)
• Reference data is a subset of master data (Country Code, Industry Classification)
• Data Representation differences - MM/DD/YYYY, DD/MM/YY, Identify potential matches, apply business rules and merge records
• MDM creates master records with consistent data representation
• Have a lookup to reference table with possible different representations, Update sources with valid state names

Module #5 - Data Integration

• Scenarios  - ETL, ELT, Batch, and Real-time Integration
• Data Source -> Integration -> Target Systems
• Source -> staging -> DW
• Calculations, Aggregations during ETL
• Batch Processing, Real-time integration

Module #6 - Analytics

• Reporting and Analytics
• Monitor, Understand and improve business
• Cross-Selling, Upselling
• Marketing Insights
• Promotional Campaigns
• Dashboards, Visualization, Alerts, Conditional Reports
• Data mining, Stats, Text Analysis

Module #7 - Data Architecture
Models, Rules, Policies that govern definition, storage

Module #8 - Privacy

• Protect Sensitive data
• Unauthorized Access
• SSN, DOB, CreditCard, SalesPlan
• Governance - Rules, Privacy - Protect Rules
• Email Protection, Antivirus, Firewalls, WIFI encryption, Cloud data storage systems, Secured Network Access   

Happy Learning!!!

Learning Notes - Marketing your Products

This fantastic link Marketing-for-Engineers  has a list of resources, curated talks, pointers from a Marketing perspective.

Some key Talks and Notes
Talk #1 - How To Create UX Personas
Persona

• Type of customers
• Industry, device, time, goals of customers
• Identify customers, patterns
• Connect with them
• Observe, Capture the findings and add more data points
• Insights into customers
• Track changes to personas over time

Talk #2 - What is an Empathy Map?

• Understand user thinking, feeling, saying and doing
• Conducting moderated sessions to understand users
• Ask a lot of open-ended questions
• Understand and prioritize user needs
• Capture explicit opinions, implicit signals/response
• Created based on aggregated users across gender/age groups etc..

Talk #3 - Customer Development and Lean Startups
Venture performance = product development + customer development + team building + luck
Venture performance = learning rate | product development | customer development | team | luck
Analyze market, product, customers
Methodology to go from unknown to known. Customers, products, business models

Step by Step process

• Define
• Layout Steps
• Make Observations
• Analyze data
• Conclusions / Further Experiments

Ways to Reach out

• Reach out to existing network
• Learn from competitors
• Local meetup
• Social media
• Local incubators

Zero Cost Marketing

• Blogs
• Free Resources / Tools
• Collaborating with like-minded folks

Startup - Search for a business model, Transition to next phase scale-up, Execute the model

Happy Learning!!!

Learning Notes - Edge Devices - Bench marking

Paper #1 - A Survey on Edge Benchmarking

Edge benchmarking parameters

• I/O throughput
• Data staleness
• End-to-end communication or computation latency

Devices

• Intel Movidius Myriad X VPU
• NVIDIA 128-core Maxwell and 256-core Pascal architecture-based GPU
• Google Edge TPU

Paper #2 - MLPERF TRAINING BENCHMARK
Tasks Considered

• Image classification
• Object detection (lightweight)
• Instance segmentation and object detection (heavyweight)
• Recommendation
• Reinforcement learning 

Modifiable Hyperparameters

• Batch size, Learning-rate schedule parameters
• Optimizer: Adam or Lazy Adam, Learning rate
• Maximum samples per training patch

MLPerf Training v0.6 Results
Paper #3 - Early Experience in Benchmarking Edge AI Processors with Object Detection Workloads

Paper #4 - pCAMP: Performance Comparison of Machine Learning Packages on the Edges

More Reads

• A Hybrid Approach to Privacy-Preserving Federated Learning
• EdgeCNN: Convolutional Neural Network Classification Model with small inputs for Edge Computing
• The Big Benchmarking Roundup
• How to set up a Postgres database on a Raspberry Pi
• A Step by Step guide to installing PyTorch in Raspberry Pi
• Federated Learning On Raspberry Pi
• FEDERATED LEARNING OF A RECURRENT NEURAL NETWORK ON RASPBERRY PIS
• How to set up a Raspberry Pi with Ubuntu from almost any device, completely headless, with just WiFi
• Object Detection on NVIDIA Jetson TX2
• Benchmarking Edge Computing

Keep Thinking!!!

Day #1 - Data Management Course Notes

Course Link

Key Notes
Module #1 - Introduction Data Management 

• Development and execution of Architecture, policies, procedures to manage data

Key Capabilities (People, Process, Technology Aspects for each Capability)

• Metadata management
• Data Quality
• MDM
• Data Governance
• Data Integration
• Analytics
• Data Privacy
• Data Architecture

Data Element - Representation of data. Attributes, permissible values, Identification defined.
Critical Data Element - Key elements capturing business process. Examples - Business Facts, Support Business Process, Data appears in Key Reports, Unique Identifiers - CustomerId, SupplierId
Metadata management

• Data structures from different models
• Information about Attributes, models, columns, glossary

Data - Definition, Business Rules, Ownership, Logical Data Model, Physical data - Schema
Data Sources - OLTP, OLAP, Integration - Data Movement
Business Metadata - From Business Perspective, ownership. Customer Name - Client Name, Legal Name, Trade Name. Rules to validate those names
Roles - Business Owner, Data owner, technical owner
Technical Metadata - Entities, Attributes, Mutual Relationships, Associations
Data Lineage - Traceable path from data sources, data marts, data warehouses
Identify Data Elements, Collect Business, Technical Metadata, Enforce Data standard
Tools - ETL tools, Modelling tools, BI tools, Domains, Definitions, values, Hierarchies. With all structured, unstructured data this would be done at data lake.

Module #2 - Data Governance

• Availability, Usability, Integrity, and Security of Data
• Establish a process for standards
• Same policies across the organization
• Leadership, Data Standards, Ownership, Monitoring, Change Control, Executive Support
• Hierarchy - Business Sponsor - Council - Data owners

Module #3 - Data Quality Management

• Approach, policy, procedure for accuracy, timeliness, completeness, and consistency of data in system and data flows
• Data Questions like accuracy, validity, on-time arrival, completeness, uniqueness, consistency

Technical tasks

• Data Profiling, Set Rules, RCA for identified issues, Resolutions, Set a threshold and identify accuracy percentage detected

Happy Learning!!!

Learning Notes - Fashion Recommendation Papers

Paper - REDEFINING THE OFFLINE RETAIL EXPERIENCE: DESIGNING PRODUCT RECOMMENDATION SYSTEMS FOR FASHION STORES
Keynotes

• Leverage sensor technology, novel customer services
• Smart fitting rooms that offer garment recommendations

Attributes

• Sensor capabilities of smart fitting rooms
• Algorithms for Brick & Mortar recommendation systems
• Contextual attributes

Algorithms

• Content-based methods were similarities between item features are taken into account
• Collaborative-filtering approaches where product suggestions are based on the previous behavior of users with similar preferences

User cold start problem

• Using social media profiles to deduce customers’  preferences
• Adoption of association rule mining algorithms for product recommendations

Absence of explicit product ratings

• Propose combining them with clustering approaches
• Make association rules less generic and more customer-group-specific

Contextual Information in Brick and Mortar Stores

• Special attention must be paid to the selection of contextual attributes
• Location and time
• Trends and occasions
• User locations
• Customer interactions with products
• Store locations
• season, occasion, weather
• Categories activity (e.g., trying on garments)

Analysis of Data

• Patterns of buying across seasons

How to handle a cold start?
1. When customer attributes / product recommendation not available, use the popular item (Frequently bought from transactions)
2. Recommendations based on products customers bring into fitting rooms, Recommendations based on Apriori
3. Target individual customers if they identify themselves

Smart Mirror: Intelligent Makeup Recommendation and Synthesis

Key Notes - Model to generate/identify facial features, facial attributes, and makeup attributes recommendations
ML work

• Facial feature extraction - facial landmark point extraction
• Regions of Lips, eye, hair and face points extracted
• Makeup recommendation and synthesis. Recommendations based on eye shadow, skin color, and lip color
• Apply recommendations for eye, lipstick, hair color
• Apply makeup on the face

Keep Thinking!!!

Learning Notes - Convex Optimization in Python with CVXPY

Key Notes
Convex Optimization problem

• Decision variable to solve
• Objective Function
• Inequality Functions
• Linear equality constraints
• All functions are convex
• Convexity - Positive curvature, curve upwards like parabola

Useful in below fields

• Applies in various fields
• All ML algos based on Convex Optimization
• SpaceX landing Convex Optimization

How to Solve ?

• Using Solvers for particular form of problem
• Linear program, quadratic problem, second order con program

Convexity Verification

• Using Disciplined convex programming
• Determine curvature of every node in expression tree

cvxpy

• Developed in python
• Modelling framework
• Parameter assignment
• Simulation visualization
• Energy Management (Load Demand, Battery Charge / Discharge / Price margin / Load)

Keep Thinking!!!

What is your learning strategy?

Interesting Question - Link
Some answers worth noticing

1. Unless you are working on that tech stack actively, you cannot remember it all
2. Learn fundamentals. Learn only things based on YOUR needs
3. Interviews aren’t totally reflective of the job but one‘s got to pass the interview before getting to the job.
4. Demonstrate competency of core concepts, get the job, and then rise to the occasion as fast as possible
5. Start building the things that you want to build, and you'll learn what you need to along the way.
6. By deliberately selecting your objectives and evaluating possible solutions based on those objectives.
7. Build stuff. Pick out ideas and just build them to build them
8. As a human, you can't be good at everything. You always need to balance between being average in a lot of topics and good in a few topics.
9. Think of a side project you would enjoy that includes a handful of these technologies and start building it. 

My Approach - Every problem these days requires dusting previous memories, read up minimally to recollect paste efforts, spend time connecting the dots and applying to the context 

Happy Learning!!!

Weekend Learning - Convex Optimization - Stephen Boyd, Professor, Stanford University

Key Notes
Mathematical Optimization

• Choices of a vector/numbers
• Constraint - Legal / Technical / Physics
• Judged by objectives
• Examine on profit/utility

Purpose

• Make good actions
• Reduce risk/ cost is objective / action
• Constraint come from the manufacturing process

Variables

• Vector x could be trades, schedule 
• Resource allocation 
• Optimize signals

AI / Stats / ML

• X - parameters to model
• Constraints (impose requirements)
• Optimization used for worst-case analysis

Optimization-based models

• Aggregate small number of agents
• Simplistic assumptions and formulate the problem
• Predictive ability of models

Convex Optimization

• Minimize objective
• Constraint to hold
• Linear constraints
• Constraints and linear functions will curve up

Why?

• Methods available to solve them

Different application areas

• Spacex landing is effort of optimization
• Optimal trajectory to landing path
• 10 times a second
• Networking / Circuit design

How to use ?

• Formulate as convex problem

Examples

Example #1 - Radiation treatment planning

• Things decided are actions
• linear y = Ax
• options - beam diverges / tissues / hits bone scatters
• Overcharge / Undercharge

Example #2 - Image in painting

• Guess the lost parts
• Minimize function / Convex problem
• Remove 5% of pixels

SVM

• Predict boolean outcome
• spam/ fraud 
• Old school - gradient method
• Convex Optimization - Differentiability irrelevant

Lasso

• Methods for sparse model construction
• With 1/5th measurements analyze

Solving

• Define in High Level language
• Solved by solver
• Helps in rapid prototyping

Large Scale Distributed Optimization

• Grid Updates
• Image / Video processing

Read Ferenc Huszár's answer to Why is Convex Optimization such a big deal in Machine Learning? on Quora Happy Learning!!!

Notes - Webinar - The Role of Digital Product Identity in Fashion Digital Transformation

Key Notes

• Customers make well-informed decisions
• Customer use digital platforms to validate their thoughts
• Website, Social media content helps to connect during a crisis situation
• Digital lies on top of the supply chain of the product
• Customers get access to both digital/physical product
• Advertisement, Social media images are digital assets
• All information related to products categorized as Assets, Information, and Projects
• Platform for management of products, catalog, prices
• As the product evolves the digital assets also evolve with time
• AI is accelerating with COVID situation
• Speaker from Company - Link

Keep Thinking!!!