Docker MariaDB Installation

More Reads - Link
Happy Learning!!!

Day #2 - Data Management Course Notes

Link
Module #4 - Master Data Management
Key Notes

• Master data, agreed and shared across Enterprise (Customer, Employee)
• Reference data is a subset of master data (Country Code, Industry Classification)
• Data Representation differences - MM/DD/YYYY, DD/MM/YY, Identify potential matches, apply business rules and merge records
• MDM creates master records with consistent data representation
• Have a lookup to reference table with possible different representations, Update sources with valid state names

Module #5 - Data Integration

• Scenarios  - ETL, ELT, Batch, and Real-time Integration
• Data Source -> Integration -> Target Systems
• Source -> staging -> DW
• Calculations, Aggregations during ETL
• Batch Processing, Real-time integration

Module #6 - Analytics

• Reporting and Analytics
• Monitor, Understand and improve business
• Cross-Selling, Upselling
• Marketing Insights
• Promotional Campaigns
• Dashboards, Visualization, Alerts, Conditional Reports
• Data mining, Stats, Text Analysis

Module #7 - Data Architecture
Models, Rules, Policies that govern definition, storage

Module #8 - Privacy

• Protect Sensitive data
• Unauthorized Access
• SSN, DOB, CreditCard, SalesPlan
• Governance - Rules, Privacy - Protect Rules
• Email Protection, Antivirus, Firewalls, WIFI encryption, Cloud data storage systems, Secured Network Access   

Happy Learning!!!

Day #1 - Data Management Course Notes

Course Link

Key Notes
Module #1 - Introduction Data Management 

• Development and execution of Architecture, policies, procedures to manage data

Key Capabilities (People, Process, Technology Aspects for each Capability)

• Metadata management
• Data Quality
• MDM
• Data Governance
• Data Integration
• Analytics
• Data Privacy
• Data Architecture

Data Element - Representation of data. Attributes, permissible values, Identification defined.
Critical Data Element - Key elements capturing business process. Examples - Business Facts, Support Business Process, Data appears in Key Reports, Unique Identifiers - CustomerId, SupplierId
Metadata management

• Data structures from different models
• Information about Attributes, models, columns, glossary

Data - Definition, Business Rules, Ownership, Logical Data Model, Physical data - Schema
Data Sources - OLTP, OLAP, Integration - Data Movement
Business Metadata - From Business Perspective, ownership. Customer Name - Client Name, Legal Name, Trade Name. Rules to validate those names
Roles - Business Owner, Data owner, technical owner
Technical Metadata - Entities, Attributes, Mutual Relationships, Associations
Data Lineage - Traceable path from data sources, data marts, data warehouses
Identify Data Elements, Collect Business, Technical Metadata, Enforce Data standard
Tools - ETL tools, Modelling tools, BI tools, Domains, Definitions, values, Hierarchies. With all structured, unstructured data this would be done at data lake.

Module #2 - Data Governance

• Availability, Usability, Integrity, and Security of Data
• Establish a process for standards
• Same policies across the organization
• Leadership, Data Standards, Ownership, Monitoring, Change Control, Executive Support
• Hierarchy - Business Sponsor - Council - Data owners

Module #3 - Data Quality Management

• Approach, policy, procedure for accuracy, timeliness, completeness, and consistency of data in system and data flows
• Data Questions like accuracy, validity, on-time arrival, completeness, uniqueness, consistency

Technical tasks

• Data Profiling, Set Rules, RCA for identified issues, Resolutions, Set a threshold and identify accuracy percentage detected

Happy Learning!!!

NOSQL Basics

[You may also like - NOSQL - can it replace RDBMS Databases]

Deep Dive is very important to understand the basics/fundamentals of product design. I have explored a couple of NOSQL database products. Based on readings from blogs/papers. I have tried to document the underlying fundamentals behind NOSQL Databases

Tip #1 - NOSQL Stands for “Not Only SQL”
Tip #2 - ACID properites - What it is all about ?

From Earlier Post - ACID Properties short RECAP

• Atomicity - Transaction is one unit of work. All or None. Either all of its data modifications are performed or none of them are performed
• Consistency - Transaction must leave the database in a consistent state. Maintain data integrity. Governing Data Structures for Indexes/Storage must be in a correct state. If a Transaction violates a constraint it must be failed.
• Isolation - Keep Transaction Separate. Concurrency is governed by Isolation levels.
• Durability - In the case of System failures changes persist and can be recovered on abnormal termination

Tip #3 - What is CAP Theorem. In Every NOSQL White paper there is a reference to the CAP theorem.

CAP stands for consistency, availability and partition tolerance 

Short and easy summary of it I found from link

• Consistency - Consistent (Latest) Data Reflected querying any server in Distributed Environment
• Availability - Data Returned from Server irrespective it is latest / last updated data
• Partition Tolerance - System is available even if individual nodes are down

As per CAP Theorem only two parameters can be targeted for complete support. To Summarize it

• As per CAP Theorem, RDBMS targets Consistency & Partition Tolerance
• NoSQL targets Availability and Partition Tolerance

Tip #4 - What is MVCC? While working on NoSQL DB, I noticed MVCC for versioning/managing locks.  

  

MVCC refers to Multiversion concurrency control. MVCC Managing providing latest committed updates for read transactions by versioning. Here with versions present Reads will not block writes. MSSQL 2005 onwards we have Snapshot isolation feature. This is also based on the versioning concept. Reposting my notes on how snapshot isolation is achieved in MSSQL

READ COMMITTED SNAPSHOT using Row Versioning in Microsoft SQL Server 2005 onwards (Applicable for 2008, 2012..) 

a. How it works - A new data snapshot is taken and remains consistent for each statement until the statement finishes execution.   uses a version store and reads the data from the version store.

b. How it solves the concurrency issues  

• SELECT statements do not lock data during a read operation  (readers do not block writers, and vice versa).  

c. Performance Advantages 

• SELECT statements can access the last committed value of the row, while other transactions are updating the row without getting blocked
• Reduces disk contention on the data files. Reducing locking resources, readers do not block writers. No more deadlocks involving readers and writers.  

d. Resource usage and overhead

•  Row versioning increases resource usage during data modification as row versions are maintained in tempdb. tempdb growth, contention. Additional memory usage.

Tip #5 - Below are common list of features implemented by NoSQL Databases and advantages of it 

• NO Schema Reqd - Data Types need not be defined
• CouchDB also uses MVCC for managing versions of data (Good Read Link )
• Auto Sharding - Spread data across servers to scale out
• Support for Replication

Still I have a long way to go to understand NOSQL, I am planning to explore NOSQL Database Architecture in detail in coming posts.

More Reads - NOSQL Patterns

Consistency Models
MongoDB 4.2.6
YugaByte DB 1.3.1
Aerospike 3.99.0.3

Happy Learning!!!!