Big Data Conference - The Fifth Elephant

Last two days I attended big data conference The Fifth Element. Below listed sessions were very good

• Scaling Data Google Scale
• Inmobi Real Time Analytics Architecture
• A Herd of Elephants - Navigating the Hadoop Ecosystem
• Messaging architecture at Facebook

Some notable quotes from Big Data Conference

“Data is the new raw material for any business on pair with Capital, People and Labour”

“We are in the age of Data Revolution”

“Big Data is all about gaining useful insights from data while keeping cost of retrieval and storage low”

“If all you have is a hammer; everything that you look at will look like a nail” (Traditional BI
Approach won’t fit all problems)

Unstructured data rules the world!!


I will posting my notes from the sessions in coming posts. You can also checkout the videos of sessions from link


Happy Learning!!!

Windows Azure Internals - Webinar Notes

Excellent webinar with in-depth discussion on Azure design, application deployment, underlying data centre and application management architecture.

Webinar notes on Windows Azure Internals

• Currently Data Centre running in 8 locations 
• New Data Centres coming up to meet growing demands
• Data Centre Operating System - Fabric Controller (Equivalent of OS for Single Machine)
• Run on Blade Servers
• Fabric controller manages hardware, provisioning it, deploying and managing Applications
• 1000 machines constitute to one cluster. Cluster is managed by Fabric controller
• Primary and Multiple Secondary Fabric controllers for Highavailability purpose
• 5 Fabric Controllers running on a cluster was demoed in this webinar
• Primary Fabric Controller maintains the current state of the System and Syncs with other Fcs
• DLA Architecture (Old Network Topology), Quantum10 Architecture  - bisected graph - (New Network Topology)
• Quantum10 Architecture adopted from bing architecture
• Software Load balancer instead of Hardware Load balancer. Software Load balancer also performs port mapping
• Fabric Controller has OS Images + Host OS (Azure OS) + Maintenance OS (Used for first time boot) + Role   

Sequence of How app is Deployed on individual Node

• Maintance OS runs on first time boot
• Windows Azure runs next step
• Fabric Control agent starts and Connects to Fabric Controller
• Machine Ready to Deploy App

RDFE Concepts (Red Dog Front End” (RDFE))

• Manages Subscription management, Billing, User Access & Service Management
• Responsible for picking clusters and deploying services (Apps)

Fabric Controller Service Deployment Steps

• Process Service model files
• Allocate compute and node resources
• Prepare nodes
• Configure Network attributes (Load balancer, Virtual IP addresses mapped to Dynamic IP of blades)

Based on DeploymentId Identify Servers running the applications to troubleshoot issues

Fabric Controller Primary connects to FC Agent, Fabric Controller agent executes commands on the VM (Node)

• Best Practice use latest version of windows OS for best performance

On PaaS role instance 3 Disks

• Resource Disk (Size of Disk = Size of VM)
• Differencing Disk (Ensure other systems not affected, Reset option is available)
• Role Image
• New VMs created for every new version of application
• Old Role is removed and New Role Image Created

Iaas Operation

• No Cached Image
• Blob Storage offered
• Only one Disk OS Disk by default
• IaaS offers different disk types
• OS Disk, Data Disk (Random Reads), Striped Data Disk and Temp Disk (Local disk spindles can be shared)

Health Maintenance

• Overview of crash management, timeouts, server issues
• Restart, Automatic Recovery logic before finally requesting for human interference for failed recovery

Looking forward for more Windows Azure Sessions...

Happy Learning!!!

Google Cloud Platform

Google Cloud Platform offerings....

• Google Big Query - Big Data Analytics (PaaS - Targeted for Data Processing)
• Google App Engine is an application hosting and development platform (PaaS)
• Google Compute Engine - Manage large-scale workloads computing  (IaaS)
• Google Cloud Storage - Store and manage data (IaaS)

More Reads
What is (isn't) Google App Engine?
Google Cloud Storage Introduction


Happy Learning!!

My Big Data Notes - HDFS, MapReduce

Data - Contains / Represents meaningful information
Big Data - Data that challenges current limits in terms of volumes and speed of data generation

Big Data refers to large data generated from social media, ecommerce sites, data from mobile devices, sensors etc. Data volume generated is huge and the rapid rate of data growth

Big Data Technology refer to technology that helps to process large volumes of data in an economically viable way with latest technologies using commodity hardware

Hadoop ecosystem is the key for Big Data Technology. Hadoop technologies are inspired by Google’s infrastructure

• Processing – Mapreduce inspired from Google Map Reduce
• File Storage – HDFS inspired from Google File System
• Database – HBase - Inspired from Big Table
• Pig, Hive - for Analytics

Source - Link

• Storage Layer is - HDFS
• Processing Layer - Map Reduce

HDFS (Hadoop Distributed File System) 

• File System Written in Java
• High throughout, Effective to read large chunks of data
• Runs on Commodity hardware 
• Fault Tolerant ( Data is replicated, Data not accessible is made available with backups available from it, making it highly available system)
• Scalable (Ability to scale - add / remove hardware for file system storage / processing for Map Reduce Jobs)
• Run on Diverse Environments

HDFS - Split, Scatter, Replicate and manage data across servers

HDFS Internals 

• Master Slave Architecture based
• One NameNode and multipel DataNodes
• NameNode manages data storage, allocation, processing
• DataNode - Read / Write Operations performed upon instruction from NameNodes
• Status of DataNode is sent as HeartBeats

Map Reduce 

• Parallel data processing framework
• Designed to execute jobs in parallel
• Two Phases Mapper and Reducer
• Mapper splits jobs into parallel jobs and executes them in parallel
• Reduce consolidates the results obtained from individual jobs
• Mapper phase need to be completed for Reduce job to start working
• Computation is performed on raw data, computation is performed where data is available (Data locality)
• Moving code to where data is located is much cheaper, efficient approach for large data volumes

Map Reduce - Distributed, Parallel data processing framework

HBase 

• NOSQL database hosted on top of HDFS
• Columnar based Database
• Targeted for Random Reads, Real time query processing
• HBase uses HDFS as its data storage layer, This takes care of fault tolerance, scalability aspects

Hive

• Targeted for Analytics
• Natural choice for SQL Developers is Hive 

Pig

• Scripting language for analytics queries

In next set of posts we will see in detail about Hbase, Hive and Pig

 

More Reads

 

Hadoop: Basic Concepts” webinar recording released

OSSCube releases “The Motivation for Hadoop” webinar resources

 

Happy Learning!!!

  

Hadoop Quick Bytes

Hadoop Quick Bytes

 

Have reviewed couple of youtube sessions on Hadoop Basics. Listed below are short one liners and fundamentals of Hadoop framework

Hadoop - Open Source Framework, Targeted for Batch / Offline Data Processing, Data & I/O intensive Applications

HDFS - Split, Scatter, Replicate and Manage Data across nodes

Map Reduce - Divide tasks, Co-locate parts of data, and manage failure across nodes

Map Reduce is a Paradigm shift 

• Operate on File Splits
• Operate on one block of file
• Operate on Key, Value Pair
• Processing is not to move data
• Move code to where data is available
• Data Locality is the key in Map Reduce Programming Approach

 

HDFS Features

• Fault Tolerant - When Nodes Fail, Replicated & Data Distributed is leveraged to recover lost data
• Self-Healing - Rebalance Fail, When a task allocated to a node fails, Job is reallocated to another free node 
• Scalable - Ability to store data in new nodes and participate in executing map reduce jobs

Key Strategy Shift - Map Reduce Job is executed where the data is stored. This is in sharp contrast to traditional ETL process where data is loaded (Delta pull) from production systems, perform data cleansing and loading it for target system for refreshing data marts.

Happy Learning!!!!

 

Columnar Database Query Execution Basics

This post is to learn answer for How Query execution works in a Columnar database.


Row based storage - This is B-Tree based storage. If we have defined Clustered index on the table, Data would be ordered in based on the Clustered index key. There are a bunch of posts in msdn / blogs to explore more on this. Early Materialization happens in case of row based storage.


Coming to Columnar storage, How it works. From link Query execution in columnar database.

For column based Storage

• Late Materialization (Load all required columns and then perform required join / filters)
• Invisible join is another interesting concept

This link  was very useful about columnar databases. Below slide from tutorial is very good on fetching required columns and apply required filters for column store.

Happy Learning!!!

Hadoop Basics - Part III

Three google research papers to understand MapReduce, BigData evolution

• Bigtable: A Distributed Storage System for Structured Data
• The Google File System
• MapReduce: Simplified Data Processing on Large Clusters

More Reads
Google: Cluster Computing and MapReduce

Happy Learning!!!

TSQL Tip for a Day

Today's post is learning from my question. Below is sample example

 

Step 1 - Create Table and Load Sample Data 

CREATE TABLE TestC

(Comments Char(100),)
INSERT INTO TestC VALUES('A'),('B'),('C'),('D'),('E'),('F'),('G')SELECT * FROM TestC 

Step 2 - Pivot Query

DECLARE @Query VARCHAR(MAX)

DECLARE @cols VARCHAR(MAX)
SET @cols = STUFF((SELECT distinct ',' + QUOTENAME(Comments) 
            FROM [dbo].[TestC]
            FOR XML PATH(''), TYPE
           ).value('.', 'NVARCHAR(MAX)') 
        ,1,1,'')

 

SET @Query = '

SELECT * from ( SELECT *
    FROM TestC ) AS P 
PIVOT( MIN(Comments) FOR Comments in ('+ @cols + ') ) pvt'
 EXEC(@Query)    

 

Table Entries

 

Query Result

 

Couple of Interesting TSQL questions and answers from dba stackexchange site.

What is the difference between select count(*) and select count(any_non_null_column)?

• COUNT(*) will include NULLS
• COUNT(column_or_expression) won't.

Next Set of Questions List

• SQL query to convert columns into rows
• Generating large strings for test data
• How to select the first row of each group?
• SQL query that concatenate values from duplicate rows in a single table
• Select multiple rows from one row based on column values

Happy Learning!!!

Hadoop Basics - Part II

[You may also like - Hadoop Basics - Part I]

Next Post is learning about Streaming Data Access in HDFS
Summarising from the link

• HDFS is targeted for batch processing
• Emphasis is high throughput for Data Reads

This looks ok but how it is achieved ? This question was useful to understand it - What is meant by “streaming data access” in HDFS?

The answer is very easy to interpret and understand. Please find below answer and underlined is important lines from the answer.

Since Data Stored Sequentials and Read Sequentially. Cost of Random Reads, Time to locate Data Node is minimal.

Another beautiful explanation from Google Research Paper - Google File System

Please feel free to add your comments.

Happy Learning!!!

Hadoop Basics - Part I

This post is to get started with Hadoop basics. This post is my notes on Hadoop feature - Fault Tolerance.

HDFS (Hadoop Distributed File System) 

• One of the key features is Fault Tolerance
• Inbuilt capability to handle data failure issues. Multiple copies of same dataset is managed by the system
• Once a particular dataset is not accessible system can replace with another accessible copy of same data set 

How this is achieved

HDFS is based on Master - Slave Architecture

Master - NameNode (Manages the Data)

• Many to 1 relationship between NameNode & DataNode
• NameNode manages the Data - How it is stored in DataNodes, How Data is Replicated between DataNodes is managed by NameNode
• The Namenode receives a Heartbeat and a BlockReport from each DataNode in the cluster
• Namenode uses a transaction log called the EditLog to record every change that occurs to the filesystem meta data
• EditLog is stored in the Namenode’s local filesystem

Slave - DataNode (Stores the Data)

• A file is split into one or more blocks and set of blocks are stored in DataNodes (Each file is a sequence of blocks)
• DataNodes: serves read, write requests, performs block creation, deletion, and replication upon instruction from Namenode.
• BlockReport contains all the blocks on a Datanode (source - Link1, Link2) 

If there is an issue with DataAccess Heartbeat would be a indicator for Data Issues. In such cases NameNode identifies and replaces with replicated DataNode copy available to be used as alternative for inaccessible DataNode.

 

 

Happy Learning!!!