Distributed Systems - Session #1

Key Notes

• Storage, Big Data, File Sharing
• The infrastructure that requires more than one computer
• High Performance, Parallelism
• Fault Tolerance - Two computer does the same things. One Fails another picks up. Availability / Recoverability, Replication
• Systems are inherently physically distributed
• To achieve security goals
• Handle unexpected failure patterns (Partial Failures)
• Challenges are Concurrency, Partial Failures
• Academic Curiosity -> Real-world Examples
• Lectures, Research papers for ideas, implementation details, labs, exams
• Map Reduce - Map Function on each of the input files, Obvious Parallelism available. The output is a list of Key-Value Pairs. Maps -> Intermediate Output -> Reducers. Collects all instances, all maps. 

Happy Learning!!!

Queues vs Logs

Queues - Someone will publish a message in the broker, Consumers can read from the queue. Long back I worked on this on SQL Server Message Broker
Logs - All the information/transactions in SQL is implemented as WAL (Write ahead logging). At some point when a checkpoint is reached the transactions are written to disk. Commands are applied and data changes saved to disk.

How logs can be used?

• Read the transaction and replay it elsewhere (Allows multiple consumers without blocking each other)
• Keep logs read-only and let everyone read it (persist it as long as needed)
• Read information in logs in sequence (maintain sequence to replay it in order)

So, Logs can be read across multiple readers and it enables scaling :)
Tools have evolved but the fundamentals are the same. Kafka is similar to a log playback system (distributed log processing) which helps to scale, publish and consume data.

Happy Learning!!!

Pig Overview Notes

Pig

• Primarily for semi structured data
• So called 'Pig' as it processes all kinds of data
• Pig is data flow language not a procedural language
• Map Reduce - Java Programmers, Hive - for TSQL folks, Pig (Rapid Prototyping & increased productivity)
• Pig is on client side, need not be on cluster
• Execution Sequence - Query Parser -> Semantic Checking -> Logical Optimizer (Variable level) -> Logical to physical translator -> Physical to M/R translator -> MapReduce Launcher
• Ping Concepts - Map - array, Tuple - ordered list of data ,Bag - Unordered collection of tuple
• Pig - for client side access, Hive will work only within cluster, semi structured data
• Hive - Best suited for SQL style analytics, structured data
• MR - Audio Video Analytics Map Reduce Approach is the only option

Happy Learning!!!

Hive Overview Notes

• Data Warehousing package built on top of hadoop
• Managing and querying structured data
• Apache Derby embedded DB used by Hive
• metastore_db folder for persistence of data
• Suitable for WORM - Write Once Read Many Times Access Pattern
• Core Components are Shell, Metastore, Execution Engine, Compiler (Parse, Plan, Optimize), Driver
• Tables can be created as Internal Tables, External Table (Pointing to external file)
• When Internal Tables are dropped schema + data is dropped. For external referencing tables only Schema is dropped not data. Both Internal and External tables reside in HDFS
• Data files for created tables would be available in location /user/hive/warehouse
• Partitioning in Hive - Hash Value % Number of buckets - that particular row will go into that bucket
• Partition table should always be an Internal Hive Table

Happy Learning!!!

Hadoop Ecosystem Internals

Hadoop Internals - This post is quick summary from learning session.

Data Copy Basics (Writing data to HDFS)

• Network Proximity during Data Storage (First 2 Ips closest to client)
• Data Storage size in 64MB Blocks
• Data Replication Copy by default 3
• Client gets error message when Primary Node Data Write Operation Errors
• Blocks will be horizontally split on different machines
• Slave uses SSH to connect to master (Communication between Nodes also SSH)
• Client communication through RPC
• Writing happens parallel-y, replication happens in a pipeline

Analysis / Reads (Reading Data from HDFS)

• Client -> Master -> Nearest Ips returned for Nodes
• Master knows performance utilization of nodes, It would allocate machine which is least used for Processing (Where data Copy exists)

Concepts

• Namenode - Metadata
• DataNode - Actual Data
• chmod 755 - Owner Write permission, others read and execute
• Rack - Physical Set of Machines
• Node - Individual machine
• Cluster - Set of Racks

Learning Resources

• hadoopilluminated  
• Different NOSQL DBs Comparisions 
• Apache Knox, Apache Sentry - Security Frameworks for Hadoop
• HiBench - Hadoop Benchmarking suite

Tools

• WinSCP
• Putty
• Mputty

Happy Learning!!!

APIs Good Read

I'm learning Big Data basics, checking real life architectures to understand the technology, implementation. Interesting slide on API's. I am sharing the same.


Happy Reading!!!

Exploring Hortonworks Sandbox - Part I (on Windows 7)

Setup Steps

1. Downloaded Virtual Box from Link
2. Howtonworks windows tutorial Link
3. Download 1.8GB Hortonworks Sandbox from Link 
4. After Configuring it, Ran through the first tutorial - Link
5. Started the Server and Open-up in browser IP address http://192.168.56.101:8000/ from Win7 Machine
6. Sandbox was setup and configured to use 192 series IP Address. Was able to use the Win7 browser interface to perform file upload, query operations
7. Credentials to logon on Server Login: root, Password: hadoop
8. Command to shutdown is poweroff

First Example Notes

• Downloaded example data from Link 
• Upload worked in Google Chrome not in IE
• Poweroff is the command to poweroff the sandbox machine
• Uploading data, running basic Select queries worked fine

More Info Tutorials

My Feedback

• Impressive easy setup and easy to use
• Got Started in < 2 hrs
• Good Learning Start

More Reads

WTF does a Data Scientist do all day?

How do I become a data scientist?

What are some software and skills that every Data Scientist should know?

Read Quote of Joe Blitzstein's answer to Data Science: What is it like to design a data science class? on Quora

Read Quote of Nishant Neeraj's answer to Big Data: What should be ideal size, skill set and composition of team for a successful Big Data implementation in an organization? on Quora

Read Quote of Sean Owen's answer to Job Interviews: How can a computer science graduate student prepare himself for data scientist/machine learning intern interviews? on Quora

Read Quote of Pronojit Saha's answer to Data Science: What are some software and skills that every Data Scientist should know? on Quora

Read Quote of Ye Zhao's answer to How do I become a data scientist? on Quora

How does one begin to learn data science?

Harvard Data Science Course  

Software engineer's guide to getting started with data science


Happy Learning!!!