Weekend Reading - Resilient Distributed Datasets

 Paper Link 

Key notes

• Resilient Distributed Datasets (RDDs) - a distributed memory abstraction that lets programmers perform in-memory computations on large cluster
• Data reuse is common in many iterative machine learning and graph algorithms, including PageRank, K-means clustering, and logistic regression. 
• RDDs are a good fit for many parallel applications because these applications naturally apply the same operation to multiple data items. 
• RDDs are fault-tolerant, parallel data structures that let users explicitly persist intermediate results in memory, control their partitioning to optimize data placement
• If a partition of an RDD is lost, the RDD has enough information about how it was derived from other RDDs to recompute
• Formally, an RDD is a read-only, partitioned collection of records. RDDs can only be created through deterministic operations on either (1) data in stable storage or (2) other RDDs
• RDD has enough information about how it was derived from other datasets (its lineage) to compute its partitions from data in stable storage
• The main difference between RDDs and DSM is that RDDs can only be created (“written”) through coarse-grained transformations, while DSM allows reads and writes to each memory location

Keep Thinking!!!

Day #261 - Setting up Spark on my Ubuntu - Big Data Setup - Part V

Happy Data Thinking!!!

Spark Overview

I remember Spark keyword appeared during Big Data Architecture discussions in my Team, I never looked more into Spark. Session by Jyotiska NK on Python + Spark: Lightning Fast Cluster Computing was a useful starter about Spark. (slides of talk)

Spark 

• In memory cluster computing framework for large scale data processing
• Developed using scala with Java + Python APIs
• This is not meant to replace hadoop. It can sit on top of Hadoop
• References on Spark Summit for Slides / Videos to learn from past events - link 

Python Offerings

• PySpark, Data pipeline using spark
• Spark for real time / batch processing

Spark Vs Map Reduce Differences

This section was session highlighter. They way how data is handled between Map Reduce Execution and Spark Approach is Key.

Map Reduce Approach - Load Data from Disk into RAM, Mapper, Shuffler, Reducer are the different approaches. Processing is distributed. Fault Tolerance is achieved by replicating data 

Spark - Load data in RAM, Keep it until you are done, Data is cached in RAM from disk for iterative processing. If data is too large, rest is spilled into disk. Interactive processing of datasets without having to load data in memory. RDD (Resilent distributed datasets)

RDD - Read Only collection of objects across machines. On losing information this can still be recomputed. 

RDD Operations

• Transformations - Map, Filter, Sort, flatmap
• Action - Reduce, Count, Collect, Save to local data in disk. Action usually involves disk operations

Read Quote of Ramzi Alqrainy's answer to What are use cases for spark vs hadoop? on Quora

More Reads
Testing Spark Best Practices
Gatling - Open Source Perf Test Framework
Spark Paper

Happy Learning!!!