Learning Ruby - Part I

In continuation with previous post Ruby Getting Started we will look at examples for loop, multiplication tables, File I/O operations. I am referring to Beginning Ruby by Peter Cooper for learning Ruby Programming Syntax

Example program multiple.rb

x = 1;

#Example 1 - Multiplication using 10.times;
puts "From 1 to 10 Multiplication of 5";
10.times do
       a = x*5;
       puts a;
       x = x+1;
end;
#Example 2 - Loop using upto;
puts "From 10 to 20 Multiplication of 5";
1.upto(10) do
       a = x*5;
       puts a;
       x = x+1;
end;
puts "Arrays"
#Example 3 - Arrays list all elements;
arrayA = [1,2,3,4,5];
arrayA.each {|y| puts y};
puts "Array & Strings"
arrayB = [1,2,"One","Two","Three"];
arrayB.each {|y| puts y};
#Example 4 - List the program line by Line;
File.open("d:\multiply.rb").each { |line| puts line }

 

 

Happy Learning!!! 

C# Fundamentals

This time is all about fundamentals of C# . Going beyond theoretical basics.

Tip #1 - Why Value Types are stored in Stack and not in Heap
This blog post (The Truth About Value Types) was the answer. It is a recommendation that CLR does on your behalf to store value types in stack. Short Lived storage (value types) live in stack. Long duration (reference types) live in heap

Tip #2 - Garbage Collection process, Phases Involved

Mark, Sweep and Compact. More details refer The Stack Is An Implementation Detail, Part Two

Tip #3 - Abstract Class VS Interfaces

• Interface - Can define methods but not implement them
• Abstarct Class - Can define/implement methods/members but cannot instantiate abstract classes

using System;

namespace examplecode

{

    abstract class AClass

    {

        int aMember;

        public void printVal()

        {

            Console.WriteLine("Example AbstractClass");

        }

    }

    interface iTest

    {

        void printValInterface();

    }

    class Program:AClass,iTest

    {

        public void printValInterface()

        {

            Console.WriteLine("Example printValInterface");

        }

        public static void Main()

        {

            Program RunProgram = new Program();

            RunProgram.printVal();

            RunProgram.printValInterface();

            Console.ReadLine();

        }

    }

}

 

Tip #4 - Why multiple inheritance is not supported in .NET

Answer from stackoverflow question (Why is Multiple Inheritance not allowed in Java or C#?) is realistic and impressive. The benefits are too less and adds a lot of complexity to support multiple inheritance.

 

Tip #5 - Disassembly Good Example

Are these objects's references on the Stack or on the Heap?

 

Couple of MSDN blog posts were very very impressive. A must read for every developer

• The Stack Is An Implementation Detail, Part One
• Why have a stack?
• Debunking another myth about value types

 

Happy Learning!!!

Ruby Getting Started

This post is about learning ruby. Tried it online using rubyfiddle. rubbyfiddle did not work with global variables i.e Variables using $ symbol.

Basic Commands

Very Usual Hello World Example

Dynamic Typing

Global variables didn't work using ruby fiddle. Installed ruby for Windows. Using Interactive Ruby Option under Programs->Ruby tried below examples

While Loop

Next example is trying out for loop. Below is the test file

Start Command Prompt with Ruby. Run the test file

Pretty easy to learn. Looking forward for more interesting posts.

More Reads
Ruby Tutorial with Code Samples
A Wealth Of Ruby Loops And Iterators
Ruby Procs And Lambdas (And The Difference Between Them)
A Unit Testing Framework In 44 Lines Of Ruby


Happy Learning!!!

Algorithm Analysis - Basics

This post is based on notes after attending Algorithms I classes at Coursera. Lecture on Analysis of Algorithms is very good. Couple of interesting snapshots/ questions captured during the course posted below

This can be computed to 1000000 / log (1000000) = 100000/6 = 1,66,6666. This should be the right answer I believe.

Another interesting slide - Order of Growth

• Big-O is an upper bound.
• Big-Theta is a tight bound, i.e. upper and lower bound.Link

Another Interesting Question

 

Good Explanation of Answer - Dropping Eggs from a Building - 100 Story Building Famous Google Puzzle

 

 

Happy Learning!!! 

Security Papers and Articles

Exploit Database has a huge repository of Security Papers and Articles.

Couple of interesting papers

• Distributed Denial of Service Attacks: Explain nation, classification and suggested Solutions
• Ragey’s Cross Site Scripting Worm Proof of Concept
• Buffer Overflows: Anatomy of an Exploit
• Microsoft Patch Analysis (Old but Good One)

Happy Learning!!!

SQL Server - Index Tuning Basics

SQL Server Query Tuning Session. - One more session to add to SQL Tuning list.

Useful pointers from the session. Couple of pointers are covered in previous posts as well.

Optimizer's choice of Join
Nested Loop Join

• Outer input is small
• Inner input has an index on the join key

Merge Join

• Medium to large inputs
• Sorted inputs and equality operator required

Hash Join

• Large inputs
• Requires equality operator, inputs need not be sorted

MAXDOP Option - 1 (Ensure single processor for execution)
Density - How many distinct values available in a particular column
Multi Column Index

• Index can be used to seek on second column if there is an equality operator on first column

Guildeline - Most Selective column should be first column where all other column predicates use the equality operator

Happy Learning!!!

Big Data Conference Notes - Part III

#6. Fifth Elephant Conference – Big Data Analytics @ InMobi

I would rate this as the best session in the conference. The journey of inmobi in managing growing data analytics and providing analytics @ real-time is impressive.

Gaurav gave a complete walkthrough from using perl to Hadoop, Pig and finally ended up building their own analytics platform on top of Hadoop

Scale of Data @ InMobi

• 3 billion impressions per day
• 100 primary dimensions and 300 derived dimensions
• 50 measures

Data Characteristics

• Highly Dynamic data and Analytic needs
• Frequent addition of new dimensions
• Dynamic query patterns
• Canned and adhoc reports
• Different kind of customers (Sales, Analysts, Executives)
• Canned Reports – Day in and Day out reports without any change

Journey of Analytics @ Inmobi

Beginning of Analytics

• Initially perl scripts
• Logs summarised using perl
• Perl could not handle increasing data volumes (Q2 2010)

Hadoop Adoption

 

• Map Reduce jobs written to aggregate logs and populate Database
• 3 machine Hadoop cluster setup was done
• Challenging was writing map reduce jobs took a lot of time
• With Increasing DB Views this was harden to accommodate with custom MR jobs creation

Hadoop, Map Reduce and Pig

• Pig was adopted; Pig was aggregated logs and pushing data into database
• For Complex operations custom MR jobs were written

More Analytics, More Data, Growing Measures

• Analytics was becoming increasing complex
• DB suffered ‘limited angle view’ problems
• Hive was not mature when they tried it out. Hive was resource consuming; it was not creating optimal jobs. Data transfer between mapper / reducer was not scalable
• Pig jobs were written for new requirements for fetching and loading data in DB

Realisation

He highlighted the challenges in adopting open source frameworks

• Too much customization and constant fine tuning required
• Difficult to absorb business changes while trying to customize the platform
• Different open source framework at different parts of stack, Difficult to integrate and maintain
• Pig not suited for business users

YODA (InMobi Inhouse Analytics Framework)

 

• Complete stack was custom built (ETL, Query Processor, Query builder, Visualization)
• Built on top of Hadoop
• SQL Like operations (sum, select, min, max UDF supported)
• Optimized for storage and queries for data model
• Protobuf was used for message exchange

Please view the session if you get a chance. It is amazing, Very informative Session

#7. Fifth Elephant Conference – Messaging Architecture @ facebook

• 10 TB messages / Day
• Haystack is used for storing facebook photos
• Needle in a haystack: efficient storage of billions of photos

Facebook principle is - “Choose best design not implementation But get things done fast”

LSM Trees

• Stores things in a set of trees
• High write throughput
• Recent data clustered
• Inherently snapshotted

Cassandra Vs HBASE

• HBase worked out
• Cassandra (Distributed Database)
• HDFS – Distributed Storage

#8. Fifth Elephant Conference – Recommendation Engine @ Flipkart

• Build on top of Hadoop, Cassandra, Redis, Memcache
• Cassandra for storing logs
• Map reduce jobs run to identify user browse history, common patterns
• Identified data stored in redis (key value pair based storage)
• Caching is done using memcache

Happy Learning!!!

Big Data Conference Notes - Part II

In Continuation with previous post.

#4. Fifth Elephant Conference –Cloud Story for Big Data by AWS Evangelist Joe Ziegler


This was a beginner session; there was not in-depth discussion on tools / architecture approach. Amazon is undisputed leader right now. The underlying tools/ techniques are now applied by other competitors Azure, VMware Cloud, Google Cloud. After Google, Amazon harnessed the power of Hadoop, Map Reduce (Elastic Mapreduce), S3 Storage and provide it on AWS Platform.


Data becomes so large that you need to innovate to store, process it. Bigger data is harder data, Multiple sources and multiple different formats of data. By end of 2012 2.7 Zeta bytes of data will be generated and 90% of it is unstructured.

Why Cloud ?

• Elastic (Spin off machines on need basis
• Pay per use
• No Capital investment
• Faster time to market
• Focus on Core Complexity

Cloud benefits 

• Reusable – Deploy take snapshot from cloud and use it to deploy later
• Managed Services – Managed hosted Hadoop environment @ Amazon
• Scale 
• Innovation

Cloud reduces cost of experimentation

S3 – Simple storage service

Beginner level session, Big Data and Cloud are best friend. Cloud provides infrastructure to host / run big data infrastructure. AWS offerings, Customer case studies were highlighted

#5. Fifth Elephant Conference – Real time Analytics @ flipkart

They explained custom real time analytics for supply chain orders / procurements etc.. Both log files and database are read, data processed and reflected in visual graphs.

Lot of custom tools developed for automating logs collection across servers, custom replication setup (multi threaded approach)

I have presented a rough architecture. More on these tools you can find by searching the net

All of them designed from open source framework and linux platform

• DB – Mysql
• ElasticSearch – Open source text based indexing (similar to DB)
• StatchD – Network Daemon Tool on Node.Js
• StatsD Layer – For RegEx patterns, Aggregates, Deviations

This is very interesting approach, Looking at both Database & Application Events to ensure data is loaded / monitored on both the ends.

This can still be simplified by storing in a NOSQL Database and querying on top of it

I am not sure if this would simplify their approach. It again depends on the production scenario / usage. All of this approach / Architecture can be implemented using .NET / Java / Python.

Happy Learning!!!

Big Data Conference Notes - Part I

This post is primarily notes taken during Big Data Conference - The Fifth Elephant.

#1. Fifth Elephant Conference - Crunching Big Data, Google Scale by Rahul Kulkarni

First Session was ‘Scaling Data Google Scale’ by Google Employee Rahul Kulkarni. Captured below are notes from the session

Session covered on Google App Engine, Google Compute Engine, How google manages processing huge volumes of data. The two primary factors around data processing are Compute at scale, Adhoc querying on large volume of data

Google App Engine 

• PaaS (Provided as Platform as a Service)
• Stats on Data processing volumes – 7.5B hits per day and 2 Trillion transactions per month

Google Compute Engine

• IaaS (Infrastructure as a service)
• Analytics workload targeted
• Supports Deploying your own cluster
• Example of how Genome processing (large data sets) was shared. GCE reduced computation time for genome processing significantly

Google White Papers

• Google whitepapers to checkout
• Dremel (2010)
• Drapper (2010) – For Tracing purpose
• Flume (2010) – Data Pipeline
• Protocol buffers (2008)
• Chubby (2006)

Other interesting white papers I have shared in my earlier posts

Google’s Approach for Data Processing (Adhoc Queries)

• Big Query Approach - Uses Column oriented storage 
• Supports Map reduce jobs as well (3 Phases Mapper, Shuffler, Reducer)
• Big Query Supports small joins, In case of joins the required data is moved to where column data is located

Google Cloud based Solution for Data

• App Engine (Front End)
• Big Query (Process Data)
• Cloud Storage (Data Storage)

Links Provided – developer.google.com

Key Learning’s

• Google cloud platform can be used for prototypes involving big data
• Columnar databases gaining market share for analytics (Hadapt, Vertica etc..)
• Bunch of new whitepapers I learnt from the session talk

#2. Fifth Elephant Conference – In Data We Believe Session Notes

Session by Harish Pillay from Redhat, Briefly covered on big data characteristics, opportunities, offerings from Red hat for Big Data

What is Data? 1’s and 0’s organized in a manner that provides meaning when interpreted

Structured Data Characteristics – Schema available, normalized, predictable, known

Unstructured Data Characteristics – Semistructured like log files, unorganized, no fixed schema

Redhat offerings for cloud, big data were discussed. Redhat Linux, JBOSS, Redhat storage and openshift products were highlighted.

#3. Fifth Elephant Conference – Hadoop ecosystem overview Session Notes

Session by Vinayak Hegde from InMobi. How they manage big data processing. What tools and framework they rely on for data processing

Introductory slides covering on data generated in large volumes from mobile, social networks, financial system, tweets, blogs etc..

He listed dozen open source projects for different layers involved in data processing. Listed below are projects I noted during the session. Data Stack was a very good slide

 

 

Session was full of tools used at each layer. Unfortunately presentation was cut short as it exceeded allowed duration. This tools list is a good starter kit to start exploring.

Key Learning’s  

• Open source tools that can be leveraged for custom Hadoop based cluster setup and management. These tools are a good place to get started for large scale Hadoop installations

Happy Learning!!! 

Protobuf and LRM Trees

This post is based on learning's from The Fifth Elephant - Big Data Conference Notes.

Protobuf - Many Sessions emphasised on adoption of Protobuf. Google's data interchange format. They output xml in terms of performance, simple and easy to use. (Link)

Protobuf tutorial - Link1, Link2
Using Protocol Buffers on .Net platform (Part I)
Protocol Buffers and WCF
Some internals of protobuf-net


LRM ( Left-to-Right-Minima) Trees – This was highlighted during Facebook messaging Architecture. Left-to-Right-Minima Trees. Academic paper on LRM tree available in link

From Quora - What is the expanded form of the "SSTable" abbreviation used by Google's BigTable?

An SSTable provides a persistent, ordered, immutable map from keys to values [1], where both keys and values are arbitrary bytes. Apache HBase, the implementation of SSTable is called HFile.
Where can I get a paper about SSTables?

Log-Structured Merge-Tree - The Log-Structured Merge-Tree - writes are always fast regardless of the size of dataset (append-only), and random reads are either served from memory or require a quick disk seek

From Link

Before going to the topic, Refresher for B and B+ Trees

• B-Trees: What is an intuitive explanation of the B-tree data structure?
• What is a B+ Tree?
• What are the advantages of B+ Tree over B Tree

Happy Learning!!!