2012 Big Data Interesting Milestones

2012 was a fantastic year in terms of Big Data and Cloud Adoption. Massive Big Data focus and Products focussed on Big Data Analytics. Fortunately the world did not end as per Mayan Prediction.

 

How Traditional Players catch up in Big Data Space

• Traditional RDBMS Vendors MSSQL, ORACLE offering their own NOSQL Solutions, Hadoop Connectors to leverage the power of Hadoop and enhance their BI offerings
• IBM BigInsights Hadoop based Analytics Platform, Informatica and Microstrategy enhanced their Big Data Capabilities this year by providing integration with Hadoop Ecosystem
• Acquire Hadoop based Startups to catch up with Big Analytics

Real Time Analytics Trend in 2012 

• Cloudera launched Impala platform to provide Real Time Analytics
• Real Time Analytics based on Twitter Storm

Big Data Fifth Elephant Conference provided good insight on Big Data Use Cases and Adoption rates

Big Data Products

• Good Read - Big Data - Promising Startups
• Amazon Announced Redshift DataWarehousing as a service
• Google Launched BigQuery, Cloud Business Intelligence Service for Big Data

Indian Startups Working on Big Data / Leveraging Hadoop Ecosystem for Analytics

• Inmobi
• Qubole
• Flipkart

2013 Focus 

• Big Data Analytics Adoption across Industries (Verticals)
• Increased Implementation / Adoption of Big Data Products
• Columnar databases gaining more focus (Vertica, H-Store)
• Databases to watch out in 2013 - Hadapt (HadoopDB), H-Store(OLTP), F1(Google Distributed RDBMS) 

Interesting Big Data Reads

• Classifying Today’s “Big Data Innovators”
• The future is now: 10 startups leading the way in ‘big data’
• Major Data Warehousing Events of 2012 (and Predictions for 2013)

Inspiring Tweet to end the post



Happy Learning!!!

  

Python Basics

Two Coursera classes (An Introduction to Interactive Programming in Python, Learn to Program: The Fundamentals) provide very good python beginner learning materials. Python is a Interpreted Language

Quick Summary and Python Notes

• Guido van Rossum is author of python, Good Wikipedia read about him in link  
• Python Practice Book
• Notes from Advaced Python Workshop
• Online Editor available at CodeSkulptor
• Also check Very Good Question Is Python Interpreted or Compiled?
• Good Book to Get Started - A Byte of Python -Swarooph
• Learning Python the hard way

Basic Examples and beginner notes posted based on course notes. All Code can be tried in CodeSkulptor.  There are a few syntax differences but this is similar to C programming language.

Example1 - Arithmetic Expressions

• Basic Arithmetic Expressions
• Variables, Assignments, Sample Exercise in link

All Scripts can be run in FireFox / Chrome in CodeSkulptor (Not in IE)

Example 2 - Working with Functions
Built In functions - max, abs

Example 3 - Custom Functions

• Example function to sum up two numbers

Example 4 - If Statement - Conditons , Else If and Else Operator

Example 5 - For Loop - Iteration

Example 6 - While Loop iteration

Happy Learning!!!

Tool Developer Notes

Tip #1 - Sending Email on Error using Log4NET

Tip#2 - Append XML Node in C#

Before Code Execution

Post Code Execution

using System;
using System.Globalization;
using System.Data;
using System.IO;
using System.Xml;
namespace ExampleCode
{
    public class ExampleCode
    {
        static void Main()
        {

            //Open XML
            XmlDocument xmlPreCountFileData = new XmlDocument();
            xmlPreCountFileData.Load("E:\\abc.xml");
            XmlElement elmXML = xmlPreCountFileData.CreateElement("Node");

            //Append Node
            //Append Existing XML Node
            string xmlDataElement = @"<A>40</A>";

            elmXML.InnerXml = xmlDataElement;
            xmlPreCountFileData.DocumentElement.AppendChild(elmXML);
            xmlPreCountFileData.Save("E:\\abc.xml");
            Console.ReadLine();
        }
    }
}



Tip #3 - Thread Enhancements in .NET 4.0

Threading made easy in .NET 4.0
C# Mulththreading Improvements in .NET 4.0
.NET 4.0 and System.Threading.Tasks


Happy Learning!!!

Big Data Trends

Excellent Presentation on Big Data Trends..

Big Data Trends from bigdatalandscape

Happy Reading!!!

Hadoop Learning questions from Quora

Quora has a good collection of questions on Hadoop ecosystem, architecture and beginner level questions. Bookmarking some of interesting questions

• HBase: Why does HBase depend on HDFS?
• How much time did it take to develop Hive at Facebook?
• Good Notes from Workshop on Algorithms for Modern Massive Data Sets
• What is the best way to learn Hadoop and Cassandra for a starter?

Recent Updates

• Cloudera Impala: Real-Time Queries in Apache Hadoop, For Real
• Does Cloudera Impala have any drawbacks when compared with Hive?
• How does Cloudera Impala compare to Vertica?

More Reads

• HBase setup (0.20.0)
• Below is a code sample of how to read from and write to HDFS in java
• Get a hadoop cluster running in 20 Minutes
• Hadoop and Hive
• How to Load Text file into Hbase in Linux Machine

Happy Learning!!!

Hadoop Single Server Setup on Ubuntu

In continuation with previous post we will install java, hadoop, configure map reduce jobs on ubuntu. The step by step details are based on details provided in link.

Many thanks to Arun and Pragalathan for helping me learn linux :)

Step 1 - Installing JDK

Step 2 - Create Hadoop Group

Step 3 - Create hduser

Step 4 - Add sshkey

Step 5 - Enable SSH

Step 6 - Install SSH (Had some issues so reinstalled it)

Step 7 - ssh to localhost

Step 8 - Disable IPV6 (Modify the entries based on steps provided in link)

Step 9 - Download Hadoop

Step 10 - Move it to Hadoop folder

Step 11 - Make permission owner hduser

Step 12 - bashrc changes

Step 13 - Check Java and Hadoop Home Path

Step 14 - Hadoop env changes

Step 15 - Modify custom xml files in below folder

Step 16 - Start Service

Step 17 - Copy files for working example

Step 18 - Hadoop file system details

Step 19 - Copy Files to Hadoop File System

Step 20 - List uploaded files

Step 21 - Run wordcount example jar

Step 22 - Check results output file

More Reads - How to Install Hadoop on Ubuntu 13.10

Happy Learning!!!

Ubuntu Installation - Part I

Guideline for setup is based on Steps provided in link

Setting it on Ubuntu desktop, step by step details with Snapshot. Download installer from link
Step 1 – Installing Ubuntu

Step 2 – Ubuntu Installation

 

Step 3 – Drive to Install

Step 4 – Provide Credentials

Step 5 – Configure Network

Follow the next steps and complete installation and logon to system.

Happy Learning!!!

Interesting Post - Learn to Code (Startup Tips)

This post is to highlight few lines from Post on Learning to Code. If you have plans to do a startup it is useful and advantageous if you know to write code, passion for technology.

Happy Learning!!!

Algorithms Course Notes - Part II

Algorithms Course Notes from Coursera

Binary Search Trees

• Binary Tree in Symmetric order (Nodes - Contain info, two links)
• Each node has left and right tree, both can be null
• Every Node Key Larger than keys in left sub tree
• Every Node key smaller than keys in right sub tree
• Inserts - Find a null link by search eligible position and insert it there

B Trees

• General Model for external storage
• Internal node key guide search
• External node has client key
• Insert at Bottom null node
• If Nodes are full it will split to allow insert
• Variants of BTree is used in DBs (B+ Tree, B* Tree)

RBT - Red Black Tree Tracks every simple path from a node to a descendant node with same number of black nodes
Red Black Trees

• Internal Left Leaning links to glue 3 nodes
• No Nodes have two red links connected
• Every path from root to null link has same number of black links
• Use a flag color to denote red or black link in implementation

Heap Sort

• Largest of all keys is the root

Reposting Important Summary Slides

Happy Learning!!!

Jenkins Installation on Windows Server

This post is about issue encountered today and resolution for the same. Below error was returned by Jenkins windows installer

There was no log file associated with it. To debug it we need the log file for this installation. Below command would provide us log file

After few google searches enable .NET 3.5 features, this did not resolve the error

By closely examining the errors you would see error

ExecFirewallExceptions: Installing firewall exception2 Jenkins (C:
\Program Files (x86)\Jenkins\jre\bin\java.exe)

Fix was it was provided in link. By enabling windows firewall installation was successful.

Posting it hoping it is useful for larger audience.

Happy Learning!!!

Learning Ruby - Part I

In continuation with previous post Ruby Getting Started we will look at examples for loop, multiplication tables, File I/O operations. I am referring to Beginning Ruby by Peter Cooper for learning Ruby Programming Syntax

Example program multiple.rb

x = 1;

#Example 1 - Multiplication using 10.times;
puts "From 1 to 10 Multiplication of 5";
10.times do
       a = x*5;
       puts a;
       x = x+1;
end;
#Example 2 - Loop using upto;
puts "From 10 to 20 Multiplication of 5";
1.upto(10) do
       a = x*5;
       puts a;
       x = x+1;
end;
puts "Arrays"
#Example 3 - Arrays list all elements;
arrayA = [1,2,3,4,5];
arrayA.each {|y| puts y};
puts "Array & Strings"
arrayB = [1,2,"One","Two","Three"];
arrayB.each {|y| puts y};
#Example 4 - List the program line by Line;
File.open("d:\multiply.rb").each { |line| puts line }

 

 

Happy Learning!!! 

C# Fundamentals

This time is all about fundamentals of C# . Going beyond theoretical basics.

Tip #1 - Why Value Types are stored in Stack and not in Heap
This blog post (The Truth About Value Types) was the answer. It is a recommendation that CLR does on your behalf to store value types in stack. Short Lived storage (value types) live in stack. Long duration (reference types) live in heap

Tip #2 - Garbage Collection process, Phases Involved

Mark, Sweep and Compact. More details refer The Stack Is An Implementation Detail, Part Two

Tip #3 - Abstract Class VS Interfaces

• Interface - Can define methods but not implement them
• Abstarct Class - Can define/implement methods/members but cannot instantiate abstract classes

using System;

namespace examplecode

{

    abstract class AClass

    {

        int aMember;

        public void printVal()

        {

            Console.WriteLine("Example AbstractClass");

        }

    }

    interface iTest

    {

        void printValInterface();

    }

    class Program:AClass,iTest

    {

        public void printValInterface()

        {

            Console.WriteLine("Example printValInterface");

        }

        public static void Main()

        {

            Program RunProgram = new Program();

            RunProgram.printVal();

            RunProgram.printValInterface();

            Console.ReadLine();

        }

    }

}

 

Tip #4 - Why multiple inheritance is not supported in .NET

Answer from stackoverflow question (Why is Multiple Inheritance not allowed in Java or C#?) is realistic and impressive. The benefits are too less and adds a lot of complexity to support multiple inheritance.

 

Tip #5 - Disassembly Good Example

Are these objects's references on the Stack or on the Heap?

 

Couple of MSDN blog posts were very very impressive. A must read for every developer

• The Stack Is An Implementation Detail, Part One
• Why have a stack?
• Debunking another myth about value types

 

Happy Learning!!!

Ruby Getting Started

This post is about learning ruby. Tried it online using rubyfiddle. rubbyfiddle did not work with global variables i.e Variables using $ symbol.

Basic Commands

Very Usual Hello World Example

Dynamic Typing

Global variables didn't work using ruby fiddle. Installed ruby for Windows. Using Interactive Ruby Option under Programs->Ruby tried below examples

While Loop

Next example is trying out for loop. Below is the test file

Start Command Prompt with Ruby. Run the test file

Pretty easy to learn. Looking forward for more interesting posts.

More Reads
Ruby Tutorial with Code Samples
A Wealth Of Ruby Loops And Iterators
Ruby Procs And Lambdas (And The Difference Between Them)
A Unit Testing Framework In 44 Lines Of Ruby


Happy Learning!!!

Algorithm Analysis - Basics

This post is based on notes after attending Algorithms I classes at Coursera. Lecture on Analysis of Algorithms is very good. Couple of interesting snapshots/ questions captured during the course posted below

This can be computed to 1000000 / log (1000000) = 100000/6 = 1,66,6666. This should be the right answer I believe.

Another interesting slide - Order of Growth

• Big-O is an upper bound.
• Big-Theta is a tight bound, i.e. upper and lower bound.Link

Another Interesting Question

 

Good Explanation of Answer - Dropping Eggs from a Building - 100 Story Building Famous Google Puzzle

 

 

Happy Learning!!!