pyCon India 2013 - Day 2 Session Notes

Please find second day session notes

Session #1 – Rasberry PI basics by Sudar Muthu

Good basic session. Speaker presented the content and demo very well. Notes from the session

• Simplest helloworld program on Rasberry PI is a light blink program
• Speaker also spoke about controlling devices
• Using PWM (Pulse width modulation) devices can be managed
• PWM.py – pull up (Higher Voltage), pull down (Lower Voltage)
• Protocols – I2C, SPI, Serial. These protocols can be used to talk to devices
• Interacting with web cam using PyGame

More Reads -  Distributed Computing Tutorials, Author website – HardwareforFun

Session #2 – Robotics Demo

•  ROS (Robotics Operating System)
• Author used RasberryPi and arduino Node

Tools

• Speech Synthesis – Festival and pyFestival
• Speech Recognition – Gstreamer, Pocket Sphinx
• Artificial Intelligence – AIML, pyAIML (Artifical Intelligence Markup language)
• GUI – QT and pyQT

Author site – Technolabz, Lentin Joseph

Session #3 – arduino and Internet of Things

Arduino – Open Source Electronics Prototyping platform

Advantages of arduino

• Easy to use
• Cheap
• Open Hardware
• Open Documents

Components

• Hw: device : electronic prototyping board
• Sw:bootloader
• Sw:libraries
• Sw: IDE
• Interfacing – Connected to computer using Bluetooth / USB

Tools

• pySerial Libarary
• arduino shows up in computer as serial device
• Library - http://code.google.com/p/permata/

Use Cases

• Talk over Serial, RF, Ethernet
• Attach Sensor and relay other readings
• Attach Actuators and make things move
• Connecting devices through web
• Security Sensor, email on touch
• Author - Avik Dhupar
• http://www.arduino.cc/

Session #4 - Testing tools Sessions (Open Source Tools)

• Fabric for distributed testing (This deployment tool can be used for distributed computing testing  )
• STAF IBM Test Automation Framework Tool -
• Nitrate Test case management tools -
• Test link – Test case management tool
• Beaker Project – Managing Automated Tests

Session #5 – Web Scraping

• Author used http://scrapy.org/
• Pablo Hoffman is the scappy developer
• Author Anuvrat Parashar provided examples on crawaling web, collecting data, extracting information from collected data 

More Tools

Jaunt Java Web Scraping & Automation

CasperJS is a navigation scripting & testing utility 

Happy Learning!!!

Day1 - pyCon2013 Notes

Hello World!!

After a long time updating my notes, learning’s in this post.

Some good one liner’s from Pycon2013

• WLT – We Love Typing
• DRY – Don’t Repeat Yourself

Today I attended Pycon 2013. This was my first python focussed learning session. Sessions were pretty good. Summarizing my notes of today's session

Session #1 - The first session was log management in inmobi 

• They have a framework built to store, manage all kinds of logs (Web Server Logs, DB Logs, Application logs, Cron Tab Logs) 
• Three steps were followed in collecting logs
• Collection of Logs – Apply Patterns – Fetch Logs
• Tools used include logstash, grok (ships with parse patterns) - http://logstash.net/ 

Author also provided other alternatives and tools for implementation

• Transport – flume, rsyslog, conduit, scribe
• Search and Analytics – hadoop, graylogz, elsa
• Storage – HDFS, Cassandra, Elastic Search
• Apache Falcon

Jordan Sissel is the brain behind logstash. His video session is also available in link

Session #2 – Django Framework beyond Basics

It was a well presented session. Although this was my first session on Django I could get the feel of the framework. Presenter Arun (http://Arunrocks.com)

Django – Web Framework. This is built on principles of Rapid App Development. How a typical web request is handled in Django framework 

• Models – Matter
• Views – Thinkers
• Templates – Should be dumb

Good Things about Django Framework

• Good Admin Interface
• Security
• Great Documentation
• Stable
• Batteries Included

Basic Definitions of Query Set (Object that interfaces with DB), Media Separate etc. Python implements class based views. Earlier it was function based generic views

Tools Discussed

• IDE – Pychan, PyDev, Emacs, Vi, Sublimetext
• Deployment Tools – Chef, Puppet, fabric
• Security Checks – ponycheckup.com for Django based sites
• ORM – SQLAlchemy

Session #3 – Third Session was Rapid development & integration of Real Time Communication in Websites

The session as pretty interactive. Demonstrating realtime video chat using google webrtc. https://github.com/cjgiridhar

Demonstrated the setup

• End User Requests sent to Tornado Web Server
• Using Web Socket / Ajax communication happens (video chat / live chat)
• Chat messages saved in Redis database

Tools

• Webserver – flask, bottle, tornado
• Javascript and webRTC

Chetan’s website - link

More Links - pythontutor

Happy Learning!!!

HSQLDB - Getting Started

This post is on learning HSQLDB. HSQLDB is Rdbms db written in java

1. Download HSQLDB from link 
2. Getting started guide useful from link 
3. Java is already installed on my laptop. Adding JAVA_HOME , PATH and CLASSPATH provided in link

Lets get started and try out some basic examples

Step 1 - To Start and create a DB

  

From command line

 

 Command text -  C:\Program Files\Java\jre7\bin>java.exe -cp "E:\HSQLDB\hsqldb-2.3.0\hsqldb-2.3.0\hsqldb\lib\hsqldb.jar" org.hsqldb.server.Server --database.0 file:mydb --dbname.0 xdb

Step 2 - DB files would be created as below

 

 

Step 3 - Opening the DB Manager

 

 

Command text - C:\Program Files\Java\jre7\bin>java.exe -cp "E:\HSQLDB\hsqldb-2.3.0\hsqldb-2.3.0\hsqldb\lib\hsqldb.jar" org.hsqldb.util.DatabaseManagerSwing

 

Command text - Connecting to DB Instance

jdbc:hsqldb:hsql://localhost/xdb

Step 4 - Basics on Table Creation

The three types of persistent tables are MEMORY tables, CACHED tables and TEXT tables

• Memory Tables - Data stored in Files
• Cached Tables - Cached detail remains in memory not in File
• Text Files - Use CSV Supported Files

Step 5 - Table Creation

 

CREATE TABLE PUBLIC.TEST_TABLE

 (COL1 INTEGER NOT NULL,

 COL2 VARCHAR(25) NOT NULL,

 PRIMARY KEY (COL1))

 

Step 6 - Load Data and Select Query

 

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 10, 'Test')

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 20, 'Ram')

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 30, 'Raj')

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 40, 'Ravi')

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 50, 'Raja')

 

SELECT * FROM "PUBLIC"."TEST_TABLE"

 

Step 7 - MVCC Basics

 

Reading only committed data, Driven by isolation level settings. More details in link

Supported Isolation levels

• SET TRANSACTION READ ONLY
• SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
• SET TRANSACTION READ WRITE, ISOLATION LEVEL READ COMMITTED 

Step 8 - MVCC Example

Start Two Instances of Data Manager

Window1 - Run below query

SET TRANSACTION ISOLATION LEVEL READ COMMITTED;

SET DATABASE TRANSACTION CONTROL MVCC;

set autocommit false;

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 10, 'Test')

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 20, 'Ram')

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 30, 'Raj')

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 40, 'Ravi')

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 50, 'Raja')

commit

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 70, '7Ravi')

INSERT INTO "PUBLIC"."TEST_TABLE"( "COL1", "COL2" ) VALUES ( 80, '8Raja')

 

Window2 - Run below query

SET TRANSACTION ISOLATION LEVEL READ COMMITTED;

SELECT * FROM PUBLIC.TEST_TABLE;

The Result from Window2 will not include 70, and 80 the uncommitted records

Step 9 - Explore the system created files - mydb.script, mydb.log file using notepad, You would see details on DB Settings and properties

References

• https://wiki.documentfoundation.org/images/c/c9/Hsqldb_guide.pdf
• Connecting HSQLDB using Java - link
• Example code for embedded HSQL for java - link
• http://hsqldb.org/doc/2.0/guide/dbproperties-chapt.html
• http://hsqldb.org/web/hsqlFAQ.html
• http://alvinalexander.com/java/jwarehouse/hsqldb/testrun/hsqldb/TestSelfTransaction.txt.shtml
• http://spectral.mscs.mu.edu/tools/hypersonicsql/doc/hsqlBeginnersGuide.html#intro

Happy Learning!!!

  

Very Good Tech Talks on Map Reduce Algorithms

One more good Session Hadoop Training: MapReduce Algorithms


More Reads
MapReduce Patterns, Algorithms, and Use Cases
The Map/Reduce design pattern demystified


Happy Learning!!!

TSQL Learning

This post is based on very good reading from Article T-SQL Misconceptions - JOIN ON vs. WHERE. These are new learning's for me


1. For Readability - Refrain from using search arguments in the ON clause, and use the WHERE clause instead
2. When Outer Join is Converted to Inner Join - Adding references to the table in the right side of a JOIN to the WHERE clause will convert the OUTER JOIN to an INNER JOIN. I'm trying out Example Code to reproduce the scenario. This Shows the behavior when Left Join in Query is Converted to Inner Join during Execution.


Step 1 - Create Tables

CREATE TABLE Table1(Col1 INT IDENTITY(1,1) PRIMARY KEY, Col2 VARCHAR(40), Col3 VARCHAR(50))

CREATE TABLE Table2(Col1 INT IDENTITY(1,1) PRIMARY KEY, Col2 VARCHAR(40), Col3 VARCHAR(50))

Step 2 - Populate Data

DECLARE @J INT

 SET @J = 1
 WHILE 1 = 1
 BEGIN
 IF @J < 1000
     INSERT INTO Table1(Col2, Col3)
     VALUES((CONVERT(VARCHAR(20),@J)+'VALUE'), (CONVERT(VARCHAR(20),@J)+'VALUE'))
     INSERT INTO Table2(Col2, Col3)
     VALUES((CONVERT(VARCHAR(20),@J)+'VALUE'), (CONVERT(VARCHAR(20),@J)+'VALUE'))
     SET @J = @J + 1
     IF @J > 1000
     BREAK;
 END


Step 3 - Run Queries

SELECT

 N1.Col1, N2.Col2
 FROM Table1 N1
 LEFT JOIN Table2 N2
 ON N1.Col1 = N2.Col1
 WHERE N2.Col1 = 100
 

SELECT

 N1.Col1, N2.Col2
 FROM Table1 N1
 LEFT JOIN Table2 N2
 ON N1.Col1 = N2.Col1
 WHERE N1.Col1 = 120


Step 4 - Execution Plan

Happy Learning!!!

Big Data Updates

Interesting Big Data Updates


[Good Read] - Big Data Analysis Takes a Big Bite Out of HPC Workloads: IDC
Summary - High Performance Servers used to run / benchmark big data workloads

[Learning Resource] - Hadoop 101: Programming MapReduce with Native Libraries

[Good Read] - Big Data Transforming Traditional DW Ecosystem - Why Hadoop and Solr in DataStax Enterprise?

Summary 

• NOSQL feasible alternative competing OLTP Space
• Big Data Replacing OLAP Ecosystem

Happy Reading!!!

MYSQL Exercises

Couple of learning exercises while working on MYSQL. Earlier MYSQL post please refer link
Learning #1 - Load data from flat file
Created a Temp table, Loaded bulk data using command - LOAD DATA LOCAL INFILE

CREATE TABLE TestTable2 ( A INT NOT NULL, B INT NOT NULL, C INT NOT NULL, D INT NOT NULL );

Dataset looks as below

LOAD DATA LOCAL INFILE 'E:\\DataUpload50K.txt' INTO TABLE TestTable2 FIELDS TERMINATED BY ','  LINES TERMINATED BY '\n';

Reference - link1 , link2
Learning #2 - Enable Remote Access to MYSQL

Execute the Grant All Privileges command by specifying username, IP Address of remote host and password to logon
GRANT ALL PRIVILEGES ON *.* TO 'username'@'XX.XX.XX.XX' IDENTIFIED BY 'pwd';
Encountered error codes 1130, 1045
Reference - link1, link2, link3

Learning #3 - Show System Configuration Settings
Command - SHOW VARIABLES
Reference - link1

Happy Learning!!!

heidisql - SQL Interface to MSSQL and MYSQL

This quick post is based on tool shared by my colleague Ambuj. Heidisql - SQL Editor for MSSQL and MYSQL

Step 1 - Pretty simple and quick to install. Downloaded and Installed it from link

Step 2 - You can setup sessions and connect to MSSQL / MYSQL instances. I have used SSMS, Atlantis SQL Explorer. This single interface for both MYSQL and MSSQL is very useful.

Happy TSQL coding !!!

Compiled List of Short Talks on Career Success

Happy Learning!!!

TSQL Formatting Tools

Back to blogging after a month long break. Things were quite busy @ work. This post is to focus on TSQL formatters. There are several code formatters that I had come across recently.

1. SQL Formatter for Notepad++ - Download and install Notepad++ plugin from link. Copying the DLL's and checking the Notepad++ Option listed below

Step 1. Copy the DLL's in plugin directory of Notepad++



Step 2. NotePad++ SQL Formatter Options

2. Online SQL Formatter - Link

3. PoorSQL - Online SQL Formatter - Link

4. SSMS Tools pack - SSMS Addons - Link

5. Instant SQL Formatter - Online version - Link

6. Atlantis Interactive SQL Server IDE - Very good IDE for MSSQL - Link

7. SQLInform - SQL Online formatter - Link

Other Free Useful Tools
1. Programmers Notepad - Link
2. TextPad - Link
3. Notepad++ - Link

Happy Learning!!!