Frequently Used Scripts & Notes

TSQL Reusable scripts - bookmarking the same

Tip #1. What is the command to truncate a SQL Server log file?

Tip #2. Settings to Capture Deadlock Trace in SQL Server Logs
DBCC TRACEON (1204, -1)
DBCC TRACEON (1222, -1)

Tip #3. Checking SQL Server Version

select SERVERPROPERTY('productversion'),SERVERPROPERTY('productlevel'),SERVERPROPERTY('edition')

Tip #4. Creating a custom firefox template

Step 1. Windows -> Run specify below command
firefox.exe -ProfileManager -no-remote

Step 2. Create custom profile

Step 3. Command to run selenium using template
Example:  java -jar selenium-server.jar -firefoxProfileTemplate “<Selenium Profile Directory>”

Usage:  java -jar selenium-server.jar -firefoxProfileTemplate C:\Users\Administrator\AppData\Roaming\Mozilla\Firefox\Profiles\sd88nd1n.FRProfile

Step 4. With Logs captured below is modified steps

Usage: java -jar C:\\SeleniumServer\\selenium-server-standalone-2.25.0.jar -port 5555 > C:\\ReportLogs\\\SeleniumServerStatus.txt 2>&1 -firefoxProfileTemplate "C:\\Users\\Administrator\\AppData\Roaming\\Mozilla\\Firefox\\Profiles\\ku6gbc8j.FRProfile"

References Link1


Happy Learning!!!

mTail - Windows Real time Log Monitoring Tool, Log Searching - Free Tools

This post is for mTail - Log monitoring tool on windows. I have used it it is very good. The benefits are

• Monitor multiple Servers with multiple instances of this tool concurrently
• Simple and easy to use
• Start / Stop option to stop in case you have viewed error

Free tools for File Search / Exception Checking in logs

• Agent Ransack
• Textpad (Find in Files Feature)
• Notepad++

Happy Learning!!

Interesting Bug

Sharing one Unique Interesting bug identified during testing

Case #1 - Property_Id values mismatch was the primary issue. Two tables were defined in two databases. The scenario is Property_Id on Database A is based on master tables defined , example Master A. Issue was## mapping of MasterA was used in DatabaseB instead of MasterB. It was difficult to identify as Property_Id values were always overlapping for a smaller set of input data

Database A

MasterA

Property_Id Value

1    A

2    B

3    C

.......

10    D

TableA

Property_Id  Local_Id (Identity Column) Value

1 1

2 2

3 3

1 4

2 5

2 6

Database B

MasterB

Property_Id Value

10    A

20    B

30    C

TableB (Expected)

Property_Id          Local_Id (Identity Column) Value

10 1

20 2

30 3

10 4

20 5

20 6

TableB (Actual)

Property_Id           Local_Id (Identity Column) Value

1 1

2 2

3 3

10 4

2 5

2 6

The bug was not easily identified as Property values in MasterA and MasterB overlapped.  The values were matching in most cases for a smaller dataset.

Happy Learning!!!

Weekend Learning - Good Session - Taming Big Data with Berkeley Data Analytics Stack

Good Session - Taming Big Data with Berkeley Data Analytics Stack 

Notes captured from the session

Big Data Use Cases (Making personalized decisions for each customer, Analyse data trends)

Data Processing Goals

• Earlier Trend - Analyse historical data
• Current Trend - Real time data processing
• Goal - Sophisticated data processing (Trend analysis, Anomaly detection)

Open Analytics Stack

• Apps - Data Analysis, Mining, Decision Driven Apps
• Data Processing - HBase, Hive, Hadoop
• Storage - HDFS
• Infrastructure - Cluster

Goals of Open Analytics Stack

• Support batch, interactive and stream processing

Implementation Notes

• Store data in memory (SSD's, 512GB of RAM)
• FB / Yahoo / Bing - Some very large jobs but vast majority are pretty small
• Aggregating inputs for other jobs fit in memory of cluster
• Parallelism of jobs, Failure Recovery, Job Scheduling handled
• Trade-off between accuracy and response time
• Single execution framework for batch, streaming and interactive computations

New layers added are mentioned in ()

• Application
• Data Processing (In Memory Processing)
• Storage (Data Management Layer), (Resource Management)
• Infrastructure
• One cluster for both MPI and Hadoop
• Spark (Batch & Interactive Apps Support)
• Spark and Shark are available in Amazon Elastic Map Reduce
• Tachyon - Storage abstraction

Architecture and Component - Screenshots

Download the components from link
AMP Lab Blog link

Good Session, Happy Learning!!!

Interactive Learning - Good Learning Sites

Wish you all a very Happy, Prosperous and Successful 2014

Compiled list of good interactive learning sites

• Very Good Learning Material - Data Structure Visualizations
• Interactive VIM Tutorial
• Learning R
• Interactive Python

More Reads

How to Learn VIM
R Tutorial
Data Analysis Tutorial
General Data Science

Happy Reading!!!

Weekend Reading Notes

Note #1 - SQL Server Database Engine Performance Tuning Basics - Must read for every TSQL Developer
Key Learning's

• Perf counter values analysis
• Temp Db configuration
• Enable Lock pages in memory
• Interpreting Avg. Disk Sec/Read values

Note #2 - Microsoft SQL Server Versions and Editions

Key Query  - SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

Note #3 - One more NOSQL DB that supports ACID transactions - FoundationDB

Note #4 - SQL Server 2014 In-Memory OLTP: Memory Management for Memory-Optimized Tables

Key Learning's

• Memory optimized Tables(MOT) reside in memory not in disks
• Steps to estimate memory for MOT
• Garbage collection of older version of records (similar to snapshot / read committed isolation levels)

Happy Learning!!!

Data Science Basics Learning Series - 2

Bayesian methods are very important concepts as part of learning. Some basic examples and simple explanations from Quora.


Read Quote of William Chen's answer to Big Data: What are the advantages of Bayesian methods over frequentist methods in web data? on Quora


Read Quote of Michael Hochster's answer to Probability: What is an intuitive explanation of Bayes' Rule? on Quora

Read Quote of Anonymous' answer to Mathematics: I have burned 200 disks and I want to make sure that they are all in perfect working order. What is the smallest size sample I could test in order to be relatively confident that 98% of all the disks are fine/burned correctly? on Quora

More Reads
How Not To Sort By Average Rating
What are the advantages of Bayesian methods over frequentist methods in web data?

Happy Learning!!!

DataScience Basics - Part I

Welcome to series of posts to learn Data Science basics. This video is a good starting material to get started

Link

Key Learnings

• Data Science term defined by Peter Naur
• How Role of Statistician differs from Data Scientist
• BI Tools vs Data Science perspectives

More Reads

• How to Think Critically VI: Bayes' Rule
• Knowing How to Change Your Mind: A Three Minute Guide to Bayesian Reasoning
• DataMind Beta Course
• R - Fiddle
• Tutorial: Basic data processing with R
• One R Tip A Day
• What method did Nate Silver use to predict US Presidential election in 2012?
• How can I use Nate Silver's methods to accurately predict future events?

Happy Reading!!!

Getting Started with Python Visualization

This post is about getting started with visualization using python. This took less than an hour to see visual data representation

Below are the steps involved for our first example

Step 1 - Install Enthought Python Distribution (EPD) from link. Download was around 230MB, This took a couple of minutes

Step 2 - This post was pretty useful on installation on Windows

Step 3 - Post Installation below setting was done

Step 4 - The first example is from python book page 37

Step 5 - Create a new file, type the code, Save and Run it

Step 6 - Below is the output 

More Examples
Pandas timeseries plot setting x-axis major and minor ticks and labels


Happy Learning!!!

Create Dummy Files to consume diskspace - fsutil command

This post is about creating dummy files to consume disk space and mimic scenarios with reduced disk space. On windows server 2008 fsutil command was useful to create files to consume disk space

Example Usage

C:\Users\Administrator>fsutil file createnew c:\dummydata5.bat 1240171806

File c:\dummydata5.bat is created

Happy Learning!!!