Weekend Reading - Webinar - Performance Testing Approach for Big Data Applications.

Very good session – Webinar - Performance Testing Approach for Big Data Applications. Few interesting notes / slides from session

• Rate of Data Ingestion - How fast system consumes data?
• Data Processing - Speed how data is processed. Testing Data processing in isolation with data sets populated. Run specific perf tests (MR Jobs, Pig, Hive Scripts)
• Data Persistence – I/O bound process. (Data Writes / Updates on DB, Garbage Collection, Monitoring Metrics)
• Complete end to end time for processing (Network Connectivity, Processing, Results)

Big Data Test Challenges

• Diverse Technologies
• Unavailability of Test Tools for Big Data Technologies / Scenarios
• Limited Monitoring / Diagnostic Solutions
• Test Scripting / Environment

Perf Test Tools

• Use cloud to simulate large infrastructure
• Cloud orchestration scripts Puppet, Chef

Approach

• Depending on usage in production identify patterns for production workload
• Fault Tolerance Scenarios
• Hadoop monitoring tools to check Map reduce jobs
• Selecting Test Clients - Custom code
• Performance / Failover tests to ensure scalability (Node failures during processing)

Test Parameters and Summary

Very Nice, Practical and useful webinar. There are a lot of posts / webinars. This one is very useful and practical.

More Reads
Evaluating SolrMeter for Performance Testing
Benchmarking with HTTPerf.js and NodeUnit

Happy Learning!!!

Weekend Reads - API Testing (Web Service Testing, Inspect Http request, Rest API Testing) - Free tools

Very good presentation on compiled list of free tools for API Testing. More Details - Free API debugging and testing tools you should know about

Tools List from the presentation and SO reads

• Rest-assured - Java DSL for easy testing of REST services
• Retrofit - A type-safe REST client for Android and Java
• RequestBin - Inspect HTTP Requests
• Hurl.it — Make HTTP Requests
• ngrok captures all traffic - Raw request/response bytes
• mitmproxy: a man-in-the-middle proxy - Intercept, modify, replay and save HTTP/S traffic
• Postman is a powerful HTTP client to help test web services easily and efficiently
• Mock your HTTP responses to test your REST API
• Chrome Rest Console - REST Console is an HTTP Request Visualizer 
• SO - Online testing a REST service
• Simple REST and HTTP API Client for .NET

While reading through the list again went back to check on SOAP, Rest Basics. API testing / Web Service testing we will be looking into only aspects Rest, SOAP based web services. Summary based on StackOverflow readings, posts. References - StackOverflow reference answers link. More details Pls check reference link. (Consolidated Answer and Detailed short summary listed below)

Rest	SOAP
REST is over HTTP. REST has no WSDL interface definition	SOAP can be over any transportprotocols such HTTP, FTP, STMP, JMS etc.
REST stands for Representational State Transfer. REST approach uses the standard GET, PUT, POST, and DELETE verbs	Simple Object Access Protocol (SOAP)  SOAP builds an XML protocol on top of HTTP  / TCP/IP.
REST is good for getting a blob of data that you don't have to work with	SOAP describes functions, and types of data. If you want to get an object, SOAP is way quicker and easier to implement
Typically uses normal HTTP methods instead of a big XML format describing everything	Has several protocols and technologies relating to it: WSDL, XSDs, SOAP, WS-Addressing
REST plays well with AJAX'y web pages. If you keep your requests simple, you can make service calls directly from your JavaScript, and that comes in very handy.	SOAP is useful from a tooling perspective because the WSDL is so easily consumed by tools. So, you can get Web Service clients generated for you in your favourite language.

More Reads

HTTP vs REST vs SOAP

REST and SOAP: When Should I Use Each (or Both)?

From AWS Blog - 80% REST / 20% SOAP usage pattern

Example - Learn REST: A Tutorial
Restful Load Testing

Happy Learning!!!

APIs Good Read

I'm learning Big Data basics, checking real life architectures to understand the technology, implementation. Interesting slide on API's. I am sharing the same.


Happy Reading!!!

Tablediff - Quickly compare column data and schema between two Database Tables on Different Machines

Today Learning's 

• Tablediff gets installed with replication features
• Prepare Tablediff for multiple tables
• Run it from a batch file and log results in text file
• One Click Results Displayed :). Tested on SQL 2012

Example Batch File 1

TableDiff.bat
cd "C:\Program Files\Microsoft SQL Server\110\COM"
tablediff.exe -sourceserver SourceDBServerIP -sourcedatabase TestDB -sourcetable Table1 -destinationserver DestDBServerIP  -destinationdatabase TestDB -destinationtable Table1 -c -dt DiffsTable1
tablediff.exe -sourceserver SourceDBServerIP -sourcedatabase TestDB -sourcetable Table2 -destinationserver DestDBServerIP  -destinationdatabase TestDB -destinationtable Table2 -c -dt DiffsTable2

Explanation

• Option –c - Perform column level comparisons
• Option –dt – Output different records to particular table (drop and create if exists)
• By checking the log and the diff tables created we can find error code and description of differences
• By adding -strict option it expects the same schema (column order). I have not used it in above example
• Add -sourceuser,  -sourcepassword , -destinationuser, -destinationpassword parameters in case of sql credentials. If not specified windows credentials will be used

Batch File 2 - Capture Log Results
RunTD.bat

C:\\TableDiff.bat > C:\\TableDiffResults.txt 2>&1

One Click RunTD.bat and check logs for results. SO link

Happy Learning!!!

Two interesting bugs - Web service changes, DB column position changes

Two interesting bugs and learning's from them

Bug#1 - Due to install / upgrade paths it resulted in different ordinal position of same tables. Due to usage of select * this resulted in wrong update on columns

Automation Solution - With SQL command (link) we can figure out ordinal position differences between two same tables. Querying two environments for each tables, dumping the results in a XML. XML comparison  between them would highlight tables with different cardinal positions for two same tables.

Bug #2 - When web services are changed, the optional parameters introduced, signatures changed often fails to support backward compatibility 

Automation Solution - SOA model is open source tool for wsdl comparison. This can be used to validate wsdl differences between versions. Example code from SO

More Reads

Determine space used for each table in a SQL Server database

Happy Bugging & Learning!!!

Good QA Reads

Good QA Reads

• Test Manager in Agile - Great summary in terms of responsibilities (Collaboration, Hands on technical manager, High level perspective of overall execution, Planning, Managing escalations)
• Bug tracking for agile teams - Good Suggestions in Managing bugs with fixed timelines, Tracking without tools, Find patterns etc..
• 7 Commandments of Growth-Oriented Leadership - Good Thoughts - Take risks, compete, be transparent, patient listener

More Reads

Reporting session based testing

How to implement UI testing without shooting yourself in the foot

Aaron Hodder on using mind-mapping software as a visual test management tool

Happy Reading!!!

TSQL Learning's @ Work - Date Conversion yyyymmdd to mm/dd/yyyy, xp_cmdshell Tips

Tip#1 - Converting YYYYMMDD into MM/DD/YYYY Format

SELECT CONVERT(VARCHAR(10),CONVERT(DATETIME, '20120101', 121),101)

Tip #2 - Coverting HHMMSS into HH:MM:SS

SELECT STUFF(STUFF('211532', 3, 0, ':'), 6, 0, ':')

Tip #3 - xp_cmdshell does not work with temp tables / table variables. Use Global Temp Tables ## (Stackoverflow :) link)

Tip #4 - Enable xp_cmdshell in SQL Server - SO Link

Tip #5 - Testing Dynamic SQL Code with EXEC command - SO Link

Happy TSQL Revising!!!

Interesting Learning Notes

Note #1 - What is difference between performance and scalability problem ?

• Performance problem - Fixing performance issue for a website example - Pageload time is very high for homepage of website
• Scalability problem - Scaling website to support 10X user base than current user base

This post was very useful for above answer Performance v Scalability – For Employers

Note #2 - Read / Write Advantages / Disadvantages for Normalized / Denomarmalized Databases ?

• Normalized - Insert in multiple tables, Highly consistent
• Denormalized - Easy Insert, Consistency issue with updates (Multiple  versions of Records may exist)

I loved reading this post again and again - Data storages and read vs write controversy. Post is Very Simple, intuitive and clear.

Session - Advanced techniques for Web functional testing

• Tip #1 - Changing window size during execution set_window_size
• Tip #2 - Screenshots comparison using needle (Python based)

More Reads

Successful Scalability Principles

Bayesian statistics made simple

Software Carpentry: Lessons Learned

Performance Testing and Profiling: A Virtuous Cycle

Happy Learning!!!

Interesting TSQL Question

Check the below TSQL Example involving NULL. What is the result for below three queries

Tested on SQL 2012 Setup

Create TABLE Test1

(Id           int,

 Score int not null)

Create TABLE Test2

(Id           int,

 Score2       int not null)

 insert into Test1(Id, Score) VALUES (NULL,10),(10,100),(20,200)

 insert into Test2(Id, Score2) VALUES (NULL,11),(10,110),(20,220)

Case #1 

 Select * from  Test1 JOIN Test2

 ON Test1.Id = Test2.Id

Case #2 

 Select * from  Test1 LEFT OUTER JOIN Test2

 ON Test1.Id = Test2.Id

Case #3

 Select * from  Test1 LEFT OUTER JOIN Test2

 ON Test1.Id = Test2.Id

 ORDER BY Test1.Id DESC

Results are

Happy Learning!!!

SQL Server Database Restore Notes

While installing SQL 2012 on Windows Server 2008 R2

• Enable windows 3.5 from Windows Add / Remove Features
• While trying to restore SQL 2012 backup on SQL 2008, Error 'media family on device is incorrectly formed' for version mismatch is misleading

SQL Version to check query

SELECT

SERVERPROPERTY('ProductVersion') AS ProductVersion,

SERVERPROPERTY('ProductLevel') AS ProductLevel,

SERVERPROPERTY('Edition') AS Edition,

SERVERPROPERTY('EngineEdition') AS EngineEdition;

GO

Useful Notes

• How to extract the contents from an .ISO file without burning the .ISO to disc.
• Copying a SQL Server Database to Another Environment - (RESTORE FILELISTONLY command was very useful)
• How I completely remove a SQL Server database – Guaranteed to work – At least on my current setup - (Stop Services -> Delete Db Files -> Start Services -> Delete DBs) - 2 mins deletion

Quick Script encompassing the steps from above notes

Step 1 - Stop SQL Server
NET stop MSSQLSERVER

Step 2 - Remove DB Folders (MDF, LDF Files)
rmdir /s /q "C:\Databases\VOL\Data\"
rmdir /s /q "C:\Databases\VOL\Log\"

Step 3 - Recreate Same file path for Restore
mkdir  "C:\Databases\VOL\Data\"
mkdir  "C:\Databases\VOL\Log\"

Step 4 - Start SQL Server
NET start MSSQLSERVER

Step 5 - Run DB Restore script
Db Restore Command

Restore Script

• Identify mdf, log files from bak files
• Set DB to single user to restrict access
• Restore DB
• Set DB to multi user mode

RESTORE FILELISTONLY

FROM DISK = N'E:\TestDB_backup_20150210_000143.bak'

use master

ALTER DATABASE TestDB SET SINGLE_USER WITH ROLLBACK IMMEDIATE

RESTORE DATABASE TestDB

FROM DISK = N'E:\TestDB_backup_20150210_000143.bak'

WITH REPLACE,

FILE = 1

    , MOVE N'TestDB_DAT'

        TO N'C:\databases\vol\DATA\TestDB.MDF'

    , MOVE N'TestDB_LOG'

        TO N'C:\databases\vol\LOG\TestDB_log.LDF'

    , NOUNLOAD, STATS = 10

   

ALTER DATABASE TestDB

SET MULTI_USER

WITH ROLLBACK IMMEDIATE

GO

Tweak it as you need :)

Happy Learning!!!