Learnings in performance testing (JBOSS Tuning, Windows TCP / IP Changes)

Tuning JBOSS performance by increasing the RAM, number of max threads to be used, increasing TCP / IP Connections limit. Below are good references for the same

• Configure the max limit for concurrent TCP connections (Number of ports are increased upto 60K to accept more connections)
• Avoid TCP/IP Port Exhaustion
• Java (JVM) Memory Model and Garbage Collection Monitoring Tuning 
• JBoss Performance Tuning part 1
• Configuring the Windows registry - Increase Memory Performance 
• 10 Examples of HotSpot JVM Options in Java
• Tuning Garbage Collection Outline
• Key Lesson - Set -Xms and -Xmx close to each other or equal for a faster startup (removes constant resizing of JVM). But if you make a poor choice the JVM can't compensate for it.
• Key Lesson - You may need to increase the MaxPermSize, eg: -XX:MaxPermSize=128m

JBOSS parameters tweaked

File	Property	Comments
run.bat	"-Xms"
		The -Xms value is the space in memory that is committed to the VM at init. The JVM can grow to the size of –Xmx
	"-Xmx"
		The -Xmx value determines the size of the heap to reserve at JVM initialization
	"-XX:MaxPermSize"
..\deploy\jboss-web.deployer\server.xml -> "<Connector>" tag	"maxThreads"	The maximum number of request processing threads to be created by this Connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200.
	"acceptCount"	The maximum queue length for incoming connection requests when all possible request processing threads are in use
	"connectionTimeout"	The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented
..\conf\jboss-service.xml -> "org.jboss.util.threadpool.BasicThreadPool"	"KeepAliveTime"	How long a thread will live without any tasks in milliseconds
	"MaximumPoolSize"	The max number of threads in the pool
	"MaximumQueueSize"	The max number of tasks before the queue is full
..\conf\jboss-service.xml -> "org.jboss.remoting.transport.Connector"	"numAcceptThreads"	The number of threads that exist for accepting client connections
	"maxPoolSize"	The number of server threads for processing client
	"clientMaxPoolSize"	Client side maximum number of active socket connections. This basically equates to the maximum number of concurrent client calls that can be made from the socket client invoker

More Reads

Big List Of 20 Common Bottlenecks

Memory Leaks, Load Balancing and Deployment Settings: Testing Lessons learned from the eCommerce industry

How to Approach Application Failures in Production

Hands-On Tutorial: 5 Steps to Identify Java and .NET Memory Leaks

Key Performance Metrics For Load Tests Beyond Response Time- Part II

Key Performance Metrics For Load Tests Beyond Response Time- Part I

Performance Considerations in Distributed Applications

Happy Learning!!!

Weekend Reading Notes

Note #1 - SQL Server Database Engine Performance Tuning Basics - Must read for every TSQL Developer
Key Learning's

• Perf counter values analysis
• Temp Db configuration
• Enable Lock pages in memory
• Interpreting Avg. Disk Sec/Read values

Note #2 - Microsoft SQL Server Versions and Editions

Key Query  - SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

Note #3 - One more NOSQL DB that supports ACID transactions - FoundationDB

Note #4 - SQL Server 2014 In-Memory OLTP: Memory Management for Memory-Optimized Tables

Key Learning's

• Memory optimized Tables(MOT) reside in memory not in disks
• Steps to estimate memory for MOT
• Garbage collection of older version of records (similar to snapshot / read committed isolation levels)

Happy Learning!!!

Cloud Based Performance Testing Tool

This post is based on my Stackoverflow QA Answer for Question - What are the tiers of testing that should be done on a large scale distributed system?

On top of it I did some more readings and found some more interesting reads bookmarked below

• The great debate: Windows Azure vs. Amazon Web Services
• Apple icloud runs on both Windows Azure and AWS
• The Evolution of IT Towards Cloud Computing

Coming to utilizing cloud for QA purpose. Soasta is a leader in cloud based performance / functional testing. Developing Test Expertise in cloud based apps is different from traditional test approach. Particularly, How much load cloud can scale up is the key.

One of biggest advantage on their approach mentioned in blogpost. Using Cloudtest-grid to scale up/scale down load for Performance Testing, Simulating live traffic. CloudTest Lite is a cloud based free tool for performance testing Web apps/Cloud based apps.

Many thanks to Patrica for her post on CloudTest Lite. This post was the beginning of my learning on CloudTest Lite. I am looking forward to explore this tool in coming weeks. Tool looks promising for performance testing web applications.

Happy Learning!!

SQL Server Query Tuning

SQL Server Query Tuning two good posts

Query Tuning Fundamentals: Density, Predicates, Selectivity, and Cardinality
Summary from post

• Density - Measure of how often duplicates would occur
• Predicates - Join columns / filter clauses
• Cardinality - Number of rows returned by query operator

Wide vs. Narrow Plans
Turning a Scan into a Seek With A Pointless Predicate

Thanks to authors of the post. This post is a good reference for me.

More Reads
SQL Server 2008 - Cache Dynamic SQL
Performance Tuning Redefined with SQL Server 2008
Who dropped objects from database?
SQL Performance Trouble Shooting – the basics from Microsoft
Usefull DMV Query
24HOP: SQL Server Performance Tools
How can you tell if an index is being used?
Troubleshooting Performance Problems in SQL Server 2008
Performance Monitor, SQL Counters and Tresholds
What are plan guides and how it could help in managing performance of the queries
How to Find the Amount of Fragmentation on Your SQL Server Instance
SQL Bits Session - High Scale OLTP – LL From SQLCAT Performance Labs
SQL Bits - The Dark Arts-a.k.a Performance Tuning, indexing and query plans
SQL Server Index Tuning for Mere Mortals
The Difficulty with Deadlocks
Completely Legal SQL Performance Enhancements Video
Automating SQL Execution Plan analysis

Tool Link

Adam Mechanic Presentation in Teched - US on Performance Tuning

Very Good Reads!!

Web Service - Performance Testing Fundamentals

[You may also like - Using open source tools for performance testing]
[Next Post in Series - Web Services and Web Testing using VSTS]
[You may also like - 7 Most Stunning Website Crashes/ Failures of 2011 predominantly due to high traffic]
Performance testing is all about ensuring scalability and availability of the system to respond to business needs. Performance testing will ensure application can handle the desired load. Technically we must be answer below questions.

• Identify response time for requests during Average/Peak Load
• Numbers of Requests handled in XX Hours
• Avg. Response Time for each request. %% Requests meeting SLA, Missing SLA during different Loads
• Monitor System Behavior using counters - Database -> Transactions/Sec, Deadlocks/Sec

“How to performance test website / webservice” - Guidelines Listed Below
Functional Knowledge
Before you performance test your applications know all work flows/ business logic implemented in your application. Identify Data Access Layer/Business Logic layer and understand how it works. When you know the business logic flow in the application identifying the core test scenarios and test mix would be easy deal. Here the key is you need to know the application to evaluate numbers provided by business eg: how many orders per hour / site visits per peak hour.

Ex:
On an typical business Day
1. 20% is Sales through the Site
2. 30% is Orders Update through the Site
3. 20% of time No Access
4. 30% Enquiries (Search/Enquiries)

During festive Season

1. 60% is Sales through the Site
2. 20% is Orders Update through the Site
3. 5% of time No Access
4. 15% Enquiries (Search/Enquiries)

With this distribution in place you can identify the workflows for each of the transaction (Order Placement/Enquiry/Search). Outcome of this phase is you know the functionality implementation of this application to identify critical workflows in the system.

Identify Performance Test Scenarios (Performance Goals)
Here is where business requirements are converted into performance test scenarios
Ex: 19000 orders a day is expected volume, Peak volume per hour 5000 -> This translates to 1.3 requests/sec for peak volume, Average .21 Req/sec for average load
All business scenarios will be converted into test scenarios and expected result (test pass criteria) will also be mentioned in the test scenario document.

Test Scenario 1 – Average Load
1. 20% is Sales through the Site
2. 30% is Orders Update through the Site
3. 30% Enquiries (Search/Enquiries)
Test Scenario 2 – Peak Load
During festive Season
1. 60% is Sales through the Site
2. 20% is Orders Update through the Site
3. 20% Enquiries (Search/Enquiries)

The outcome of this phase is
• Test scenarios are identified
• Test Pass Criteria identified
• Test Scenarios are reviewed and signed off by stakeholders of the application

Tools/Scripts, Environmental Setup
In a Real-world scenario DEV/Test environment would not have same Hardware configuration and setup compared to Production environment. Production environment would have NLB Servers, High end processors. It is very important to test in production-like environment. Performance test environment would mimic production environment in terms of hardware configuration/setup.

Test data is another crucial factor for performance testing. It’s recommended 90% of tests must pass in a test run to consider the results for analysis. Test Data must have the same volume of production data available in production. Using production data copy is ideal. But it may not be possible because of security or privacy concern. Sample data can be created by repeating a pattern of data. 

• Identify tools to running the tests; Code test scripts/generate test data.
• Performance counters need to be identified and setup (ex: Sql Performance counters, ASP.NET Counters, Biztalk counters specific to the App)

Outcome of this phase is environment is ready, test scripts are coded for identified performance test scenarios.

Test Runs – Test Execution
In a performance test run each request-response time is recorded. Time is the crucial factor in performance testing. Test runs are normally done for 30 mins to 2 hour duration window for short test. During these tests, the results are documented to give performance insights. Also, errors and problems are reported and fixed. It’s recommended to conclude to do 3 test runs for each identified scenarios with same test data. Consistency of the behavior of the system, test results obtained across test runs need to be captured.
In the end, a test run for an extended period of time such as 12 to 24 hours should be done to check for excessive resource usage or resource leakage.

Results Collection and Analysis
Logs are collected; performance counter values are collected and analyzed to valuate test pass criteria

Result would look like this
Functionality

• Test Mix
• Req/Sec Results
• System Health Behavior / Observations
• SLA Met, SLA Slipped during different Loads

Functional Testing -> Performance Testing -> Test pass criteria passes-> Signoff for Production Release else reiterate the cycle.

A few real-time performance test scenarios listed in link

• Test Limits of a Download Server for Large Downloads: Probing the maximum number of users without deterioration of performance, identify CPU-IO-Network bottleneck issues,
• Webserver Burn-In Test: Testing a server with constant load for 8 hours and deterimine the stability of the system, response times to user requests, identify CPU-IO-DB-Network bottleneck issues,
• Probing a Webserver with 3000 Users: Running a test with up to 3000 users against an IIS6 server and monitoring the performance, response time for requests, identify CPU-IO-DB-Network bottleneck issues

Bookmarks

What Does Performance Testing Mean?
Performance Testing Guidance for Web Applications
http://channel9.msdn.com/wiki/default.aspx/PerformanceWiki.PerformanceTestingGuidance

http://blogs.msdn.com/nikhiln/archive/2007/02/05/howto-performance-test-asp-net-web-services-using-vsts.aspx
http://www.codeplex.com/PerfTesting

http://blogs.msdn.com/vstsloadtestblog/articles/592814.aspx

Very Good Read from ACE Team blogs on Performance tuning

Hardening IIS
Hardening IIS whitepaper
ACE Team Performance Checklist
How to simulate IE Caching in VSTS 2008
Threat Modeling – Sanity Check List
Application Security Guidance - Session Management
Improve .Net Applications Performance Effectively And Efficiently
Web Load Testing with Microsoft Visual Studio 2010
Top Tips for Managing .NET Application Performance
Performance Testing Tools and Load Testing Tools for Today’s Web and Mobile Applications
How To Do A Good Performance Investigation


Happy Reading!!

SQL Perf Tuning - Performance Counters and Values

I have come across lot of blogs with queries to find Top IO, CPU queries.

• Learn to differentiate IO Issue, CPU Issue, Network Issue
• IO Issue - Excessive IO can be due to a SCAN on a big table. There are ways to convert SCAN to SEEK. In a OLTP system query are expected to be highly selective, fetch few records
• CPU Issue - During processing we might have temp tables, Sort a Temp tables. Sort Operator is usually CPU intensive. Wait Stats sys.dm_os_wait_stats DMV provides more info. Please check Joe Sack wait stats link mentioned below
• Disk Issue - PhysicalDisk/LogicalDisk counter in below bookmarked excel and values defined for it for interpreting Disk Issues
• Network Latency - Network Interface counter in below excel provides values for counters and interpreting them

List of Counters and Optimal values for them

From Perfmon counters interpret from values of perfmon counters. Attached XL is very very good.
Without writing any query how do we find expensive queries

• Activity monitor check recently expensive queries
• Goto Particular Database that you want to check. List Reports for Reports->Standard Reports->Top Transactions by Age, Locks
• Check Parallelism Enabled/Not Link

Saleem has written a good article on step-by-step Analysis for query troubleshooting. Link is

• Best Practices for Troubleshooting Slow Running Queries
• Q&A: Performance Tuning Redefined with SQL Server 2008
• Are you using SQL's Missing Index DMVs?
• Top SQL Server 2005 Performance Issues for OLTP Applications
• SQL Server: 4-Step Performance Troubleshooting Methodology--Introduction
• Wait Stats by Joe Sack
• Who is Active? v9.57: Fast, Comprehensive DMV Collection - What's Really Happening on Your Server?
• Drum Roll, Please...The Debut of The SQL DMV All-Stars Dream Team!
• Top 8 Queries to Improve DB Design
• SQL Performance Optimization
• The SQL Server Wait Type Repository… - Very Good
• SQL Server I/O Bottleneck, I don't have one, YES YOU DO!
• SQL Server Performance Tools - Boise Code Camp Presentation
• Performance Counters Analysis : SQL Server , IIS , MOSS , BizTalk , Exchange , and Active Directory

Happy Learning!!!

Perf Testing - Revisited

Couple of useful suggestions

Powershell - Your friend for Performance Testing projects

Couple of useful Tools posted in Codeplex. I need to check them

SQL Load Test

Fundamentals of Storage Systems – Capturing IO Patterns

VSTT 2008 Quick Reference Guide

WCF Load Test

How To Add File Upload To Team System Web Test

VS2010 – Load Test Network Emulation Profile

Load Agent and Load Controller Installation Guide

Load Agent and Load Controller Configuration Guide

VSTS 2010 Features explained by shivaji...Read it here

CodedUITest

Working with Tempdb

Compiled list of posts for tempdb related learning resources

• T-SQL:Generate statements to add TEMPDB datafiles
• Operations that heavily stress out tempdb
• Compilation of SQL Server TempDB IO Best Practices
• Taming the Tempdb Tempest - WI SQL Server Virtual User Group, 22 Apr 2011
• TempDB Configuration
• TempDB Monitoring and Troubleshooting: Out of Space
• TempDB Monitoring and Troubleshooting: DDL Bottleneck
• TempDB Monitoring and Troubleshooting: IO Bottleneck
• TempDB matters for a healthy SQL Server (Best Practices)
• Troubleshoot SQL Server 2008 Features
• Working with tempdb in SQL Server 2005
• Storage Top 10 Best Practices
• Managing TempDB in SQL Server:TempDB Basics
• Tempdb Perf mon counters

Happy Learning!!!

Useful DMV Queries and Perfmon Guidance

Useful DMV queries

Getting the Most Out of SQL Server Profiler Using Templates

Monitoring for Blocked Processes On SQL 2005 - Extended Version

The Debut of The SQL DMV All-Stars Dream Team!

How to Tune a Database Using the Database Tuning Advisor (DTA)

SQL Server 2005 Performance Statistics Script

Understanding Perfmon Counters while troubleshooting SQL Server Performance Issues

Monitoring SQL Server health

Detecting SQL Server 2005 Blocking

sys.dm_exec_requests

Troubleshooting Performance Problems in SQL Server 2005

A great write-up on performance tuning

I/O Bottlenecks

Not more excuses of missing indexes with Activity Monitor in SQL Server 2008

SQL Server - Performance Counter Guidance

Trending and monitoring for performance

DBA Headquarters: SQL Server DBA Automation Tools

Top Tips for Effective Database Maintenance

Updated SQL Server 2008 Diagnostic Information Queries

SQL Server 2005 Replication Learnings

Transaction replication notes from by Jose Barreto

#1.
Configuring Replication for Partitioned Tables Using T-SQL Here
MSDN link for Scripting Replication here
One more Replication Setup Script here

Which objects in the database are published?
<<Publish DB>>
SELECT * FROM sysarticles
SELECT * FROM syspublications

<<Distribution DB>>
Use Distribution
GO
SELECT * FROM distribution..mspublications

--Cleanup uncleaned subscriptions
delete from mspublications where publication_id=7
Note: You might have to delete from MSsubscriber_schedule as well. For me deleteting from mspublications fixed the issue.

<<SubscriberDB>>
USE <<SubscriptionDB>>
SELECT * FROM MSsubscriptions
SELECT * FROM MSsubscriber_info

#2.
While setting up transaction replication on a table that has millions of records. Initial snapshot would take time for the records to be delivered to subscriber. In SQL 2005 we have an option to create the tables on both transaction and publish server, populate dataset and setup replication on top of it. When you add subscription with command EXEC sp_addsubscription set The @sync_type = 'replication support only'. Example here

#3.
Replication - Difference between push and pull subscription
Distribution agent for push subscriptions typically runs on the Distributor. For pull subscriptions, the distribution agent typically runs on the Subscriber.

In a Pull Subscription
1. Distribution agent resides on subscriber instead of distributor because subscriber initiates replication request
2. Distributor sends data to distributor agent on subscriber
3. Replication by pull subscription has control at subscriber side instead of publisher

In a Push Subscription

1. Replication in push subscription has control in publisher, subscribers do not need to initiate replication request
2. Changes in publisher could be sent to distributor by demand, by schedule or continuously.

I found this link useful for highlighting above differences Thanks to the author.

Best Practice as suggested by msdn link here says pull subscriptions perform much better than push subscriptions in a WAN scenario.

Transaction Replication Deep Dive

Got the below error today "The row was not found at the Subscriber when applying the replicated command. (Source: MSSQLServer, Error number: 20598)".

Found workaround for this to be to skip this error at Distributor Agent. You can add the parameter to the distribution agent -SkipErrors 20598. Steps mentioned in Microsoft KB here

Replication Performance Tuning Guidelines

1. Performance counters to check for replication - SQLServer:Replication Dist.-Dist:Delivery Latency
2. Inserting Trace Tokens and check time taken to replicate data between publisher-distributor and subscriber

SQL 2008 R2 MSDN Replication Documentation

Merge Replication Step by Step
Troubleshooting SQL Server Transactional Replication
Log Reader Agent Fails with the Error “The Log Reader Agent failed to construct a replicated command from log sequence number (LSN)”
Customized Alerts for Transactional Replication
How to resolve when Distribution Database is growing huge (+25gig)
Using sp_repldone to mark all pending transactions as having been Replicated
Checking Replication Latency with T-SQL
Determine Transactional Replication workload to help resolve data latency
Troubleshooting LogReader Timeout executing sp_replcmds
How to cleanup Replication Bits
Distribution latency in transactional replication: Is a volume surge the culprit?
Troubleshooting Transactional Replication - PART 1
Troubleshooting Transactional Replication - PART 2
Troubleshooting Transactional Replication - PART 3
How to add an article to an existing Transactional Subscription initialized through backup
How to replicate some fields of two different tables?
Adding a column to Destination table
SQL 2005 Transaction Replication – Adding new article to an existing publication
Replication Agent has not logged a message in 10 minutes
SQL Server Replication Explorer
Divide and Conquer Transactional Replication using Tracer Tokens
All About Automatically Monitoring Replication Agent Failures
Replication features in various editions of SQL Server 2005/2008
Undocumented Gotchas of Transactional Replication

Happy Reading!!

Web Services and Web Testing using VSTS

[Previous Post in Series - Performance Testing Explained]

Web Testing using VSTS

Steps for VSTS performance testing is mentioned here

• http://blogs.msdn.com/nikhiln/pages/performance-testing-asp-net-web-services-using-vsts.aspx
• http://blogs.msdn.com/slumley/pages/custom-data-binding-in-web-tests.aspx
• http://blogs.msdn.com/edglas/archive/2008/02/27/are-web-tests-functional-tests.aspx
• http://blogs.msdn.com/vstsloadtestblog/archive/2006/09/22/765608.aspx
• http://blogs.msdn.com/edglas/pages/content-index-for-web-tests-and-load-tests.aspx
• How To: Web Service Load Testing Using VSTS 2010

Data binding for Load Test

A Few guidelines for performance testing

Performance test plan should include test scenarios (At different loads, expected results), Signoff criteria defined, Performance counters to be analyzed, Details of Performance Test Bed (Hardware configuration). The tasks for performance testing would include - Environment Setup, Test Data Preparation, Test Execution Time, Bug Fix Time line, Targeted Signoff Date.

1. Identify the Performance Counters for IIS, DB

2. Benchmark the environment, Hardware configuration before starting test run
3. Conduct couple of Trial runs to verify Test Environment has no Setup, Hardware issues
4. For a typical web serices or Web App performance testing 80% of requests should pass through to consider test run successful
5. While Validating/Identifying Issues validate bottleneck is at SQL or IIS level. Server Side Trace could be enabled to capture SP:StmtCompleted Events
6. Provide a clear report on perf test results.

Example:

Test Scenario:
Test Data Mix:
Test Duration:
Key Findings:
Perf Counters:
Detailed Results:
Conclusion:

• Performance testing videos at codeplex
• Performance testing for web apps - http://msdn.microsoft.com/en-us/library/bb924375.aspx
• VSTS Web Test Step-by-Step Primer: 7-Minute Video
• How to: Create a web test plug-in to extract and store an array of values using Visual Studio Team system
• SQL - SQLIOSim
• VSTT 2008 Quick Reference Guide
• VSTS 2010 Feature: Web Test Recorder Plugins
• How to: Create a web test plug-in to extract and store an array of values using Visual Studio Team system
• Visual Studio UI Test Extensibility–Scenarios & Guiding Principles - VSTT 2010

Happy Reading!!

Performance Testing Challenges

http://www.perftestplus.com/resources/perf_challenges_ppt.pdf

Thanks,
Sivaram

Using open source tools for performance testing

A good must watch video on Performance Testing



HTTP performance testing with httperf, autobench and openload

More Reads
Jmeter - SQL query performance test with JMeter
Performance testing with JMeter
Performance Tests with Jmeter, Maven and Hudson

Happy Reading!!