Big data Testing Tools - Functional and Performance

This post is based on my learning notes on functional test tools for Big Data ecosystem. Earlier posts, we have read the basics of big data ecosystem components. Sharing the first version of Test Tools Analysis for Functional Testing

Product / Area	Testing Tools	Test Approach	Programming Language	Reference
HiveMQ	MQTT Testing Utility, Tsung			link, link1
Storm	storm test		Clojure	link
Hive, Pig	Beetest, Pigmix, Apache DataFu	Query HIVE (Similar to TSQL)		link
Map Reduce Jobs	MRUnit, MRBench			link
Analytics		Lift charts, Target shuffling, Bootstrap sampling to test the consistency of the model		link
HBASE	Junit, Mockito, Apache MRUnit			link
	Jmeter Plugins for Hadoop, HBASE, Cassandra			link, link1

More Tools

• HiBench - Hadoop Benchmarking suite
• Gatling - Open Source Perf Test Framework
• Corbetura - Effectiveness of testing, Code Coverage library for Java
• Open Source Load Testing Tools: Which One Should You Use?
• Stress Testing JMeter and Gatling
• Gatling: Take Your Performance Tests to the next Level
• Write Your Own Load Tester in Under an Hour
• Custom LoadGenerator Tool Identifies the Issues Your Application Faces Under Stress
• An example of repeating code using a worker thread – without using timers c#
• Distributed Load generation Platform with Server Side Monitoring Capabilities.
• TestNG + Selenium – Load Testing Example
• How to Hit Your Restful Web Service Using Jmeter – Perform a Simple Load Test
• Jmeter Enhancement
• Big Data Performance Testing
• Apache JMeter Part 3: Testing Website Performance
• Conductor - A framework for testing distributed systems

Performance Testing Tools Analysis

Area	Tool	Comments
HBASE	Inbuilt tool usage – $ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1 $ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10240 randomWrite 1 More Reads - link Automation scripts for comparing different HBase BlockCache implementations - link Hbase write throughput link	PerformanceEvaluation (Inbuilt HBASE tool) - (Validate Read / Writes / Scans performance in the environment etc..)
YCSB	Performance testing HBase using YCSB , link1, link2
All Big Data Areas (HBASE, Hadoop, MapReduce	Sandstorm commercial cloud / in premise tool for Big Data QA
Kafka	Benchmarking Apache Kafka with SimpleConsumer in Java Kafka performance testing Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)
Spark	Testing Spark Best Practices

More Reads - Impetus Perf Engineering Blog
Rest-assured - Java DSL for easy testing of REST services
Retrofit - A type-safe REST client for Android and Java
G7 - Tools for Big Data

Cloud Testing Tools

Test Environment Setup using Cloud Infrastructure

• Load generation in cloud for on premises application
• Load generator on premises, application on cloud
• Both load generator and application both on cloud

Amazon Cloud Pieces

• EC2 - Elastic Compute Cloud -> CPUs
• EBS - Elastic Block Storage -> Database
• S3 - Simple Storage Services -> Storage
• Ec2 Dream Tool for connecting to multiple cloud providers - link

Blazemeter

• Distributed geographical performance test tool
• For First level of testing only
• Upload and run your custom Jmeter Scripts through blazemeter

Load Test Tools

• Flood.io
• loadfocus

Security Testing Tools

• NTOSpider
• Burp Proxy

Blazemeter walkthrough Example

Step 1 - Create Load Test

Step 2 - Configure URL

Step 3 - Start Run

Step 4 - Reports

You can also upload Jmeter scripts and execute it through blazemeter

Happy Learning!!!