What Is Performance Testing: Guide to Speed, Scalability and Reliability

Users don’t wait. If a page stalls, a checkout hangs, or a dashboard times out, people leave and systems buckle under the load. Performance testing is how teams get ahead of those moments. It measures how fast and stable your software is under realistic and extreme conditions. Done right, it gives you hard numbers on speed, scalability, and reliability, and a repeatable way to keep them healthy as you ship new features.

Problem:

Modern applications are a web of APIs, databases, caches, third-party services, and front-end code running across networks you don’t fully control. That complexity creates risk:

• Unpredictable load: Traffic comes in waves—marketing campaigns, product launches, or seasonal surges create sudden spikes.
• Hidden bottlenecks: A single slow SQL query, an undersized thread pool, or an overzealous cache eviction can throttle the entire system.
• Cloud cost surprises: “Autoscale will save us” often becomes “autoscale saved us expensively.” Without performance data, cost scales as fast as traffic.
• Regressions: A small code change can raise response times by 20% or increase error rates at high concurrency.
• Inconsistent user experience: Good performance at 50 users says nothing about performance at 5,000 concurrent sessions.

Consider this real-world style example: an ecommerce site that normally handles 200 requests per second (RPS) runs a sale. Marketing expects 1,500 RPS. The team scales web servers but forgets the database connection pool limit and leaves an aggressive retry policy in the API gateway. At peak, retries amplify load, connections saturate, queue times climb, and customers see timeouts. Converting that moment into revenue required knowledge of where the limits are, how the system scales, and what fails first—exactly what performance testing reveals.

Possible methods:

Common types of performance testing

Each test type answers a different question. You’ll likely use several.

• Load testing — Question: “Can we meet expected traffic?” Simulate normal and peak workloads to validate response times, error rates, and resource usage. Example: model 1,500 RPS with typical user think time and product mix.
• Stress testing — Question: “What breaks first and how?” Push beyond expected limits to find failure modes and graceful degradation behavior. Example: ramp RPS until p99 latency exceeds 2 seconds or error rate hits 5%.
• Spike testing — Question: “Can we absorb sudden surges?” Jump from 100 to 1,000 RPS in under a minute and observe autoscaling, caches, and connection pools.
• Soak (endurance) testing — Question: “Does performance degrade over time?” Maintain realistic load for hours or days to catch memory leaks, resource exhaustion, and time-based failures (cron jobs, log rotation, backups).
• Scalability testing — Question: “How does performance change as we add resources?” Double pods/instances and measure throughput/latency. Helps validate horizontal and vertical scaling strategies.
• Capacity testing — Question: “What is our safe maximum?” Determine the traffic level that meets service objectives with headroom. Be specific: “Up to 1,800 RPS with p95 < 350 ms and error rate < 1%.”
• Volume testing — Question: “What happens when data size grows?” Test with large datasets (millions of rows, large indexes, deep queues) because scale often changes query plans, cache hit rates, and memory pressure.
• Component and micro-benchmarking — Question: “Is a single function or service fast?” Useful for hotspot isolation (e.g., templating engine, serializer, or a specific SQL statement).

Key metrics and how to read them

Meaningful performance results focus on user-perceived speed and error-free throughput, not just averages.

• Latency — Time from request to response. Track percentiles: p50 (median), p95, p99. Averages hide pain; p99 reflects worst real user experiences.
• Throughput — Requests per second (RPS) or transactions per second (TPS). Combine with concurrency and latency to understand capacity.
• Error rate — Non-2xx/OK responses, timeouts, or application-level failures. Include upstream/downstream errors (e.g., 502/503/504).
• Apdex (Application Performance Index) — A simple score based on a target threshold (T) where satisfied ≤ T, tolerating ≤ 4T, and frustrated > 4T.
• Resource utilization — CPU, memory, disk I/O, network, database connections, thread pools. Saturation indicates bottlenecks.
• Queue times — Time waiting for a worker/thread connection. Growing queues without increased throughput are a red flag.
• Garbage collection (GC) behavior — For managed runtimes (JVM, .NET): long stop-the-world pauses increase tail latency.
• Cache behavior — Hit rate and eviction patterns. Cold cache vs warm cache significantly affects results; measure both.
• Open vs closed workload models — Closed: fixed users with think time. Open: requests arrive at a set rate regardless of in-flight work. Real traffic is closer to open, and it exposes queueing effects earlier.

Example: If p95 latency climbs from 250 ms to 900 ms while CPU remains at 45% but DB connections hit the limit, you’ve likely found a pool bottleneck or slow queries blocking connections—not a CPU bound issue.

Test data and workload modeling

Good performance tests mirror reality. The fastest way to get wrong answers is to test the wrong workload.

• User journeys — Map end-to-end flows: browsing, searching, adding to cart, and checkout. Assign realistic ratios (e.g., 60% browse, 30% search, 10% checkout).
• Think time and pacing — Human behavior includes pauses. Without think time, concurrency is overstated and results skew pessimistic. But when modeling APIs, an open model with arrival rates may be more accurate.
• Data variability — Use different products, users, and query parameters to avoid cache-only results. Include cold start behavior and cache warm-up phases.
• Seasonality and peaks — Include known peaks (e.g., Monday 9 a.m. login surge) and cross-time-zone effects.
• Third-party dependencies — Stub or virtualize external services, but also test with them enabled to capture latency and rate limits. Be careful not to violate partner SLAs during tests.
• Production-like datasets — Copy structure and scale, not necessarily raw PII. Use synthetic data at similar volume, index sizes, and cardinality.

Environments and tools

Perfect fidelity to production is rare, but you can get close.

• Environment parity — Mirror instance types, autoscaling rules, network paths, and feature flags. If you can’t match scale, match per-node limits and extrapolate.
• Isolation — Run tests in a dedicated environment to avoid cross-traffic. Otherwise, you’ll chase phantom bottlenecks or throttle real users.
• Generating load — Popular open-source tools include JMeter, Gatling, k6, Locust, and Artillery. Managed/cloud options and enterprise tools exist if you need orchestration at scale.
• Observability — Pair every test with metrics, logs, and traces. APM and distributed tracing (e.g., OpenTelemetry) help pinpoint slow spans, N+1 queries, and dependency latencies.
• Network realism — Use realistic client locations and latencies if user geography matters. Cloud-based load generators can help simulate this.

Common bottlenecks and anti-patterns

• N+1 queries — Repeated small queries per item instead of a single batched query.
• Chatty APIs — Multiple calls for a single page render; combine or cache.
• Unbounded concurrency — Unlimited goroutines/threads/futures compete for shared resources; implement backpressure.
• Small connection pools — DB or HTTP pools that cap throughput; tune cautiously and measure saturation.
• Hot locks — Contended mutexes or synchronized blocks serialize parallel work.
• GC thrashing — Excess allocations causing frequent or long garbage collection pauses.
• Missing indexes or inefficient queries — Full table scans, poor selectivity, or stale statistics at scale.
• Overly aggressive retries/timeouts — Retries can amplify incidents; add jitter and circuit breakers.
• Cache stampede — Many clients rebuilding the same item after expiration; use request coalescing or staggered TTLs.

Best solution:

The best approach is practical and repeatable. It aligns tests with business goals, automates what you can, and feeds results back into engineering and operational decisions. Use this workflow.

1) Define measurable goals and guardrails

• Translate business needs into Service Level Objectives (SLOs): “p95 API latency ≤ 300 ms and error rate < 1% at 1,500 RPS.”
• Set performance budgets per feature: “Adding recommendations can cost up to 50 ms p95 on product pages.”
• Identify must-haves vs nice-to-haves and define pass/fail criteria per test.

2) Model realistic workloads

• Pick user journeys and arrival rates that mirror production.
• Include think time, data variability, cold/warm cache phases, and third-party latency.
• Document assumptions so results are reproducible and explainable.

3) Choose tools and instrumentation

• Pick one primary load tool your team can maintain (e.g., JMeter, Gatling, k6, Locust, or Artillery).
• Ensure full observability: application metrics, infrastructure metrics, logs, and distributed traces. Enable span attributes that tie latency to query IDs, endpoints, or user segments.

4) Prepare a production-like environment

• Replicate instance sizes, autoscaling policies, connection pool settings, and feature flags. Never test only “dev-sized” nodes if production uses larger instances.
• Populate synthetic data at production scale. Warm caches when needed, then also test cold-start behavior.

5) Start with a baseline test

• Run a moderate load (e.g., 30–50% of expected peak) to validate test scripts, data, TLS handshakes, and observability.
• Record baseline p50/p95/p99 latency, throughput ceilings, and resource usage as your “known good” reference.

6) Execute load, then stress, then soak

• Load test up to expected peak. Verify you meet SLOs with healthy headroom.
• Stress test past peak. Identify the first point of failure and the failure mode (timeouts, throttling, 500s, resource saturation).
• Soak test at realistic peak for hours to uncover leaks, drift, and periodic jobs that cause spikes.
• Spike test to ensure the system recovers quickly and autoscaling policies are effective.

7) Analyze results with a bottleneck-first mindset

• Correlate latency percentiles with resource saturation and queue lengths. Tail latency matters more than averages.
• Use traces to locate slow spans (DB queries, external calls). Evaluate N+1 patterns and serialization overhead.
• Check connection/thread pool saturation, slow GC cycles, and lock contention. Increase limits only when justified by evidence.

8) Optimize, then re-test

• Quick wins: add missing indexes, adjust query plans, tune timeouts/retries, increase key connection pool sizes, and cache expensive calls.
• Structural fixes: batch operations, reduce chattiness, implement backpressure, introduce circuit breakers, and precompute hot data.
• Re-run the same tests with identical parameters to validate improvements and prevent “moving goalposts.”

9) Automate and guard your pipeline

• Include a fast performance smoke test in CI for critical endpoints with strict budgets.
• Run heavier tests on a schedule or before major releases. Gate merges when budgets are exceeded.
• Track trends across builds; watch for slow creep in p95/p99 latency.

10) Operate with feedback loops

• Monitor in production with dashboards aligned to your test metrics. Alert on SLO burn rates.
• Use canary releases and feature flags to limit blast radius while you observe real-world performance.
• Feed production incidents back into test scenarios. If a cache stampede happened once, codify it in your spike test.

Practical example: Planning for an ecommerce sale

Goal: Maintain p95 ≤ 350 ms and error rate < 1% at 1,500 RPS; scale to 2,000 RPS with graceful degradation (return cached recommendations if backend is slow).

1. Workload: 60% browsing, 30% search, 10% checkout; open model arrival rate. Include think time for browse flows and omit it for backend APIs.
2. Baseline: At 800 RPS, p95 = 240 ms, p99 = 480 ms, error rate = 0.2%. CPU 55%, DB connections 70% used, cache hit rate 90%.
3. Load to 1,500 RPS: p95 rises to 320 ms, p99 to 700 ms, errors 0.8%. DB connection pool hits 95% and queue time increases on checkout.
4. Stress to 2,200 RPS: p95 600 ms, p99 1.8 s, errors 3%. Traces show checkout queries with sequential scans. Connection pool saturation triggers retries at the gateway, amplifying load.
5. Fixes: Add index to orders (user_id, created_at), increase DB pool from 100 to 150 with queueing, add jittered retries with caps, enable cached recommendations fallback.
6. Re-test: At 1,500 RPS, p95 = 280 ms, p99 = 520 ms, errors 0.4%. At 2,000 RPS, p95 = 340 ms, p99 = 900 ms, errors 0.9% with occasional fallbacks—meets objectives.
7. Soak: 6-hour run at 1,500 RPS reveals memory creep in the search service. Heap dump points to a cache not honoring TTL. Fix and validate with another soak.

Interpreting results: a quick triage guide

• High latency, low CPU: Likely I/O bound—database, network calls, or lock contention. Check connection pools and slow queries first.
• High CPU, increasing tail latency: CPU bound or GC overhead. Optimize allocations, reduce serialization, or scale up/out.
• Flat throughput, rising queue times: A hard limit (thread pool, DB pool, rate limit). Increase capacity or add backpressure.
• High error rate during spikes: Timeouts and retries compounding. Tune retry policies, implement circuit breakers, and fast-fail when upstreams are degraded.

Optimization tactics that pay off

• Focus on p95/p99: Tail latency hurts user experience. Optimize hot paths and reduce variance.
• Batch and cache: Batch N small calls into one; cache idempotent results with coherent invalidation.
• Control concurrency: Limit in-flight work with semaphores; apply backpressure when queues grow.
• Right-size connection/thread pools: Measure saturation and queueing. Bigger isn’t always better; you can overwhelm the DB.
• Reduce payloads: Compress and trim large JSON; paginate heavy lists.
• Tune GC and memory: Reduce allocations; choose GC settings aligned to your latency targets.

Governance without red tape

• Publish SLOs for key services and pages. Keep them visible on team dashboards.
• Define performance budgets for new features and enforce them in code review and CI.
• Keep a living playbook of bottlenecks found, fixes applied, and lessons learned. Reuse scenarios across teams.

Common mistakes to avoid

• Testing the wrong workload: A neat, unrealistic script is worse than none. Base models on production logs when possible.
• Chasing averages: Median looks fine while p99 burns. Always report percentiles.
• Ignoring dependencies: If third-party latency defines your SLO, model it.
• One-and-done testing: Performance is a regression risk. Automate and re-run on every significant change.
• Assuming autoscaling solves everything: It helps capacity, not necessarily tail latency or noisy neighbors. Measure and tune.

Quick checklist

• Clear goals and SLOs defined
• Realistic workloads with proper data variance
• Baseline, load, stress, spike, and soak tests planned
• Full observability: metrics, logs, traces
• Bottlenecks identified and fixed iteratively
• Automation in CI with performance budgets
• Production monitoring aligned to test metrics

In short, performance testing isn’t a one-off gate—it’s a continuous practice that blends measurement, modeling, and engineering judgment. With clear objectives, realistic scenarios, and disciplined analysis, you’ll not only keep your app fast under pressure—you’ll understand precisely why it’s fast, how far it can scale, and what it costs to stay that way.

Some books about performance:

These are Amazon affiliate links, so I make a small percentage if you buy the book. Thanks.

• Systems Performance (Addison-Wesley Professional Computing Series) (Buy from Amazon, #ad)
• Software Performance Testing: Concepts, Design, and Analysis (Buy from Amazon, #ad)
• The Art of Application Performance Testing: From Strategy to Tools (Buy from Amazon, #ad)

Overview on Performance Testing

What is Performance Testing?

What are the features of Telerik Test Studio?

About Telerik Test Studio

- Telerik test studio is a software testing tool that has been developed for carrying out different  kinds of testing such as the functional testing for web and desktop applications, performance testing, mobile application testing and load testing etc. 

- Telerik developed it to suit all kinds of web and desktop applications eliminating the need for writing the code.

- This testing tool lets you build easy collaboration between the QA professionals and the software developers. 

- The tool has been built in such a way so as to support several web technologies such as the HTML, Silverlight, javascript, asp.net and ajax. 

Features of Telerik Test Studio

1. The test recording in test studio does not depend on the scripts.
2. It supports cross – browser testing for a number of browsers such as Mozilla Firefox, Chrome, Internet explorer and safari.
3. Application testing is supported for applications developed in HTML, Silverlight, asp.net MVC and ajax.
4. It supports the web element abstraction and focuses on reusing them.
5. The test studio can be integrated with the 2008 and 2010 editions of the visual studio.
6. The tool can be integrated with the Nunit, Xunit and MbUnit.
7. The test studio validates the user interface based up on the sentences.
8. It has got a visual storyboard that lets the team members know about the progress done so far.
9. It supports the customization of the test in vb.net and C#.
10. It can also be used to carry out exploratory testing.
11. Telerik test studio can be integrated with the HP quality center.
12. It can also be integrated with the Telerik Team pulse.

- In other words, Telerik test studio provides one simple solution for most of the testing needs. 

- A special version of the test studio called the test studio for iOS has been developed using which the developers can create tests for apple products. 

- Using this tool, the developers can carry out the testing directly on the device. 

- Here, the element detection is not based up on image.

- Instead, object based recording is used.

- It supports development platforms such as the monotouch, Xcode and phonegap.

- One can test both the native and hybrid apps.

- For creating load tests you can either capture them or use the fiddler logs.

- You can run the functional tests as the performance tests and gather data regarding the client rendering time, network latency and server processing time.

- It is quite easy to get started with and quite easy to use too. 

- To create even the most complex functional, performance and load tests what that it all takes is navigation, pointing and clicking. 

- The test studio further helps in streamlining the teamwork by providing a common platform on which the developers and testers can work together. 

- Another thing about telerik test studio, it fits any budget since in one software testing tool you get the both performance and functional testing tools. 

- The license of this tool is quite affordable and can be easily obtained. 

- Some other features of test studio are:

1. Lightswitch – powered apps automation
2. Multi – browser support
3. Manual testing
4. Telerik controls automation
5. Provides native support for the kendo UI widgets
6. Data – driven testing
7. Support for browser dialog and html popups
8. Support for frames and frame sets
9. Historical view of the tests
10. Test customization
11. Single virtual user license type
12. Reporting dashboard
13. GUI test recorder
14. Visual studio professional and team system integration
15. Simple interface
16. Built – in data grid
17. Code-less conditional logic support

Explain Telerik Test Studio? What kinds of testing are supported by Telerik Test Studio?

About Telerik Test Studio

- Telerik test studio is another in the line of software testing tools. 

- This testing tool has been developed by Telerik test studio for performing functional testing for web and desktop, load testing, performance testing, mobile application testing and so on. 

- This software testing tool can be used to test all kinds of desktop and web applications. 

- This software testing tool can be used to test all kinds of desktop and web applications without having to write the test code. 

- The Telerik tool eases the collaboration between the quality assurance people and the software developers.

- A number of web technologies such as AJAX, HTML, ASP.NET, javascript, MVC and Silverlight are supported by the Telerik test studio. 

What kind of testing are done by Telerik Test Studio?

This tool can be used to carry out the following kinds of testing:

1. Functional Testing: 

- Almost any application (including AJAX, MVC, WPF, Silverlight and HTML) can be automated using Telerik. 

- In addition to the applications, it also supports the visual studio 2010.  

- Telerik comes with a plug – in for visual studio plus a standalone application that can be used by QA professionals for creating tests for cross browser testing. 

- These tests can be customized via visual studio or they can be converted in to unit tests.

- Most QAs use the Telerik test studio for testing the AJAX and asp.NET applications along with the client – side functionality. 

- They are also used for binding the test steps on to a data source. 

- The java script functions can be directly invoked and validated from the .net code.

- Telerik is even a Silverlight UI testing automation tool that does not depends up on the code.

- The tool comes with a rich UI object model because of which the users are able to automate the tests for the entire range of the UI elements. 

- The tool after plugging in to the Silverlight application directly can automate and access all the elements present in the Silverlight application.

- Telerik test studio has made it possible to test the reliability of the java apps. 

- It has got built – in translators for the following:

             a)    HTML5 controls

             b)    Cross – browser support

             c)    Javascript event handling

            d)    Codeless test automation of the various multimedia elements.

- Telerik makes the collaboration between the QAs and software developers effortless even if there location is not same.

- Both the visual studio plug – in and application make use of the same file formats and repositories.

- This allows the QAs to keep a check of the tests in the main application while the software developers can extend it with the code. 

- It becomes easy for the team members to know what has already been recorded by the other members via the built – in visual storyboard tool.

2. Performance Testing

- Telerik allows the analyzation of the metrics on the client and the server side, exploration of the traffic requests and counter data.

- It allows you to compare the results to the already defined benchmarks for goal setting or regression detection. 

- The existing UI tests can be supplemented with the performance metrics so that the data such as the network latency, server processing time, client rendering time etc. can be included.

3. Load testing

- Telerik test studio can be used to build tests up on the existing ones and run them so as to get a better understanding of the performance of their website under extreme conditions. 

- The load agents of the telerik test studio make use of the hyper threaded and multi-core CPUs for generating a very high user load.

4. Test studio for iOS

- This is a special version developed for automated testing of the iPad, iPhones and so on. 

Give an overview of the process of system performance validation with IBM Rational Performance Tester?

IBM brought the rational performance tester in to picture as an effort to automate the rigorous process of performance testing. Automated performance testing was required to make it possible to deliver high quality software to the end users.

- The Rational Performance Tester is a convenient means for accelerating the software system’s performance without impacting its quality. 

- Software performance bottlenecks can be identified very easily with the help of IBM rational performance tester. 

- Not only the presence of bottlenecks but also their causes can be identified.  

- It is important to make predictions regarding the behavior and performance of the software systems and applications under extreme load conditions. 

- Making wrong predictions can get the whole organization, its revenue as well as the customer’s at risk. 

This can prove to be hell devastating in today’s software engineering world where the competition is already soaring high. 

- Hence, a proper testing solution is required that would not only help you validate the performance of your product but also optimize and verify it under a number of load conditions. 

- IBM rational performance tester gives you more than just testing. 

- Using it, the scalability factor of many of the applications can be calculated. 

- The rational performance tester supports code free programming i.e., the programming knowledge is not much required. 

In this article we provide an overview of the system performance validation using the rational performance tester. 

- Performance tests are contained within performance test projects.

- Instead of creating a new project, a test can be simply recorded which in turn will create a project immediately. 

- After a test has been recorded, it can be edited so as to include data pools, data correlation and verification points. 

- Data pools are required for obtaining variable data. 

- At verification points, it is confirmed whether the test is running as expected or not. 

- Lastly the data correlation ensures that appropriate data is being returned for the request. 

- Furthermore, protocol specific elements can also be added to a test.

- The items to which you make modifications appear in italics whenever a test is edited. 

- After saving the test it changes in to regular type. 

- A workload can be emulated by adding user groups and other elements to a new schedule.

- After being done with all the additions to the schedule, it can be run. 

- You can either run it locally or with a default launch configuration. 

- The playback might be slowed down because of the delays in HTTP requests. - The performance results of the application can be evaluated after being generated dynamically during the execution process. 

- After execution, the results can be regenerated for analyzation and viewing purpose. 

- The coding problems which are known to cause disturbances in the performance can be found and fixed using the problem analysis tools. 

- The data regarding the response time breakdown can be obtained using an application that has been deployed in a production environment. 

- Another means for collecting this data is from a distributed application implemented in a testing environment. 

- Resource monitoring data including available network, disk usage, network throughput, processor usage and so on can also be collected and imported. 

- Resource monitoring data is important since one can have a comprehensive view of the system or application and this is helpful in the determination of problems.

- All the collected and imported data can be analyzed and later can be used in the location of the exact cause of the problem that is hampering the system performance.

What are the metrics that can be used during performance testing?

One of the most important parts that together make up the complete and effective software testing is the performance testing. Perhaps no software system or application can do without performance testing and testing of any software system or application is incomplete without this one. 

"The determination of the performance of a software system or application in terms of how and when and responsiveness and stability under different particular work loads is nothing but performance testing".

Apart from this there are several other attributes that can be taken care of by the performance testing like for example:

1. Scalability
2. Reliability
3. Resource usage and so on.

The concept of performance testing falls under the concept of performance engineering. The practice of performance testing is aimed at building the performance in to the architecture as well as design of the software system or applications before the actual coding of the software system or application begins. 

We have so many other types of testing that together make up the complete performance testing and have been mentioned below:

1. Load testing
2. Stress testing
3. Soak testing or Endurance testing
4. Spike testing
5. Isolation testing and
6. Configuration testing

Metrics used during Performance Testing

The performance testing cannot be carried out all alone by itself! Rather it is supported by some specific metrics called performance metrics. In this article we shall discuss about the metrics that are to be used during the performance testing.  We shall discuss them one by one:

 1. Average response time: 

This is the time that takes in to consideration all the response cycle ups or round trip requests until a particular point is reached and is the mean of all the response times. Response times can be measured in any of the following way:

      a) Time to last byte or 

      b) Time to first byte

   

      2. Peak response time: 

    This is similar to the previously mentioned metric and represents the longest cycle at a particular point in a particular test. Peak response time is an indication of some potential problem in our software system or application.

   

     3. Error rate: 

   The occurrence of errors under load during the processing of the requests is perhaps the most expected thing during the testing. it has been noted that the most of the errors are reported when the load on the application reaches a peak point and further from there the software system or application is not able to process any requests. Thus error rate provides a percentage of the HTTP status code responses as errors on the servers.

    

     4. Throughput: 

   This performance metric gives you a percentage of the flow of data back and forth from the servers and is measured in units of KB per second.

   

    5. Requests per second (RPS): 

    It gives a measure of the requests that are sent to the target server including requests for CSS style sheets, HTML pages, JavaScript libraries, flash files, XML documents and so on.

    6. Concurrent users: 

    This performance metric is perhaps that best way for expressing the load that is being levied on the software system or application during the performance testing. This metric is not to be equated to the RPS since there are possibilities of one user only generating a high number of requests and on the other hand a user not constantly generating requests.

     7. Cross result graphs: 

     It shows difference between post tuning and pre-tuning tests. 

What are different characteristics of baseline testing?

Most of us don’t know what is a baseline actually? So let us first clear up what does it actually means?

Generally a baseline is defined as a line that forms the base for any construction or for measurement, comparisons or calculations. In the context of engineering and science it refers to the point of reference. Other tests like load tests, stress tests, resilience tests; baseline tests also form a very essential and important part of the performance testing.

It’s a very crucial part of a good software system or application.

Baseline testing is very famous for improving the overall performance of the software system. It has been found that the baseline testing identifies nearly 85 percent of the software system or application issues.

Baseline testing also helps a great deal in solving most of the problems that are discovered. A majority of the issues are solved through baseline testing.

The whole implementation and idea of how the baseline testing should be performed should be clear in the mind of the software tester. He should know that why the baseline testing is being done?

Some questions ?
- What is the baseline testing referring to in the software system or application?
- How the baseline testing is to be carried out or what is the test plan?
- How the baseline tests are to be executed?
- How this baseline testing differs from performance testing?

Advantages of Baseline Testing
- The main advantage here is that a large amount of time is saved by baseline testing.
- Actually it saves the time overhead.
- If we look at the whole scenario, the performance testing is the most time consuming process and complex also. Often the software testers or developers are not able to spare that much time for carrying out the performance testing very efficiently. So usually what some testers do is that they reduce the number of baseline tests. But the fact is that the baseline testing only saves much of the time.
- Reducing the number of baseline tests causes potential errors and bugs which in turn consume more time later in getting identified and corrected.
- Baseline testing is where most of the time can be saved.
- A performance testing plan must compulsorily include baseline testing and load testing. If more time is there endurance testing, volume testing and stress testing should also be included.

The testers or the developers should have a clear and good understanding of the testing that is to be performed.
- Baseline is usually performed for each script with 1, 2, 5, 10, 20, 30 and at the most 50 users to check the baselines.
- This is done typically for determining the response times. Generally baseline tests are carried out for each individual script that is a part of the load testing and any identified problem is immediately isolated.
- To get good and useful results the baseline tests should be executed at least for 20- 30 minutes.
- In baseline testing the test data is not lost when a test run fails and the data can be prepared for the next test quickly.
- Baseline testing reduces the time consumption of load testing also.
- Baseline testing should be properly monitored. Improper monitoring requires repetition of tests which is again wastage of time.
- Expecting meaningful results without proper monitoring is meaning less. Time over time baseline testing has proved itself to be a very important part of performance testing.
- Baseline testing shows the improvement of the software system or application when the problems and errors are fixed.
- Baseline testing should be done carefully without adhering to any shortcuts. By the time you begin the load testing, your software system will already be performing well.

What are different characteristics of performance testing?

Performance means a lot more than actually just testing the performance of a software system or application. It covers a wide range of concepts of software engineering and functionalities.

In performance testing, a software system is not merely tested on the basis of its functionalities, specifications and requirements but, it is also tested on the basis of the software system’s or application’s final performance characteristics which are measurable.

- Performance testing is both quantitative and qualitative kind of testing.
- In the field of software engineering, performance testing is typically done to determine the effectiveness and speed of a software system, hardware system, computer or device etc.

- Being a quantitative process, performance testing involves some lab tests like measurement of response time and MIPS (short form for “millions of instructions per second”) at which a software system performs.

- It also involves tests for testing the qualitative assets of a system like scalability, reliability and inter- operability.

- Often performance testing and stress testing are performed conjunction-ally.

- It’s a general kind of testing done to determine the behavior of a system whether hardware or software in the terms of stability and responsiveness when the system is provided with a significant workload.

- It is also carried out to measure, validate, verify and investigate the qualitative attributes of the system like resilience and resource usage.

- Performance testing is a sub category under performance engineering.
- It’s a kind of testing which aims to incorporate performance into the architecture and design of software or a hardware system.
- It’s basically done before the actual coding of the program.

Performance testing consists of many sub categories of testing. Few have been discussed in details below:

1.Stress testing:
This testing is done to determine the limits of the capacity of the software application. Basically this is done to check the robustness of the application software. Robustness is checked against heavy loads i.e., to say above the maximum limit.

2. Load testing:
This is simplest of all the testings. This testing is usually done to check the behavior of the application software or program under different amounts of load. Load can either be several users using the same application or the difficulty level or length of the task. Time is set for task completion. The response timing is recorded simultaneously. This test can also be used to test the databases and network servers.

3. Spike testing:
This testing is carried out by spiking the particular and observing the behavior of the concerned application software under each case that whether it is able to take the load or it fails.

Endurance testing:
As the name suggests the test determines if the application software can sustain a specific load for a certain time. This test also checks out for memory leaks which can lead to application damage. Care is taken for performance degradation. Throughput is checked in the beginning, at the end and at several points of time between the tests. This is done to see if the application continues to behave properly under sustained use or crashes down.

5.Isolation testing:
This test is basically done to check for the faulty part of the program or the application software.

6.Configuration testing:
This testing tests the configuration of the application software application. It also checks for the effects of changes in configuration on the software application and its performance.

Before carrying out performance testing some performance goals must be set since performance testing helps in many ways like:

- Tells us whether the application software meets the performance criteria or not.
- It can compare the performance of two application soft wares.
- It can find faulty parts of the program.

What are different characteristics of software performance testing?

Software performance is indeed an important part of software engineering and software development plan. It can be defined as the testing carried out to determine the quality and standard of the response and stability of the software system under a certain work load. Sometimes it an also be used to examine other qualitative aspects of the software system like scalability, reliability, security, stress, and resource usage.

In actual, the software performance testing is essentially a part of performance engineering. This is a very crucial testing methodology and is gaining popularity day by day. It is a testing methodology which seeks to raise the standards of the performance factors of the design of the software system. It is also concerned with the architecture of the internal structure of a software system or application.

Performance testing tries to build excellent performance into the architecture and design of the software system before the actual coding of the software application or system. Performance testing consists of many sub testing genres.

Few have been discussed below:

- Stress testing
This testing is done to determine the limits of the capacity of the software application. Basically this is done to check the robustness of the application software. Robustness is checked against heavy loads i.e. above the maximum limit.

- Load testing
This is the simplest of all the testing methods. This testing is usually done to check the behavior of the application or software or program under different amounts of load. Load can either be several users using the same application or the difficulty level or length of the task. Time is set for task completion. The response timing is recorded simultaneously. This test can also be used to test the databases and network servers.

- Spike testing
This testing is carried out by spiking the particular and observing the behavior of the concerned application software under each case that whether it is able to take the load or it fails.

- Endurance testing
As the name suggests the test determines if the application software can sustain a specific load for a certain time. This test also checks out for memory leaks which can lead to application damage. Care is taken for performance degradation. Throughput is checked in the beginning, at the end and at several points of time between the tests. This is done to see if the application continues to behave properly under sustained use or crashes down.

- Isolation testing
This test is basically done to check for the faulty part of the program or the application software.

- Configuration testing
This testing tests the configuration of the application software application. It also checks for the effects of changes in configuration on the software application and its performance.

Before carrying out performance testing some performance goals must be set since performance testing helps in many ways like:
- Tells us whether the application software meets the performance criteria or not.
- It can compare the performance of two application soft wares.
- It can find faulty parts of the program.

There are some considerations that should be kept in mind while carrying out performance testing. They have been discussed below:

- Server response time:
It is the time taken by one part of the application software to respond to the request generated by another part of the application. The best example for this is HTTP.

- Throughput
It can be defined as the highest number of users who use concurrent applications and that is expected to be handled properly by the application.
A high level plan should be developed for performing software performance testing.

What is Performance testing for software applications?

Performance testing is required in every field. Without doing some validation for performance testing, quality and success cannot be said to be achieved. Similarly in the field of computer science and engineering, performance testing in software applications is of great importance. Performance testing is done to find out the execution speed and time of the program, and to ensure its effectiveness. Software performance testing basically involves some quantitative tests that can be performed (in a computer lab for example), number of millions of instructions per second (MIPS) and measurement of response time. It also involves some tests for qualitative aspects such as scalability, interoperability and reliability.
Stress testing is carried out simultaneously with performance testing. So finally we can define software performance testing as a testing in software engineering that is done to find out the measure of some qualitative or quantitative aspect under a specific workload. Sometimes, it is also used to relate other quantitative and qualitative aspects such as usage of resources, scalability and reliability. Software performance testing is a concept of performance engineering which is very essential to build good software.
Performance testing consists of many sub testing genres. Few have been discussed below:
1. Stress testing: This testing is done to determine the limits of the capacity of the software application. Basically this is done to check the robustness of the application software. Robustness is checked against heavy loads i.e., to say above the maximum limit.
2. Load testing: This is simplest of all the testings. This testing is usually done to check the behavior of the application software or program under different amounts of load. Load can either be several users using the same application or the difficulty level or length of the task. Time is set for task completion. The response timing is recorded simultaneously. This test can also be used to test the databases and network servers.
3. Spike testing: This testing is carried out by spiking the particular and observing the behavior of the concerned application software under each case that whether it is able to take the load or whether it fails.
4. Endurance testing: As the name suggests the test determines if the application software can sustain a specific load for a certain time. This test also checks out for memory leaks which can lead to application damage. Care is taken for performance degradation. Throughput is checked in the beginning, at the end and at several points of time between the tests. This is done to see if the application continues to behave properly under sustained use or crashes down.
5. Isolation testing: This test is basically done to check for the faulty part of the program or the application software.
6. Configuration testing: This testing tests the configuration of the application software application. It also checks for the effects of changes in configuration on the software application and its performance.

Before carrying out performance testing some performance goals must be set since performance testing helps in many ways like:
1. Tells us whether the application software meets the performance criteria or not.
2. It can compare the performance of two application soft wares.
3. It can find faulty parts of the program.
There are some considerations that should be kept in mind while carrying out performance testing. They have been discussed below:
1. Server response time: This is the time taken by one part of the application software to respond to the request generated by another part of the application. The best example for this is HTTP.
2. Throughput: Can be defined as the highest number of users who use concurrent applications and that is expected to be handled properly by the application.

Good book on performance testing on Amazon (link).

How to execute Performance Tests?

Performance testing involves executing the same test case multiple times with data variations for each execution, and then collating response times and computing response time statistics to compare against the formal expectations. Often, performance is different when the data used in the test case is different, as different number of rows are processed in the database, different processing and validation come into play, and so on. By executing a test case many times with different data, a statistical measure of response time can be computed that can be directly compared against a formal stated expectation.

Network sensitivity tests are variations on load tests and performance tests that focus on the Wide Area Network (WAN) limitations and network activity. Network sensitivity tests can be used to predict the impact of a given WAN segment or traffic profile on various applications that are bandwidth dependent. Network issues often arise at low levels of concurrency over low bandwidth WAN segments. Very chatty applications can appear to be more prone to response time degradation under certain conditions than other applications that actually use more bandwidth. For example, some applications may degrade to unacceptable levels of response time when a certain pattern of network traffic uses 50% of available bandwidth, while other applications are virtually un-changed in response time even with 85% of available bandwidth consumed elsewhere.

This is particularly important test for deployment of a time critical application over a WAN. Also, some front end systems such as web servers, need to work much harder with dirty communications compared with clean communications encountered on a high speed LAN in an isolated load and performance testing environment.

Performance Tests Precedes Load Tests

The best time to execute performance tests is at the earliest opportunity after the content of a detailed load test plan have been determined. Developing performance test scripts at such an early stage provides opportunity to identify and re mediate serious performance problems and expectations before load testing commences. For example, management expectations of response time for a new web system that replaces a block mode terminal application are often articulated as 'sub second'. However, a web system, in a single screen, may perform the business logic of several legacy transactions and may take two seconds. Rather than waiting until the end of a load test cycle to inform the stakeholders that the test failed to meet their formally stated expectations, a little education up front may be in order. Performance tests provide a means for this education.

Another key benefit of performance testing early in the load testing process is the opportunity to fix serious performance problems before even commencing load testing. When performance testing of a 'customer search' screen yields response times of more than ten seconds, there may well be a missing index, or poorly constructed SQL statement. By raising such issues prior to commencing formal load testing, developers and DBAs can check that indexes have been set up properly.

Performance problems that relate to size of data transmissions also surface in performance tests when low bandwidth connections are used. For example, some data, such as images and "terms and conditions" text are not optimized for transmission over slow links.

Pre-requisites for Performance Testing

A performance test is not valid until the data in the system under test is realistic and the software and configuration is production like.

- Production Like Environment
Performance tests need to be executed on the same specification equipment as production if the results are to have integrity.Lightweight transactions that do not require significant processing can be tested but only substantial deviations from expected transaction response times should be reported. Low bandwidth performance testing of high bandwidth transactions where communications processing contributes to most of the response time can be tested.

- Production Like Configuration
The configuration of each component needs to be production like. For example: database configuration and operating system configuration. While system configuration will have less impact on performance testing than load testing, only substantial deviations from expected transaction response times should be reported.

- Production Like Version
The version of software to be tested should closely resemble the version to be used in production. Only major performance problems such as missing indexes and excessive communications should be reported with a version substantially different from the proposed production version.

- Production Like Access
If clients will access the system over a WAN, dial up modems, DSL, ISDN, etc. then testing should be conducted using each communication access method. Only tests using production like access are valid.

- Production Like Data
All relevant tables in the database need to be populated with a production like quantity with a realistic mix of data. Low bandwidth performance testing of high bandwidth transactions where communications processing contributes to most of the response time can be tested.

Targeted Infrastructure Tests and Performance Testing

TARGETED INFRASTRUCTURE TESTS

Targeted Infrastructure tests are isolated tests of each layer and or component in an end to end application configuration.
- It includes communications infrastructure, load balancers, web servers, application servers, crypto cards, citrix servers allowing for identification of any performance issues that would fundamentally limit the overall ability of a system to deliver at a given performance level.
- Each test can be quite simple.
- Targeted infrastructure testing separately generates load on each component and measures the response of each component under load.
- Different infrastructure tests require different protocols.

PERFORMANCE TESTS

These are the tests that determine end to end timing of various time critical business processes and transactions, while the system is under low load, but with a production sized database.
- This sets best possible performance expectation under a given configuration of infrastructure.
- It also highlights very early in the testing process if changes need to be made before load testing should be undertaken.
- Performance testing would highlight such a slow customer search transaction which could be re mediated prior to a full end to end load test.
- The best practice to develop performance tests is with n automated tool such as WinRunner, so that the response times from a user perspective can be measured in a repeatable manner with a high degree of precision. The same test scripts can later be re-used in a load test and the results can be compared back to the original performance tests.
- A key indicator of the quality of a performance test is repeatability. Re-executing a performance test multiple times should give the same set of results each time. If the results are not same each time, then the differences in results from one run to the next can not be attributed to changes in the application, configuration or environment.

What is the purpose of load tests?

The purpose of any load test should be clearly understood and documented. A load test usually fits into one of the following categories:
- Quantification of risks :
Determine, through formal testing, the likelihood that system performance will meet the formal stated performance expectations of stakeholders, such as response time requirements under given levels of load. This is traditional quality assurance(QA) type test. The load testing does not mitigate risk directly, but through identification and quantification of risk, presents tuning opportunities and an impetus for remediation that will mitigate risk.

- Determination of minimum configuration : Determine, through formal testing, the minimum configuration that will allow the system to meet the formal stated performance expectations, so that extraneous hardware, software and the associated cost of ownership can be minimized. This is a Business Technology Optimization (BTO) type test.

Basis for determining the business functions/processes to be included in a test

- High Frequency Transactions : The most frequently used transactions have the potential to impact the performance of all of the other transactions if they are not efficient.
- Critical Transactions : The more important transactions that facilitate the core objectives of the system should be included, as failure under load of these transactions has the greatest impact.
- Read Transactions : At least one READ ONLY transaction should be included, so that performance of such transactions can be differentiated from other more complex transactions.
- Update Transactions : At least one update transaction should be included so that performance of such transactions can be differentiated from other transactions.

What are Load Tests - End to End performance tests

Load Tests are end to end performance tests under anticipated production load. The objective such tests are to determine the response times for various time critical transactions and business processes and ensure that they are within documented expectations. Load tests also measures the capability of an application to function correctly under load, by measuring transaction pass/fail/error rates. An important variation of the load test is the network sensitivity test which incorporates WAN segments into a load test as most applications are deployed beyond a single LAN.

Load tests are major tests, requiring substantial input from the business, so that anticipated activity can be accurately simulated in a test environment. If the project has a pilot in production then logs from the pilot can be used to generate 'usage profiles' that can be used as part of the testing process, and can even be used to drive large portions of load test.

Load testing must be executed on today's production size database, and optionally with a projected database. If some database tables will be much larger in some months time, then load testing should also be performed against a projected database. It is important that such tests are repeatable, and give the same results for identical runs. They may need to be executed several times in the first year of wide scale deployment, to ensure that new releases and changes in database size do not push response times beyond prescribed service level agreements.

Validation phase - System Testing - Performance Testing - Capacity Planning - Stress Testing

Stress testing is another term that is used for performance testing. Though, load and stress testing are used synonymously for performance related efforts, their goal is different.
Unlike load testing, where testing is conducted for specified number of users, stress testing is conducted for the number of concurrent users beyond the specified limit. The objective is to identify the maximum number of users the system can handle before breaking down or degrading drastically. Since the aim is to put more stress on system. Think time of the user is ignored and the system is exposed to excess load.
The goals of stress testing are:
- It is the testing beyond the anticipated user base.
- It identifies the maximum load a system can handle.
- It checks whether the system degrades gracefully or crashes at a shot.

Stress testing also determines the behavior of the system as user base increases. Let us take an example of online shopping application to illustrate the objective of stress testing. It determines the maximum number of concurrent users an online system can service which can be beyond 1000 users. However there is a possibility that the maximum load that can be handled by the system may found to be same as anticipated limit.

The inference drawn from stress testing are:
- Whether the system is available or not?
- If yes, is the available system stable?
- If yes, is it moving towards unstable state?
- When is the system going to break down or degrade drastically?

Validation phase - System Testing - Performance Testing - Capacity Planning - Load testing

Load testing is much used industry term for the effort of performance testing. Here, load means the number of users or the traffic for the system. Load testing is defined as the testing to determine whether the system is capable of handling anticipated number of users or not.
In load testing, the virtual users are simulated to exhibit the real user behavior as much as possible. Even the user think time such as how users will take time to think before inputting data will also be emulated. It is carried out to justify whether the system is performing well for the specified limit of load.

Goals of Load Testing:
- Testing for anticipated user base.
- Validates whether the system is capable of handling load under specified limit.

The objective of load testing is to check whether the system can perform well for specified load. The system may be capable of accommodating more than say 1000 concurrent users. But, validating that is not under the scope of load testing. No attempt is made to determine how many more concurrent users the system is capable of servicing.
The inference drawn from load testing is :
- Whether the system is available?
- If yes, is the available system stable?

Validation phase - System Testing - Performance Testing - Capacity Planning and Bug Fixing

Most traditional applications are designed to respond to a single user at any time, most web applications are expected to support a wide range of concurrent users. As a result, performance testing has become a critical component in the process of deploying a web application. Performance testing has proven to be most useful in the capacity planning area.

- Capacity Planning
Capacity planning is about being prepared. Suppose if you want to know if your server configuration is sufficient to support two million visitors per day with average response time of less than five seconds etc. In capacity planning, you need to set the hardware and software requirements of your application so that you will have sufficient capacity to meet anticipated and unanticipated user load. One approach in capacity planning is to load-test your application in a testing server farm. By simulating different load levels on the farm using a web application performance testing tool such as WAS, you can collect and analyze the test results to better understand the performance characteristics of the application. Also, you may want to test the scalability of the application with different hardware configurations. You should load test your application with different numbers of clustered servers to confirm that the application scales well in cluster environment.

- Bug Fixing
There are some errors that may not occur until the applications is under high user load. Performance testing helps to detect and fix such problems before launching the application. It is therefore recommended that developers take an active role in performance testing their applications, especially at different major milestones of the development cycle.