What Is Performance Testing: Guide to Speed, Scalability and Reliability

Users don’t wait. If a page stalls, a checkout hangs, or a dashboard times out, people leave and systems buckle under the load. Performance testing is how teams get ahead of those moments. It measures how fast and stable your software is under realistic and extreme conditions. Done right, it gives you hard numbers on speed, scalability, and reliability, and a repeatable way to keep them healthy as you ship new features.

Problem:

Modern applications are a web of APIs, databases, caches, third-party services, and front-end code running across networks you don’t fully control. That complexity creates risk:

• Unpredictable load: Traffic comes in waves—marketing campaigns, product launches, or seasonal surges create sudden spikes.
• Hidden bottlenecks: A single slow SQL query, an undersized thread pool, or an overzealous cache eviction can throttle the entire system.
• Cloud cost surprises: “Autoscale will save us” often becomes “autoscale saved us expensively.” Without performance data, cost scales as fast as traffic.
• Regressions: A small code change can raise response times by 20% or increase error rates at high concurrency.
• Inconsistent user experience: Good performance at 50 users says nothing about performance at 5,000 concurrent sessions.

Consider this real-world style example: an ecommerce site that normally handles 200 requests per second (RPS) runs a sale. Marketing expects 1,500 RPS. The team scales web servers but forgets the database connection pool limit and leaves an aggressive retry policy in the API gateway. At peak, retries amplify load, connections saturate, queue times climb, and customers see timeouts. Converting that moment into revenue required knowledge of where the limits are, how the system scales, and what fails first—exactly what performance testing reveals.

Possible methods:

Common types of performance testing

Each test type answers a different question. You’ll likely use several.

• Load testing — Question: “Can we meet expected traffic?” Simulate normal and peak workloads to validate response times, error rates, and resource usage. Example: model 1,500 RPS with typical user think time and product mix.
• Stress testing — Question: “What breaks first and how?” Push beyond expected limits to find failure modes and graceful degradation behavior. Example: ramp RPS until p99 latency exceeds 2 seconds or error rate hits 5%.
• Spike testing — Question: “Can we absorb sudden surges?” Jump from 100 to 1,000 RPS in under a minute and observe autoscaling, caches, and connection pools.
• Soak (endurance) testing — Question: “Does performance degrade over time?” Maintain realistic load for hours or days to catch memory leaks, resource exhaustion, and time-based failures (cron jobs, log rotation, backups).
• Scalability testing — Question: “How does performance change as we add resources?” Double pods/instances and measure throughput/latency. Helps validate horizontal and vertical scaling strategies.
• Capacity testing — Question: “What is our safe maximum?” Determine the traffic level that meets service objectives with headroom. Be specific: “Up to 1,800 RPS with p95 < 350 ms and error rate < 1%.”
• Volume testing — Question: “What happens when data size grows?” Test with large datasets (millions of rows, large indexes, deep queues) because scale often changes query plans, cache hit rates, and memory pressure.
• Component and micro-benchmarking — Question: “Is a single function or service fast?” Useful for hotspot isolation (e.g., templating engine, serializer, or a specific SQL statement).

Key metrics and how to read them

Meaningful performance results focus on user-perceived speed and error-free throughput, not just averages.

• Latency — Time from request to response. Track percentiles: p50 (median), p95, p99. Averages hide pain; p99 reflects worst real user experiences.
• Throughput — Requests per second (RPS) or transactions per second (TPS). Combine with concurrency and latency to understand capacity.
• Error rate — Non-2xx/OK responses, timeouts, or application-level failures. Include upstream/downstream errors (e.g., 502/503/504).
• Apdex (Application Performance Index) — A simple score based on a target threshold (T) where satisfied ≤ T, tolerating ≤ 4T, and frustrated > 4T.
• Resource utilization — CPU, memory, disk I/O, network, database connections, thread pools. Saturation indicates bottlenecks.
• Queue times — Time waiting for a worker/thread connection. Growing queues without increased throughput are a red flag.
• Garbage collection (GC) behavior — For managed runtimes (JVM, .NET): long stop-the-world pauses increase tail latency.
• Cache behavior — Hit rate and eviction patterns. Cold cache vs warm cache significantly affects results; measure both.
• Open vs closed workload models — Closed: fixed users with think time. Open: requests arrive at a set rate regardless of in-flight work. Real traffic is closer to open, and it exposes queueing effects earlier.

Example: If p95 latency climbs from 250 ms to 900 ms while CPU remains at 45% but DB connections hit the limit, you’ve likely found a pool bottleneck or slow queries blocking connections—not a CPU bound issue.

Test data and workload modeling

Good performance tests mirror reality. The fastest way to get wrong answers is to test the wrong workload.

• User journeys — Map end-to-end flows: browsing, searching, adding to cart, and checkout. Assign realistic ratios (e.g., 60% browse, 30% search, 10% checkout).
• Think time and pacing — Human behavior includes pauses. Without think time, concurrency is overstated and results skew pessimistic. But when modeling APIs, an open model with arrival rates may be more accurate.
• Data variability — Use different products, users, and query parameters to avoid cache-only results. Include cold start behavior and cache warm-up phases.
• Seasonality and peaks — Include known peaks (e.g., Monday 9 a.m. login surge) and cross-time-zone effects.
• Third-party dependencies — Stub or virtualize external services, but also test with them enabled to capture latency and rate limits. Be careful not to violate partner SLAs during tests.
• Production-like datasets — Copy structure and scale, not necessarily raw PII. Use synthetic data at similar volume, index sizes, and cardinality.

Environments and tools

Perfect fidelity to production is rare, but you can get close.

• Environment parity — Mirror instance types, autoscaling rules, network paths, and feature flags. If you can’t match scale, match per-node limits and extrapolate.
• Isolation — Run tests in a dedicated environment to avoid cross-traffic. Otherwise, you’ll chase phantom bottlenecks or throttle real users.
• Generating load — Popular open-source tools include JMeter, Gatling, k6, Locust, and Artillery. Managed/cloud options and enterprise tools exist if you need orchestration at scale.
• Observability — Pair every test with metrics, logs, and traces. APM and distributed tracing (e.g., OpenTelemetry) help pinpoint slow spans, N+1 queries, and dependency latencies.
• Network realism — Use realistic client locations and latencies if user geography matters. Cloud-based load generators can help simulate this.

Common bottlenecks and anti-patterns

• N+1 queries — Repeated small queries per item instead of a single batched query.
• Chatty APIs — Multiple calls for a single page render; combine or cache.
• Unbounded concurrency — Unlimited goroutines/threads/futures compete for shared resources; implement backpressure.
• Small connection pools — DB or HTTP pools that cap throughput; tune cautiously and measure saturation.
• Hot locks — Contended mutexes or synchronized blocks serialize parallel work.
• GC thrashing — Excess allocations causing frequent or long garbage collection pauses.
• Missing indexes or inefficient queries — Full table scans, poor selectivity, or stale statistics at scale.
• Overly aggressive retries/timeouts — Retries can amplify incidents; add jitter and circuit breakers.
• Cache stampede — Many clients rebuilding the same item after expiration; use request coalescing or staggered TTLs.

Best solution:

The best approach is practical and repeatable. It aligns tests with business goals, automates what you can, and feeds results back into engineering and operational decisions. Use this workflow.

1) Define measurable goals and guardrails

• Translate business needs into Service Level Objectives (SLOs): “p95 API latency ≤ 300 ms and error rate < 1% at 1,500 RPS.”
• Set performance budgets per feature: “Adding recommendations can cost up to 50 ms p95 on product pages.”
• Identify must-haves vs nice-to-haves and define pass/fail criteria per test.

2) Model realistic workloads

• Pick user journeys and arrival rates that mirror production.
• Include think time, data variability, cold/warm cache phases, and third-party latency.
• Document assumptions so results are reproducible and explainable.

3) Choose tools and instrumentation

• Pick one primary load tool your team can maintain (e.g., JMeter, Gatling, k6, Locust, or Artillery).
• Ensure full observability: application metrics, infrastructure metrics, logs, and distributed traces. Enable span attributes that tie latency to query IDs, endpoints, or user segments.

4) Prepare a production-like environment

• Replicate instance sizes, autoscaling policies, connection pool settings, and feature flags. Never test only “dev-sized” nodes if production uses larger instances.
• Populate synthetic data at production scale. Warm caches when needed, then also test cold-start behavior.

5) Start with a baseline test

• Run a moderate load (e.g., 30–50% of expected peak) to validate test scripts, data, TLS handshakes, and observability.
• Record baseline p50/p95/p99 latency, throughput ceilings, and resource usage as your “known good” reference.

6) Execute load, then stress, then soak

• Load test up to expected peak. Verify you meet SLOs with healthy headroom.
• Stress test past peak. Identify the first point of failure and the failure mode (timeouts, throttling, 500s, resource saturation).
• Soak test at realistic peak for hours to uncover leaks, drift, and periodic jobs that cause spikes.
• Spike test to ensure the system recovers quickly and autoscaling policies are effective.

7) Analyze results with a bottleneck-first mindset

• Correlate latency percentiles with resource saturation and queue lengths. Tail latency matters more than averages.
• Use traces to locate slow spans (DB queries, external calls). Evaluate N+1 patterns and serialization overhead.
• Check connection/thread pool saturation, slow GC cycles, and lock contention. Increase limits only when justified by evidence.

8) Optimize, then re-test

• Quick wins: add missing indexes, adjust query plans, tune timeouts/retries, increase key connection pool sizes, and cache expensive calls.
• Structural fixes: batch operations, reduce chattiness, implement backpressure, introduce circuit breakers, and precompute hot data.
• Re-run the same tests with identical parameters to validate improvements and prevent “moving goalposts.”

9) Automate and guard your pipeline

• Include a fast performance smoke test in CI for critical endpoints with strict budgets.
• Run heavier tests on a schedule or before major releases. Gate merges when budgets are exceeded.
• Track trends across builds; watch for slow creep in p95/p99 latency.

10) Operate with feedback loops

• Monitor in production with dashboards aligned to your test metrics. Alert on SLO burn rates.
• Use canary releases and feature flags to limit blast radius while you observe real-world performance.
• Feed production incidents back into test scenarios. If a cache stampede happened once, codify it in your spike test.

Practical example: Planning for an ecommerce sale

Goal: Maintain p95 ≤ 350 ms and error rate < 1% at 1,500 RPS; scale to 2,000 RPS with graceful degradation (return cached recommendations if backend is slow).

1. Workload: 60% browsing, 30% search, 10% checkout; open model arrival rate. Include think time for browse flows and omit it for backend APIs.
2. Baseline: At 800 RPS, p95 = 240 ms, p99 = 480 ms, error rate = 0.2%. CPU 55%, DB connections 70% used, cache hit rate 90%.
3. Load to 1,500 RPS: p95 rises to 320 ms, p99 to 700 ms, errors 0.8%. DB connection pool hits 95% and queue time increases on checkout.
4. Stress to 2,200 RPS: p95 600 ms, p99 1.8 s, errors 3%. Traces show checkout queries with sequential scans. Connection pool saturation triggers retries at the gateway, amplifying load.
5. Fixes: Add index to orders (user_id, created_at), increase DB pool from 100 to 150 with queueing, add jittered retries with caps, enable cached recommendations fallback.
6. Re-test: At 1,500 RPS, p95 = 280 ms, p99 = 520 ms, errors 0.4%. At 2,000 RPS, p95 = 340 ms, p99 = 900 ms, errors 0.9% with occasional fallbacks—meets objectives.
7. Soak: 6-hour run at 1,500 RPS reveals memory creep in the search service. Heap dump points to a cache not honoring TTL. Fix and validate with another soak.

Interpreting results: a quick triage guide

• High latency, low CPU: Likely I/O bound—database, network calls, or lock contention. Check connection pools and slow queries first.
• High CPU, increasing tail latency: CPU bound or GC overhead. Optimize allocations, reduce serialization, or scale up/out.
• Flat throughput, rising queue times: A hard limit (thread pool, DB pool, rate limit). Increase capacity or add backpressure.
• High error rate during spikes: Timeouts and retries compounding. Tune retry policies, implement circuit breakers, and fast-fail when upstreams are degraded.

Optimization tactics that pay off

• Focus on p95/p99: Tail latency hurts user experience. Optimize hot paths and reduce variance.
• Batch and cache: Batch N small calls into one; cache idempotent results with coherent invalidation.
• Control concurrency: Limit in-flight work with semaphores; apply backpressure when queues grow.
• Right-size connection/thread pools: Measure saturation and queueing. Bigger isn’t always better; you can overwhelm the DB.
• Reduce payloads: Compress and trim large JSON; paginate heavy lists.
• Tune GC and memory: Reduce allocations; choose GC settings aligned to your latency targets.

Governance without red tape

• Publish SLOs for key services and pages. Keep them visible on team dashboards.
• Define performance budgets for new features and enforce them in code review and CI.
• Keep a living playbook of bottlenecks found, fixes applied, and lessons learned. Reuse scenarios across teams.

Common mistakes to avoid

• Testing the wrong workload: A neat, unrealistic script is worse than none. Base models on production logs when possible.
• Chasing averages: Median looks fine while p99 burns. Always report percentiles.
• Ignoring dependencies: If third-party latency defines your SLO, model it.
• One-and-done testing: Performance is a regression risk. Automate and re-run on every significant change.
• Assuming autoscaling solves everything: It helps capacity, not necessarily tail latency or noisy neighbors. Measure and tune.

Quick checklist

• Clear goals and SLOs defined
• Realistic workloads with proper data variance
• Baseline, load, stress, spike, and soak tests planned
• Full observability: metrics, logs, traces
• Bottlenecks identified and fixed iteratively
• Automation in CI with performance budgets
• Production monitoring aligned to test metrics

In short, performance testing isn’t a one-off gate—it’s a continuous practice that blends measurement, modeling, and engineering judgment. With clear objectives, realistic scenarios, and disciplined analysis, you’ll not only keep your app fast under pressure—you’ll understand precisely why it’s fast, how far it can scale, and what it costs to stay that way.

Some books about performance:

These are Amazon affiliate links, so I make a small percentage if you buy the book. Thanks.

• Systems Performance (Addison-Wesley Professional Computing Series) (Buy from Amazon, #ad)
• Software Performance Testing: Concepts, Design, and Analysis (Buy from Amazon, #ad)
• The Art of Application Performance Testing: From Strategy to Tools (Buy from Amazon, #ad)

Overview on Performance Testing

What is Performance Testing?

What are characteristics of autonomic system?

Autonomic systems bring both challenges as well as opportunities for the future networking. The increasing numbers of users have had a negative impact on the complexity of the networks; it has also increased by multiple folds. Autonomic systems provide a solution for this problem. 

Characteristics of Autonomic System

1. High intelligence: These systems have more intelligence incorporated in to them which lets them tackle this increasing complexity easily.
2. Business Goal: They are driven by the business goal that the quality of experience of the user must be high. Even with the changing environment, there goals remain the same. But there are changes that take place in the low – level configurations. For example, when a user switches over to a low bandwidth network, the bit rate of the video has to be reduced in order to satisfy the goals of the business.
3. Complex operations: All the operations carried out in an autonomic system are complex in nature even for the simplest of the services. For example, authentication, video encoding, billing, routing, shaping, QoS prioritizing, admission control.
4. High level objectives: The human operator just has to specify the high – level objectives and it is left to the system whether it chooses to optimize one or more of the goals. In order to achieve this, the system has to translate these objectives in to low – level configurations.
5. Adaptability: The system has the ability to adapt itself to the current environment.
6. Policy continuum: There are a number of perspectives to this as mentioned below:

Ø  Business view: Includes guidelines, processes and goals.

Ø  System view: The service should be independent of the technology as well as the device that is being used.

Ø  Network view: It should be specific to technology but independent of the device.

Ø  Device view: Both technology and device specific.

Ø  Instance view: Operation should be specific to an instance.

7. Elements: The elements of the network are assumed to be heterogeneous by the autonomic communication systems whereas in plain autonomic computing the elements are taken to be as the homogeneous.
8. Distributed: These systems work up on a distributed environment.
9. Complexity: The complexity in autonomic systems is more because of the complex autonomic loop that includes the following operations:

Ø  Interaction between the context  and the business goals

Ø  The MAPE (monitor, analyze, plan and execute) loop.

10. Reliability: In autonomic systems, the network has the authority to decide for itself focusing on high level objectives. Autonomic systems rely heavily up on artificial intelligence. However, there are issues associated with artificial intelligence like it becomes difficult to intervene in between when the things go wrong.It is quite difficult to know whether the system is doing the things it is supposed to do or not.

11. Scalability: This is another major characteristic of autonomic systems. It is required to keep track of the large amounts of knowledge and information. Autonomic systems have three tools to take care of this:

Ø Distributed ontologies

Ø Distributed large – scale reasoning

Ø Exchanging only the useful information

ØDistributing information among the different components of the autonomic network.

But in these cases, detection of the conflicts is a difficult task. For handling the various interactions taking place the various autonomic components efficient protocols are required. 

Currently two approaches have been suggested for developing the autonomic networking systems namely:

1. Evolutionary Approach: Incorporating the autonomic behavior in to the pre – existing infrastructure. This approach will consist of updates in increments till a fully autonomic system is developed. This approach is more likely to be adopted even though it requires a lot of patchwork.

2.  Clean slate approach: This approach is focused up on re – designing of the internet.

Give an overview of the process of system performance validation with IBM Rational Performance Tester?

IBM brought the rational performance tester in to picture as an effort to automate the rigorous process of performance testing. Automated performance testing was required to make it possible to deliver high quality software to the end users.

- The Rational Performance Tester is a convenient means for accelerating the software system’s performance without impacting its quality. 

- Software performance bottlenecks can be identified very easily with the help of IBM rational performance tester. 

- Not only the presence of bottlenecks but also their causes can be identified.  

- It is important to make predictions regarding the behavior and performance of the software systems and applications under extreme load conditions. 

- Making wrong predictions can get the whole organization, its revenue as well as the customer’s at risk. 

This can prove to be hell devastating in today’s software engineering world where the competition is already soaring high. 

- Hence, a proper testing solution is required that would not only help you validate the performance of your product but also optimize and verify it under a number of load conditions. 

- IBM rational performance tester gives you more than just testing. 

- Using it, the scalability factor of many of the applications can be calculated. 

- The rational performance tester supports code free programming i.e., the programming knowledge is not much required. 

In this article we provide an overview of the system performance validation using the rational performance tester. 

- Performance tests are contained within performance test projects.

- Instead of creating a new project, a test can be simply recorded which in turn will create a project immediately. 

- After a test has been recorded, it can be edited so as to include data pools, data correlation and verification points. 

- Data pools are required for obtaining variable data. 

- At verification points, it is confirmed whether the test is running as expected or not. 

- Lastly the data correlation ensures that appropriate data is being returned for the request. 

- Furthermore, protocol specific elements can also be added to a test.

- The items to which you make modifications appear in italics whenever a test is edited. 

- After saving the test it changes in to regular type. 

- A workload can be emulated by adding user groups and other elements to a new schedule.

- After being done with all the additions to the schedule, it can be run. 

- You can either run it locally or with a default launch configuration. 

- The playback might be slowed down because of the delays in HTTP requests. - The performance results of the application can be evaluated after being generated dynamically during the execution process. 

- After execution, the results can be regenerated for analyzation and viewing purpose. 

- The coding problems which are known to cause disturbances in the performance can be found and fixed using the problem analysis tools. 

- The data regarding the response time breakdown can be obtained using an application that has been deployed in a production environment. 

- Another means for collecting this data is from a distributed application implemented in a testing environment. 

- Resource monitoring data including available network, disk usage, network throughput, processor usage and so on can also be collected and imported. 

- Resource monitoring data is important since one can have a comprehensive view of the system or application and this is helpful in the determination of problems.

- All the collected and imported data can be analyzed and later can be used in the location of the exact cause of the problem that is hampering the system performance.

What are different characteristics of Scalability Testing?

Scalability can be essentially defined as the ability of a software application, network, process or program to effective and gracefully handle the increasing workload and effectively and easily carry out the specified tasks assigned properly. Throughput is the best example for this ability of a software application.

- Scalability as such is very difficult to define without practical examples.
- Therefore, scalability is defined based on some dimensions.
- Scalability is very much needed in communication areas like in a network, in software applications, in handling huge databases and it is also a very important aspect in routers and networking.
- Software applications and systems having the property of scalability are called scalable software systems or applications.
- They improve throughput to surprising extent after addition of new hardware devices. Such systems are commonly known as scalable systems.
- Similarly if a design, network, systems protocol, program or algorithm is suitable and efficient enough and works well when applied to greater conditions and problems in which the input data is in large amount or the problem or situation has got several nodes, they are said to be efficiently scalable.

If, during the process of increasing the quantity of input data the program fails, the program is not said to scale. Scalability is so much needed in the field of information technology. Scalability can be measured in several dimensions. Scalability testing deals with testing of these dimensions only.

The kinds of scalability testing have been discussed in detail below:

- Functional scalability testing:
In this testing new functionalities which are added to the software application or the program to enhance and improve its overall working are tested.

- Geographic scalability testing:
This testing tests the ability of the software system or the application to maintain its performance and throughput, and usefulness irrespective of distributing of working nodes in some geographical pattern.

- Administrative scalability testing:
This testing deals with the increment of working nodes in software, so that a single difficult task is divided among smaller units making it much easier to accomplish.

- Load scalability testing:
This testing can be defined as the testing of the ability of a divided program to divide further and unite again to take light and heavy workload accordingly.

There are several examples available for scalability today. Few have been listed below:

- Routing table of the routing protocol which increases in extent with respect to the increase in the extent of network.
- DBMS (data base management system) is scalable in the sense that more and more data can be uploaded to it by adding new required devices.
- Online transaction processing system can also be stated as scalable as one can upgrade it and more transactions can be done easily at one time.
- Domain name system is a distributed system and works effectively even when the hosting is on the level of World Wide Web. It is scalable.

Scaling is done basically in two ways. These two ways have been discussed below:

- Scaling out or scaling horizontally: This method of involves addition of several nodes or work stations to an already divided or distributed software application. This method has led to the development of technologies namely batch processing management and remote maintenance which were not available before the discovery of this technology.

Scaling up or scaling vertically:
Scaling up or scaling vertically can be defined as the addition of hardware or software resources to any single node of the system. These resources can either be CPUs or memory devices. This method of scaling has led to a tremendous improvement in virtualization technology.

What are different design issues and attributes for web applications?

Design model contains enough information to reflect how requirements are translated into content and executable code. Design should be specific. It is an engineering activity. It leads to a high quality product. the major attributes for quality of web applications are:

- Security of web applications is the ability of WebApp and its server environment to stop unauthorized access or threat.
- Availability plays an important attribute. Availability is the measure of the percentage of time that a web application is available for use. The expectation of a end user regarding the availability of a web application is each and every moment. Using features available on one browser or platform makes the web application unavailable to those who work on different platform or browser.
- Scalability is whether the web application and interfacing systems are able to handle significant variation in volume or will the responsiveness drop. Web application should be designed in such a way that it is able to accommodate the burden.
- Time to market is a measure of quality from a business point of view.

Assessing content quality includes :
- whether the user needs are met by determining the scope and depth of content?
- whether the background and authority of content's authors be easily identified?
- whether it is possible to determine the currency of content, last update and what was updated?
- whether the content and its location stable?
- credibility of content?
- uniqueness of content?
- whether content is valuable to targeted user?
- whether the content is well organized and easily accessible?

Categories of Heuristics of Software Testing

There is a set of characteristics that lead to a testable software.
- Operability: The better it works, more efficiently it can be tested.The system should have few bugs or no bugs that should block the execution of tests and the product should evolve in functional stages.

- Observability: What we see is what we test. It includes a distinct output should be generated for each input, the current and past system states and variables should be visible during testing, all factors affecting the output should be visible, incorrect output should be easily identified, source code should be easily accessible and internal errors should be automatically detected and reported.

- Controllability: The better we control the software, the more the testing process can be automated and optimized. Check that all outputs can be generated and code can be executed through some combination of input. Check that the software and hardware states can be controlled directly by the test engineer. Check that inputs and outputs formats are consistent and structured. Check that the test can be conveniently specified, automated and reproduced.

- Decomposability: By controlling the scope of testing, we can quickly isolate problems and perform effective and efficient testing. The software system should be built from independent modules which can be tested independently.

- Simplicity: The less there is to test, the more quickly we can test it. the points to consider in this regard are functional, structural and code simplicity.

- Stability: The fewer the changes, the fewer are the disruptions to testing. The changes to software should be infrequent, controlled and not invalidating existing tests. The software should be able to recover well from failures.

- Understandability: The more information we will have, the smarter we will test. The testers should be able to understand well the design, changes to the design and the dependencies between internal, external and shared components.

- Suitability: The more we know about the intended use of the software, the better we can organize our testing to find important bugs.

The above heuristics can be used by a software engineer to develop a software configuration which is convenient to test and verify.