What is Throughput, Turnaround time, waiting time and Response time?

In this article we discuss about four important terms that we often come across while dealing with processes. These 4 factors are:

1.  Throughput

2.  Turnaround Time

3. Waiting Time

4.  Response time

What is Throughput?

- In communications networks like packet radio, Ethernet etc., throughput refers to the rate of the successful delivery of data over the channel. 

- The data might be delivered via either logical link or physical link depending on the type of communication that is being used. 

- This throughput is measured in the terms of bps or bits per second or data packets per slot. 

- Another term common in networks performance is the aggregate throughput or the system throughput. 

- This equals to the sum of all the data rates at which the data is delivered to each and every terminal in a network. 

- In computer systems, throughput means the rate of successful completion of the tasks by the CPU in a specific period of time. 

- Queuing theory is used for the mathematical analyzation of the throughout. 

- There is always a synonymy between the digital bandwidth consumption and the throughput. 

- Another related term is the maximum throughput.This bears synonymy with the digital bandwidth capacity.

What is Turnaround Time?

- In computer systems, the total time taken by the CPU from submission of a task or thread for execution to its completion is referred to as the turnaround time. 

- The turnaround time varies depending on the programming language used and the developer of the software.

- It deals with the whole amount of time taken for delivering the desired output to the end user following the start of the task completion process. 

- This is counted among the metrics that are used for the evaluation of the scheduling algorithms used by the operating systems. 

- When it comes to the batch systems, the turnaround time is more because of the time taken in the formation of the batches, executing and returning the output.

What is Waiting Time?

 

- This is the time duration between the requesting of an action and when it occurs. 

- Waiting time depends up on the speed and make of the CPU and the architecture that it uses. 

- If the processor supports pipeline architecture, then the process is said to be waiting in the pipe. 

- When the current task in processor is completed, the waiting task is passed on to the CPU for execution. 

- When the CPU starts executing this task, the waiting period is said to be over. 

- The status of the task that is waiting is set to ‘waiting’. From waiting status, it changes to active and then halts.

What is Response Time?

 

- The time taken by the computer system or the functional unit for reacting or responding to the input supplied is called the response time. 

- In data processing, there are various situations for which the user would perceive the response time:

Ø  Time between operator entering a request at a terminal  and

Ø  The instant at which appears the first character of the response.

- Coming to the data systems, the response time can be defined as the time taken from the receipt of EOT (end of transmission) of a message inquiry and start of the transmission in response to that inquiry. 

- Response is an important concept in the real time systems and it is the time that elapses between the dispatch of the request until its completion. 

- However, one should not confuse response time with the WCET.

- It is the maximum time taken by the execution of the task without any interference. 

- Response time also differs from the deadline. 

- Deadline is the time for which the output is valid. 

What are the metrics that can be used during performance testing?

One of the most important parts that together make up the complete and effective software testing is the performance testing. Perhaps no software system or application can do without performance testing and testing of any software system or application is incomplete without this one. 

"The determination of the performance of a software system or application in terms of how and when and responsiveness and stability under different particular work loads is nothing but performance testing".

Apart from this there are several other attributes that can be taken care of by the performance testing like for example:

1. Scalability
2. Reliability
3. Resource usage and so on.

The concept of performance testing falls under the concept of performance engineering. The practice of performance testing is aimed at building the performance in to the architecture as well as design of the software system or applications before the actual coding of the software system or application begins. 

We have so many other types of testing that together make up the complete performance testing and have been mentioned below:

1. Load testing
2. Stress testing
3. Soak testing or Endurance testing
4. Spike testing
5. Isolation testing and
6. Configuration testing

Metrics used during Performance Testing

The performance testing cannot be carried out all alone by itself! Rather it is supported by some specific metrics called performance metrics. In this article we shall discuss about the metrics that are to be used during the performance testing.  We shall discuss them one by one:

 1. Average response time: 

This is the time that takes in to consideration all the response cycle ups or round trip requests until a particular point is reached and is the mean of all the response times. Response times can be measured in any of the following way:

      a) Time to last byte or 

      b) Time to first byte

   

      2. Peak response time: 

    This is similar to the previously mentioned metric and represents the longest cycle at a particular point in a particular test. Peak response time is an indication of some potential problem in our software system or application.

   

     3. Error rate: 

   The occurrence of errors under load during the processing of the requests is perhaps the most expected thing during the testing. it has been noted that the most of the errors are reported when the load on the application reaches a peak point and further from there the software system or application is not able to process any requests. Thus error rate provides a percentage of the HTTP status code responses as errors on the servers.

    

     4. Throughput: 

   This performance metric gives you a percentage of the flow of data back and forth from the servers and is measured in units of KB per second.

   

    5. Requests per second (RPS): 

    It gives a measure of the requests that are sent to the target server including requests for CSS style sheets, HTML pages, JavaScript libraries, flash files, XML documents and so on.

    6. Concurrent users: 

    This performance metric is perhaps that best way for expressing the load that is being levied on the software system or application during the performance testing. This metric is not to be equated to the RPS since there are possibilities of one user only generating a high number of requests and on the other hand a user not constantly generating requests.

     7. Cross result graphs: 

     It shows difference between post tuning and pre-tuning tests. 

What are different characteristics of software performance testing?

Software performance is indeed an important part of software engineering and software development plan. It can be defined as the testing carried out to determine the quality and standard of the response and stability of the software system under a certain work load. Sometimes it an also be used to examine other qualitative aspects of the software system like scalability, reliability, security, stress, and resource usage.

In actual, the software performance testing is essentially a part of performance engineering. This is a very crucial testing methodology and is gaining popularity day by day. It is a testing methodology which seeks to raise the standards of the performance factors of the design of the software system. It is also concerned with the architecture of the internal structure of a software system or application.

Performance testing tries to build excellent performance into the architecture and design of the software system before the actual coding of the software application or system. Performance testing consists of many sub testing genres.

Few have been discussed below:

- Stress testing
This testing is done to determine the limits of the capacity of the software application. Basically this is done to check the robustness of the application software. Robustness is checked against heavy loads i.e. above the maximum limit.

- Load testing
This is the simplest of all the testing methods. This testing is usually done to check the behavior of the application or software or program under different amounts of load. Load can either be several users using the same application or the difficulty level or length of the task. Time is set for task completion. The response timing is recorded simultaneously. This test can also be used to test the databases and network servers.

- Spike testing
This testing is carried out by spiking the particular and observing the behavior of the concerned application software under each case that whether it is able to take the load or it fails.

- Endurance testing
As the name suggests the test determines if the application software can sustain a specific load for a certain time. This test also checks out for memory leaks which can lead to application damage. Care is taken for performance degradation. Throughput is checked in the beginning, at the end and at several points of time between the tests. This is done to see if the application continues to behave properly under sustained use or crashes down.

- Isolation testing
This test is basically done to check for the faulty part of the program or the application software.

- Configuration testing
This testing tests the configuration of the application software application. It also checks for the effects of changes in configuration on the software application and its performance.

Before carrying out performance testing some performance goals must be set since performance testing helps in many ways like:
- Tells us whether the application software meets the performance criteria or not.
- It can compare the performance of two application soft wares.
- It can find faulty parts of the program.

There are some considerations that should be kept in mind while carrying out performance testing. They have been discussed below:

- Server response time:
It is the time taken by one part of the application software to respond to the request generated by another part of the application. The best example for this is HTTP.

- Throughput
It can be defined as the highest number of users who use concurrent applications and that is expected to be handled properly by the application.
A high level plan should be developed for performing software performance testing.

Volume tests - Volume Testing of Messaging systems

Volume tests are often most appropriate to messaging, batch and conversion processing type situations. In a volume test, there is often no such measure as response time. Instead, there is usually a concept of throughput. A key to effective volume testing is the identification of the relevant capacity drivers. A capacity driver is something that directly impacts on the total processing capacity. For a messaging system, a capacity driver may well be the size of messages being processed.

Most messaging systems do not interrogate the body of the messages they are processing, so varying the content of the test messages may not impact the total message throughput capacity, but significantly changing the size of the messages may have a significant effect. However, the message header may include indicators that have a very significant impact on processing efficiency. For example, a flag saying that the message need not be delivered under certain circumstances is much easier to deal with than a message with a flag saying that it must be held for delivery for as long as necessary to deliver the message, and the message must not be lost. In the former example, the message may be held in memory, but in the later example, the message must be physically written to disk multiple times.

Before conducting a meaningful test on a messaging system, the following must be known:

- the capacity drivers for the messages.
- the peak rate of messages that need to be processed, grouped by capacity driver.
- the duration of peak message activity that needs to be replicated.
- the required message processing rates.


A test can then be designed to measure the throughput of a messaging system as well as the internal messaging system metrics while that throughput rate is being processed. Such measures would typically include CPU utilization and disk activity. It is important that a test be run, at peak load, for a period of time equal to or greater than the expected production duration of peak load.

What is the need to execute Network Sensitivity Tests?

The three principle reasons for executing network sensitivity tests are as follows:
- Determine the impact on response time of WAN link.
- Determine the capacity of a system based on a given WAN link.
- Determine the impact on the system under test that is under dirty communications load.
Execution of performance and load tests for analysis of network sensitivity require test system configuration to emulate a WAN. Once a WAN link has been configured, performance and load tests conducted will become Network Sensitivity Tests.

There are two ways of configuring such tests:
- Use a simulated WAN and inject appropriate background traffic
This can be achieved by putting back to back routers between a load generator and the system under test. The routers can be configured to allow the required level of bandwidth, and instead of connecting to a real WAN, they connect directly through to each other.
When back to back routers are configured to be part of a test, they will basically limit the bandwidth. If the test is to be realistic, then additional traffic will need to be applied to the routers. This can be achieved by a web server at one end of the link serving pages and another load generator generating
requests. It is important that the mix of traffic is realistic.

For example, a few continuous file transfers may impact response time in a different way to a large number of small transmissions. By forcing extra more traffic over the simulated WAN link, the latency will increase and some packet loss may even occur. While this is much more realistic than testing over a high speed LAN, it does not take into account many features of a congested WAN such as out of sequence packets.

- Use the WAN emulation facility within LoadRunner
The WAN emulation facility within LoadRunner supports a variety of WAN scenarios. Each load generator can be assigned a number of WAN emulation parameters, such as error rates and latency. WAN parameters can be set individually, or WAN link types can be selected from a list of pre-set configurations.

It is important to ensure that measured response times incorporate the impact of WAN effects both at an individual session, as part of a performance test, and under load as part of a load test, because a system under WAN affected load may work much harder than a system doing the same actions over a clean communications link.

How to execute Performance Tests?

Performance testing involves executing the same test case multiple times with data variations for each execution, and then collating response times and computing response time statistics to compare against the formal expectations. Often, performance is different when the data used in the test case is different, as different number of rows are processed in the database, different processing and validation come into play, and so on. By executing a test case many times with different data, a statistical measure of response time can be computed that can be directly compared against a formal stated expectation.

Network sensitivity tests are variations on load tests and performance tests that focus on the Wide Area Network (WAN) limitations and network activity. Network sensitivity tests can be used to predict the impact of a given WAN segment or traffic profile on various applications that are bandwidth dependent. Network issues often arise at low levels of concurrency over low bandwidth WAN segments. Very chatty applications can appear to be more prone to response time degradation under certain conditions than other applications that actually use more bandwidth. For example, some applications may degrade to unacceptable levels of response time when a certain pattern of network traffic uses 50% of available bandwidth, while other applications are virtually un-changed in response time even with 85% of available bandwidth consumed elsewhere.

This is particularly important test for deployment of a time critical application over a WAN. Also, some front end systems such as web servers, need to work much harder with dirty communications compared with clean communications encountered on a high speed LAN in an isolated load and performance testing environment.

What is Long Session Soak Testing ?

When an application is used for long periods of time each day, the above approach should be modified, because the soak test driver is not logins and transactions per day, but transactions per active user for each user each day. This type of situation occurs in internal systems, such as ERP and CRM systems, where user logins and stay logged in for many hours, executing a number of business transactions during that time. A soak test for such a system should emulate multiple days of activity in a compacted time frame rather than just pump multiple days worth of transactions through the system.

Long session soak tests should run with realistic user concurrency, but the focus should be on the number of transactions processed. VUGen scripts used in long session soak testing may need to be more sophisticated than short session scripts, as they must be capable of running a long series of business transactions over a prolonged period of time.

The duration of most soak tests is often determined by the available time in the test lab. There are many applications that require extremely long soak tests. Any application that must run, uninterrupted for extended periods of time, may need a soak test to cover all of the activity for a period of time that is agreed to by the stakeholders. Most systems have a regular maintenance window, and the time between such windows is usually a key driver for determining the scope of soak test.

Overview of Soak testing.

Soak testing is running a system at high levels of load for prolonged periods of time. A soak test would normally execute several times more transactions in an entire day than would be expected in a busy day, to identify any performance problems that appear after a large number of transactions have been executed. Also, it is possible that a system may stop working after a certain number of transactions have been processed due to memory leaks or other defects. Soak tests provide an opportunity to identify such defects, whereas load tests and stress tests may not find such problems due to their relatively short duration.A soak test would run for as long as possible, given the limitations of the testing situation. For example, weekends are often an opportune time for a soak test.

There are some typical problems identified during soak tests are:
- Serious memory leaks that would eventually result in memory crisis.
- Failure to close connections between tiers of a multi-tiered system under some circumstances which could stall some or all modules of the system.
- Failure to close database cursors under some conditions which would eventually result in the entire system stalling.
- Gradual degradation of response time of some functions as internal data structures becomes less efficient during a long test.

Apart from monitoring response time, it is also important to measure CPU usage and available memory. If a server process needs to be available for the application to operate, it is often worthwhile to record its memory usage at the start and end of the soak test. It is also important to monitor internal memory usages of facilities such as Java virtual machines, if applicable.

How to set up a Load Test using LoadRunner?

The important thing to understand in executing such a load test is that the load is generated at a protocol level, by the load generators, that are running scripts that are developed with the VUGen tool. Transaction times derived from the VUGen scripts do not include processing time on the client PC, such as rendering (drawing parts of the screen) or execution of client side scripts such as JavaScript.The WinRunner PC(s) is utilized to measure end user experience response times. Most load tests would not employ a WinRunner PC to measure actual response times from the client perspective but is highly recommended where complex and variable processing is performed on the desktop after data has been delivered to the client.

The LoadRunner controller is capable of displaying real-time graphs of response times as well as other measures such as CPU utilization on each of the components behind the firewall. Internal measures from products such as Oracle, Websphere are also available for monitoring during test execution.
After completion of a test, the analysis engine can generate a number of graphs and correlations to help locate any performance bottlenecks.

In simplified load test, the controller communicates directly to a load generator that can communicate directly to the load balancer. No WinRunner PC is utilized to measure actual user experience. The collection of statistics from various components is simplified as there is no firewall between the controller and the web components being measured.

Validation phase - System Testing - Performance Testing - Throughput

Throughput refers to the number of client requests processed within a certain unit of time. Typically, the unit of measurement is requests per second or pages per second. From a marketing perspective, throughput may also be measured in terms of visitors per day or page views per day, although smaller time units are more useful for performance testing because applications typically see peak loads of several times the average load in a day.

The throughput of the web site is often measured and analyzed at different stages of the design, develop, and deploy cycle. For example, in the process of capacity planning, throughput is one of the key parameters for determining the hardware and system requirements of a web site. Throughput also plays an important role in identifying performance bottlenecks and improving application and system performance. Whether a web farm uses a single server or multiple servers, throughput statistics show similar characteristics in reactions to various user load levels.

The throughput of a typical web site increases proportionally at the initial stages of increasing load. However, due to the limited system resources, throughput cannot be increased indefinitely. It will eventually reach a peak, and the overall performance of the site will start degrading with increased load. Maximum throughput is the maximum number of user requests that can be supported concurrently by the site in the given unit of time. The value of maximum throughput varies from site to site. It mainly depends on the complexity of the application. As with any statistic, throughput metrics can be manipulated by selectively ignoring some of the data.

In many ways, throughput and response time are related, as different approaches to thinking about the same problem. In general, sites with high latency will have low throughput. If you want to improve your throughput, you should analyze the same criteria as you would to reduce latency. Also, measurement of throughput without consideration of latency is misleading because latency often rises under load before throughput peaks. This means that peak throughput may occur at a latency that is unacceptable from an application usability standalone.