Overview of Performance Testing Tools

In client/server applications, many users access the server application simultaneously. In such a case, the software has to be tested for its performance.
The server application continues to run on the server but only one or a few clients will be running the client application but on these client machines, many virtual users are simulated, effectively simulating an environment in which 100 users access the database.
Today many enterprise applications such as ERP software, web services, web sites, CRM software etc. need to be tested for performance testing. Performance testing is also called load testing or stress testing. These tools display the performance of the application using graphs and charts so that analysis will become easy. You can plot a graph which shows the response time as a function of the number of simultaneous users who access the database.
WAPT 6.0 is a load and stress testing tool that provides you with an easy-to-use, consistent and cost-effective way of testing web sites, web servers, and intranet applications with web interfaces. WAPT has a set of features to test web sites with dynamic content and secure HTTPS pages. It provides informative test results through descriptive graphs and reports.
The performance tools that are widely used in the software industry are Mercury Interactive's LoadRunner and Apache's JMeter.

Functional/ Regression Testing Tools

In functional testing, the functionality of the software is tested without taking into consideration the source code and implementation details. For example. consider the standard calculator software. To test the functionality, some calculations will be done and check out whether the results are correct. If the expected results are same as the displayed results, the calculator software is working fine. While you are testing the software, you need to give test cases in such a way that all the buttons of the GUI are checked.
When the software is still in development phase, the software is not likely to work perfectly, and for some test cases you may get wrong results. During the course of development, you should check the software many times using the test cases. Doing it manually will take a lot of time, the functionality/regression testing tools are of immense use in such conditions.
Using these tools, GUI operations can be recorded. While running the tool, invoke the application and give all the test cases as input. Automatically, the testing tool will record your inputs like keystrokes and mouse clicks and then the tool will create a script. The script can be run subsequently and hence you are saved botheration of manually giving these inputs again and again.
testing tools provide a beautiful mechanism through data driven testing. In data driven testing, the username and password are considered as variables, the values that these variables can take can be obtained from a database and the testing is carried out. Data driven testing is one of the most important features of these testing tools. As a test engineer, the job is to create the test script with the variable names and then you can run the test in unattended mode.
While testing in unattended mode, suppose for an application, the user has to first login, then update a database table and then logout. You created test cases for all the three functions. If the software fails at login stage itself, should you continue or stop testing. The testing tools provide you an option either to continue or stop testing. The application can be made to recover automatically to a specified state.
The following tools used are Mercury Interactive's WinRunner, Segue Software's SilkTest, IBM Rational SQA Robot.

How are testing tools categorized ? - Part II

Embedded Software Testing Tools :
It is a very challenging task to test embedded software as the timing requirements for these applications are very stringent. The code has to be optimized so that it occupies the minimum memory. IBM Rational Test Real Time is the widely used test tool in this category.
Network Protocol Testing Tools :
Testing networking and communication software has attained a lot of importance in recent years as computer networks are becoming widespread. There are number of tools available for testing networking and communication protocols. ANVL(Automated Network Validation Library) is to test routers and other networking products. Agilent Technologies, Rhode and Schwartz etc supply protocol analyzers which generate the necessary protocols based on international standards such as ITU-T standards.
Configuration Management/Bug Tracking Tools
Configuration Management is a very important process. When test engineers report bugs, the managers have to track these bugs and ensure that all the bugs are removed. Bugzilla is an open source defect tracking system. Samba's Jitterbug is a freeware defect tracking system. IBM Rational Software's Clear DDTS is a change request management software. GNU's GNATS is a freeware bug tracking and change management software.
Software Testing Management Tools
These tools help in managing process oriented software testing. Using these tools, the QA manager can create a formal test plan, allocate resources, schedule unattended testing, track the status etc. AutoTester's AutoAdviser, Compuware's QA Director are such tools.

How are testing tools categorized ?

There is a wide variety of testing tools available to carry out different types of testing. the testing tools can be divided into the following categories :
- Source code Testing Tools
The tools that fall in this category check the source code of the application. the testing is white box testing and so the implementation details are considered. A number of tools are available to check line coverage, branch coverage, and path coverage. Tools are also available to test whether the source code is compliant to the standard coding guidelines and to generate the metrics.
- Functional/Regression Testing Tools
These tools are used to test the application software and web applications such as web sites. Applications involve Graphical User Interface, the tools test the GUI objects and functionality automatically. These tools carry out black box testing.
- Performance Testing Tools :
These tools are used to carry out performance testing or stress testing. These tools are very useful to test how the application works when multiple users access the application simultaneously. These tools simulate multiple users on a single machine and hence you do not need to have many machines and many test engineers can do performance testing. E.g. AutoTester's AutoController, Mercury Interactive's Load Runner, Apache JMeter, belong to this category.
- Java Testing Tools :
A number of tools are available exclusively for testing Java applications. These tools test the applications written in Java programming language and for testing java classes. Jemmy is an open source library to create automated tests for Java GUI applications.

What are the advantages of automated testing over manual testing ?

Manual testing is time consuming, error prone and requires lot of infrastructure and manpower. All these limitations can be overcome if the testing process becomes automated. There are many advantages in using these automated testing tolls :
- To improve the quality and reliability of the software, the functionality of software can be repeatedly tested using automated testing tools.
- Testing can be done during nighttime and during holidays i.e. testing can be done unattended using automated testing tools.
- The labor involved can be reduced using automated testing tools when the software has to be tested in different hardware platforms, different operating systems, using different browsers etc.
- Performance testing can be done without the need for many computers and many test engineers. The test tools simulate multiple users on a single machine. As compared to manual testing, finding out transaction response times when multiple users access the same application will be very easy.
- By effectively using the automated testing tools, testing process can be planned and managed effectively.
- Automated testing tools give a systematic approach to the testing process.
- There is a drastic reduction in time, effort and money spent on testing.
- There is an improvement in quality and reliability of the software after using automated testing tools.
- Even if the teams are at different geographical locations, there is efficient management of the testing process.
- Test reports can be generated automatically.
- Testing can be done using tools that are available to test not only generic applications but also for DLLs, VB programs, Siebel, Power builder software etc.

Manual Testing and what are its drawbacks/limitations.

Software testing is carried out manually in many organizations. The test team generates various test cases and manually tests each and every feature after the product is reached its completion stage. If defect is found, It is rectified and again the software is tested using the test cases. There are few drawbacks or limitations of manual testing.
- Manual testing is error prone because test engineers become bored when testing has to be done repeatedly. To test a we site, each and every link has to be checked.
- Manual testing is very time consuming. The same set of operations need to be done repeatedly.
- Regression testing has to be done to ensure that changes in one portion of the software have no ill effects on other portions of the software.
- To do performance testing, many resources are required, both computer and people. On the other hand, automated tools facilitate running the software from a single client machine.
- Even to manage the testing process is complicated as the testing has to be planned, bugs have to be tracked and reliability analysis has to be performed.

Manual testing is time consuming, error prone and requires lot of infrastructure and manpower, these limitations can be overcome if the testing process is automated. The testing tools reduce manual testing to a large extent and the testing can be done done automatically.

Testing Approaches: Functional Testing versus Structural Testing and Mutation testing

In functional testing, functionality of the module is tested and structure is not considered. Test cases based on specifications and internals of modules are not considered. This type of testing is also known as black box testing. The objective here is to ensure that `correct results'' are obtained when `good inputs'' are applied to the various parts of the design, and when `bad inputs'' are applied the design operates in a predictable manner. Functional testing can therefore be considered as concentrating on checking that the data paths operate correctly.

Structural testing is used to test the implementation of the program. Structural testing, also known as white box or open box testing, is normally applied to sequential HDL code and concentrates on checking that all executable statements within each module. Structural testing involves :
- Statement coverage : This ensures that each and every statement is tested. Software tools called profilers are used to carry this statement coverage.
- Branch coverage : Each and every condition is taken, and inputs are given in such a way that each branch is executed at least once.
- Path Coverage : To test loops, this is required. For example the statement
for(i=o;i<=100;i++) is executed 101 times.

Mutation testing is required to ensure that the software does not fail. It is a good debugging mechanism. The program is modified slightly to obtain mutants of the program. Different mutants are tested with the same test cases. If the mutant fails, and the actual program works correctly, confidence is gained in the program and test cases are considered good. To produce mutants, mutation operators are defined.
- Constant replacement
- Variable replacement
- Arithmetic operator replacement
- Relational operator replacement
- Goto label replacement