Some white box testing techniques – part 2

This post is a continuation of the discussion on white box testing techniques from part 1 of this article (link).

- Path testing: This is a typical application program that consists of 100s of paths between two points of the program i.e., the entry and the exit. The number of paths goes on multiplying by the number of decisions that are encountered during the execution. This number is again increased by the execution of the loops. Each of these paths has to be tested to ensure there are no bug. This seems quite easy for a straight line having 50 – 100 lines of code but what about a large program? Counting gets difficult. With a limited number of resources (and most software projects have a limited number of resources), all the possible paths have to be covered. Path testing can be made a little easy by not consider the redundant code in the program as far as possible. Path testing is responsible for checking whether or not each and every path is getting executed at least once. Today we also have a new type of path testing, called the basis path testing which is a mixture of branch testing and path testing. Even the path testing makes use of the control flow graph. It involves calculating the cyclomatic complexity. For each & every path, a test case has to be designed.

- Statement coverage: We also know this white box testing technique by other names such as segment coverage or line coverage. This coverage methodology provides coverage only to the conditions that are true. The false conditions are neglected. This helps in the identification of the statements that are often executed in the program. It also helps in finding out the areas of the program where there is no data flow because of no execution. But all the lines of code are examined and executed. Statement coverage is important for verifying whether the code is serving its desired purpose or not. In other words it gives us a measure of the code quality. We can consider it as a partial form of path testing. Disadvantages include: false conditions are left untested; it gives no report about the loop termination and does not consider the logical operators. One line of code might be a full statement, a partial one or blank or may contain many statements. There are nested statements too. The statement coverage has to be measured first and then the test cases should be designed.

- Decision coverage: Another name for decision coverage is “all – edges coverage”. Here all the conditions that result because of a logical decision are considered. It does not matter whether the conditions are true or false. This is just the opposite of statement coverage. The outcome of a decision is nothing but a branch. So the branch coverage gives a measure of extent up to which the decisions in a program have been tested. This is why most of us often get confused with branch coverage and decision coverage and use them interchangeably. But it is more than statement coverage. The decision statements include loop control statements, if statements, if – else statements, switch – case statements i.e., the statements that result in either a true or a false value based up on the logical condition in that statement. This white box testing technique is helpful in validation of branch coverage that is whether or not all branches have been covered. It helps in ensuring that no branch gives an abnormal result that may disturb the operation of the whole program. There are certain drawbacks of the statement coverage as we saw above. These drawbacks are eliminated by decision testing. 

Some white box testing techniques – part 1

In this article we shall give a brief description of the various techniques that are used in white box testing.

- Control flow testing: This is more of being similar to structural testing and is based on the control flow model of the application that is being testing. In control flow testing certain paths among all are selected judiciously so as to provide maximum coverage. The paths are chosen so that a decided thoroughness in the testing process is maintained. This means that paths that have been selected should be enough so as to cover all the statements at least once. It is usually applied to the new software systems that have to be put under unit testing (unit level white box testing). This technique assumes that there are correct specifications, correctly assessed and defined data, no bugs in the program except those in the control flow. The number of control flow bugs is fewer in the programs that are written using object – oriented or structured programming languages. This technique makes use of a flow graph usually called the control flow graph that represents the control structure of the program.

- Data flow testing: Flowing data in a program is what keeps a program active. If there is no data flow, the program won’t be able to perform any operation. The program’s data flow needs to be tested to ensure that it is constant and efficient. Data flow testing also requires a data flow graph. This graph is required to know where and why the data is going or if it is going to correct destination or not. This testing helps in uncovering any anomalies that might restrict the flow of data in a program. Knowing these anomalies also helps in branch and path testing. A number of methods and techniques go in to data flow testing for exploring various events that are taking place in the program, whether right or wrong. It is used for checking whether all the data objects have been initialized or not and makes sure that they are used at least once in the whole program. Arrays and pointers are considered to be the two major elements that play a critical role in data flow testing. These two elements cannot be neglected. Data flow testing might include static anomaly detection, dynamic anomaly detection and anomaly detection through compliers.

- Branch testing: As the name suggests, this testing technique is used for testing all the branches of all the loops within a program. Here branch coverage plays a great role. It has to be made sure that each and every branch in the program is executed once. Thus the test cases are designed in such a way that all the branches are covered. This technique is just for complementing the white box testing at unit testing level. The programmers aim that their test cases must provide 100 percent branch coverage. Also the coverage has to be proper otherwise it can lead to other potential problems such as removal of the code that was actually correct and insertion of faulty code. But providing 100 percent coverage is not possible and we are always left with some bugs and errors that never come to the scene. Branch testing lets you uncover errors which lie in those parts of the program that are least executed or never executed. But there is a potential drawback too; which is; that it is very ineffective in uncovering errors in interactions of structures and decisions. Because of this drawback, the testers usually prefer to go for path testing.

Read more about this in Part 2 - White Box testing techniques.

What are characteristics of fuzz technique? What are uses of fuzz technique?

Fuzz testing or fuzzing as it is commonly known as, is another type of very popular mixed software testing methodology i.e., it can be either implemented as a black box testing technique or white box testing technique or even as grey box testing technique, though the latter case being very rare. 

- It also provides the options for running it manually or by some semi automated or fully automated processes. 

- Unexpected, invalid or random data is what all is fed as input data values to the test cases created for the fuzz testing of a software system or application. 

- After the arrangement of the test input data values for the test cases, the whole software system or application is subjected to an inspection to determine if at all it is experiencing any problems like crashes, memory leaks or failure of the code assertions that are nothing but built in defects in the software system or application. 

- Fuzz testing or fuzzing is actually a testing methodology that is categorized under the category of the security testing and is often used for security checks of a software system or application. 

Fuzz testing if further divided results in to two sub categories as described below:

1. Generation based fuzz testing: 

This type involves creation of entirely new set of input data based on the model input data and

2. Mutation based fuzz testing: 

This type is concerned with the generation of new set of data by mutating the existing samples.

Fuzz testing lays more emphasis more on the network protocols and the file formats of the software system or application than the other aspects of the system. But this does not hold necessary that only the network protocols and file formats should undergo fuzz testing, other aspects of the input data for the system can also be subjected to the fuzz testing. Common input for the fuzz testing basically includes:

1. Key board events
2. Mouse events
3. Environment variables and
4. API calls sequence.

The unusual input forms are:

Such as those being mentioned below can also be subjected to fuzz testing:

1. Shared memory
2. Contents of the data base of the software system or application and
3. Inter leaving of threads (precise)

The input data which manages to cross the trust boundary of the software system or application catches the attention of the testers more than any other happening during the testing. The origination of the fuzz testing is traced back to the University of Wisconsin and discovered by the professor Barton Miller. 

Uses of Fuzz Testing

The fuzz testing has emerged as a software testing technique with more than one uses:

1. For the testing of large projects having a budget just enough for the development of the test tools it can be employed as black box testing technique.
2. It is a very feasible and affordable software testing technique and has a high benefit- to - cost ratio.
3. The fuzz testing proves helpful in providing a sample stating the behavior of the software system or application generated in a random manner.
4. It very effectively demonstrates the exception handling capability of a piece of the source code of the program without crashing.
5. Fuzz testing gives an assurance for the maintenance of the overall quality of the software system or application rather than just acting as a testing tool for finding bugs.
6. Fuzz testing can also be used as a substitute for the formal methods employed for exhaustive testing of the software system.
7. One can rely on fuzz testing for determining the reliability of the software system as an application of static analysis, code audits or partial rewrites.

Explain the concepts of fuzz test technique?

WHAT IS FUZZ TESTING?

- Fuzz testing is the formal name for fuzzing which as we know is another software testing technique that involves playing with the software system or application using all types of possible invalid input test data.

- By the invalid input test data we mean it can be either unexpected data or random data i.e., any other type of test data other than the specified input data type.

- It is not necessary that the fuzz testing should always be automated; it can also be semi automatic though keeping the whole process fully automated consumes less time and effort.

- Semi automatic process is used only when there is manual interference required in the testing.

- After feeding the invalid input data to the software system or application, the behavior of the system is monitored for any exceptions like hanging, crashing or failing of the code assertions that are built in and also check for any memory leaks.

From the above discussions we can easily make out that the fuzz testing has been developed for testing of the security related issues of the software system or application. Till now two types of fuzz testing have been identified namely:

1. Mutation based testing
2. Generation based testing

CONCEPTS OF FUZZ TESTING

- Fuzz testing was developed to complement the negative testing and syntax testing.

- Both of the above types of the fuzz testing can be either employed as white box testing technique or black box testing technique or as a mix of the two techniques i.e., grey box testing.

- Whatever technique may be used, it is focussed up on the common target i.e., network protocols, file formats and so on.

- Though these two are the usually targeted elements, almost any type of input test data for the program can be subjected to the fuzz testing.

- The most common types of input data include:
1. Sequence of API calls
2. Environment variables
3. Mouse events
4. Keyboard events etc.

- Apart from just testing the input test data types, even some elements like shared memory, contents of a data base, interleaving of threads etc can also be tested by the fuzz test.

- But, usually the input that is able to cross the trust boundary of the security of the software system or application is targeted.

- Among all other approaches to fuzz testing, the one which is mostly preferred is the black box approach.

- This approach is mostly employed for the testing of large software projects having a budget for the development of the testing tools.

- Fuzz testing is included in the class of the software testing techniques that offer a high benefit – to – cost ratio.

- Like other software testing techniques, fuzz testing also has a drawback which is that it is able to develop only a sample of the behavior of the software system.

- In some rare cases, if the software passes the fuzz test, it merely indicates that only a part of the software system can effectively handle the unexpected input data types without any problem.

- This tells us that fuzz testing can be considered as an overall quality assurance factor and not merely just a bug finding tool.

- Fuzz testing is also not to be taken as a substitute neither for the formal methods nor for the exhaustive testing.

- It also gives us a gross measurement of the reliability of the software.

- Based on the results of the fuzz testing, it can be decided that which part needs a partial rewrite, static analysis or code audits.

What are different characteristics of usability testing?

Usability testing can be defined as a technique which is used in the interaction design centered around the user for evaluation of a software system, application or product by testing it out on the software product users. This can be thought of as an irreplaceable practice of usability. It is considered as irreplaceable usability practice because it generally gives direct report on how the real users use the software system or application or the product.

From the mentioned things, we can say that the usability testing is highly in contrast with inspection methods for usability.

- In usability inspection methods the experts seem to use several different methods to evaluate and check a GUI or graphical user interface without the involvement of the real users.
- The main focus of the Usability testing is on measuring a software system’s or product’s capacity of achieving its aim and objectives made by humans.

Best examples of the software systems or products that are commonly benefited from usability testing are the web sites and the web applications, the computer user interfaces, documents, databases and devices.

- The Usability testing usually measures the usability i.e., to say the ease of use of some specific object or group of many objects.
- However, the general human- computer interaction is different from this in the way that the human – computer interaction makes the attempts to formulate the principles of the universe.
- Usability testing is basically a black box testing methodology.
- It aims at observing people using the software system, product or application in their own way and discovering new errors and bugs.
- They can also find some ways for the improvement of the software system or the application.

Usability tests determine how well the output comes from the following 4 fields when tested with usability testing test cases:

1. Efficiency or the performance
It includes how much time the software system is taking to respond? In how many the software system is completing the assigned task?

2. Accuracy:
It includes determining the number of mistakes the users make? And were they recoverable or they caused the software system failure?

3. Recall:
It includes determining what the user remembers afterwards i.e., after the time of usage or after the periods of non- usage.

4. Emotional response:
It involves determining the level of satisfaction of the users about the completed tasks? Does the confidence level of person has increased or decreased? Is the user stressed? Is the user confident enough to recommend this software application to others?

Some people have a misconception of usability testing.
- They think that simply collecting opinions on some software system or documentation is called usability testing.
But this is absolutely wrong.

- Gathering of opinions on some object is simply called qualitative research or market research.
- It’s important to clear up the concept of the usability testing.
- It can be though of as a testing involving systematic observation under certain conditions which are controlled by experts.
- This is done basically to have an understanding about how people use that particular software system or application.
- But usually usability testing is used in combination with qualitative research. - This helps in better understanding of user’s expectations.
- It involves watching people use that software system or the product for its intended objective.
- For carrying out a successful usability test, one needs to create a realistic environment where the user or the tester is required to perform the tasks which can be performed by the software system and the observers watch the testing and record the results and the observations.
- Other forms of feedback are also gathered like pre and post test questionnaires, scripted instructions, customer feed back and paper prototypes etc.

What is static testing and what are its components ?

What is static testing? Static testing can be defined as a kind of software testing methodology in which the software is tested without actually being compiled and executed. Therefore, static testing is just the opposite of dynamic testing where the software is compiled and executed and then tested. Static testing does not goes into the detail of the software application but, checks for the correctness of the program or source code, document and algorithm. Basically syntax of the program is checked in static testing.
It is done manually by a team of professionals and qualified individuals in the concerned field. Major errors can be found out using this methodology. The writer of code himself/ herself can review the source code to check out for errors. There are many techniques followed for static testing. But the most commonly used ones are the following:
1. Code inspections
2. Code walk through and
3. Code reviews
From the idea of black box testing techniques, we can say that the review of specifications and requirements forms an important part of static testing. This is also done manually, though this is a tough task. Static testing forms the verification part of verification and validation. Nowadays there are some methodologies available by which static testing can be made automatic. This an be done by passing the software code through a static testing test suite that consists of a compiler or an interpreter which checks the software code for verification and syntax errors. Errors found during static testing are much easier to correct than the ones that will be found later. Professionals handling static testing are typically testers and application developers. Static testing is only concerned with verification activities.
While carrying out static testing for software applications certain standards are followed which are namely, standards for integration, deployment and coding. Usually static testing nowadays is done by some automated program or a tool. The analysis done manually by professionals is commonly known as program comprehension or sometimes it is called program understanding. The quality of analysis varies from professional to professional. Some analyze software part by part while some take the whole software code into consideration. Nowadays reverse engineering and software metrics are considered to be methodologies of static testing. Static testing is popular when it comes to the field of verification of the software code, computer systems needing high safety and location of potentially harming code.
Static testing also involves some formal methods for analysis. In formal methods, the analysis result for a software code is usually obtained by carrying out some rigorous mathematical calculations and methods. Following are some mathematical methodologies that are used under formal static testing:
1. Axiomatic semantics
2. Denotational semantics
3. Operational semantics and
4. Abstract interpretation: this technique is based on the idea that every statement executes based on its mathematical properties and values of its declaration. Out of all this technique can be regarded as the best technique.
There are some other techniques apart from mathematical techniques that can be used for formal static analysis. They are called implementation techniques and have been listed below:
1. Model checking: this technique takes into consideration the finite state of the system. If the system is not in finite state, it is made finite by using the technique of abstraction.
2. Use of assertions
No matter how many methodologies we may use for static testing, there will always be some uncertainty of the execution of the program and flaws. It cannot be said that after static analysis the program will execute 100 percent properly.

Some related books: