Some white box testing techniques – part 2

This post is a continuation of the discussion on white box testing techniques from part 1 of this article (link).

- Path testing: This is a typical application program that consists of 100s of paths between two points of the program i.e., the entry and the exit. The number of paths goes on multiplying by the number of decisions that are encountered during the execution. This number is again increased by the execution of the loops. Each of these paths has to be tested to ensure there are no bug. This seems quite easy for a straight line having 50 – 100 lines of code but what about a large program? Counting gets difficult. With a limited number of resources (and most software projects have a limited number of resources), all the possible paths have to be covered. Path testing can be made a little easy by not consider the redundant code in the program as far as possible. Path testing is responsible for checking whether or not each and every path is getting executed at least once. Today we also have a new type of path testing, called the basis path testing which is a mixture of branch testing and path testing. Even the path testing makes use of the control flow graph. It involves calculating the cyclomatic complexity. For each & every path, a test case has to be designed.

- Statement coverage: We also know this white box testing technique by other names such as segment coverage or line coverage. This coverage methodology provides coverage only to the conditions that are true. The false conditions are neglected. This helps in the identification of the statements that are often executed in the program. It also helps in finding out the areas of the program where there is no data flow because of no execution. But all the lines of code are examined and executed. Statement coverage is important for verifying whether the code is serving its desired purpose or not. In other words it gives us a measure of the code quality. We can consider it as a partial form of path testing. Disadvantages include: false conditions are left untested; it gives no report about the loop termination and does not consider the logical operators. One line of code might be a full statement, a partial one or blank or may contain many statements. There are nested statements too. The statement coverage has to be measured first and then the test cases should be designed.

- Decision coverage: Another name for decision coverage is “all – edges coverage”. Here all the conditions that result because of a logical decision are considered. It does not matter whether the conditions are true or false. This is just the opposite of statement coverage. The outcome of a decision is nothing but a branch. So the branch coverage gives a measure of extent up to which the decisions in a program have been tested. This is why most of us often get confused with branch coverage and decision coverage and use them interchangeably. But it is more than statement coverage. The decision statements include loop control statements, if statements, if – else statements, switch – case statements i.e., the statements that result in either a true or a false value based up on the logical condition in that statement. This white box testing technique is helpful in validation of branch coverage that is whether or not all branches have been covered. It helps in ensuring that no branch gives an abnormal result that may disturb the operation of the whole program. There are certain drawbacks of the statement coverage as we saw above. These drawbacks are eliminated by decision testing. 

Some white box testing techniques – part 1

In this article we shall give a brief description of the various techniques that are used in white box testing.

- Control flow testing: This is more of being similar to structural testing and is based on the control flow model of the application that is being testing. In control flow testing certain paths among all are selected judiciously so as to provide maximum coverage. The paths are chosen so that a decided thoroughness in the testing process is maintained. This means that paths that have been selected should be enough so as to cover all the statements at least once. It is usually applied to the new software systems that have to be put under unit testing (unit level white box testing). This technique assumes that there are correct specifications, correctly assessed and defined data, no bugs in the program except those in the control flow. The number of control flow bugs is fewer in the programs that are written using object – oriented or structured programming languages. This technique makes use of a flow graph usually called the control flow graph that represents the control structure of the program.

- Data flow testing: Flowing data in a program is what keeps a program active. If there is no data flow, the program won’t be able to perform any operation. The program’s data flow needs to be tested to ensure that it is constant and efficient. Data flow testing also requires a data flow graph. This graph is required to know where and why the data is going or if it is going to correct destination or not. This testing helps in uncovering any anomalies that might restrict the flow of data in a program. Knowing these anomalies also helps in branch and path testing. A number of methods and techniques go in to data flow testing for exploring various events that are taking place in the program, whether right or wrong. It is used for checking whether all the data objects have been initialized or not and makes sure that they are used at least once in the whole program. Arrays and pointers are considered to be the two major elements that play a critical role in data flow testing. These two elements cannot be neglected. Data flow testing might include static anomaly detection, dynamic anomaly detection and anomaly detection through compliers.

- Branch testing: As the name suggests, this testing technique is used for testing all the branches of all the loops within a program. Here branch coverage plays a great role. It has to be made sure that each and every branch in the program is executed once. Thus the test cases are designed in such a way that all the branches are covered. This technique is just for complementing the white box testing at unit testing level. The programmers aim that their test cases must provide 100 percent branch coverage. Also the coverage has to be proper otherwise it can lead to other potential problems such as removal of the code that was actually correct and insertion of faulty code. But providing 100 percent coverage is not possible and we are always left with some bugs and errors that never come to the scene. Branch testing lets you uncover errors which lie in those parts of the program that are least executed or never executed. But there is a potential drawback too; which is; that it is very ineffective in uncovering errors in interactions of structures and decisions. Because of this drawback, the testers usually prefer to go for path testing.

Read more about this in Part 2 - White Box testing techniques.

Some advantages and disadvantages of white box testing

White box testing can be applied at the following levels in software testing process:
- Unit testing
- Integration testing
- System testing

However, commonly it is carried out at the unit level. White box testing at this level is used for exploiting the paths through the program units, and for testing paths between different units, and on the system level for testing paths between the various sub-systems. In this article we discuss about the advantages and disadvantages of using this testing methodology.

Advantages: This testing methodology ranks among the widely used testing methodologies, with an increasing number of people using it, although it does require technical knowledge and capabilities. We have three major advantages from it.
First, in this technique it is beneficial to have knowledge about the source code unlike the other methodologies where the tester should not know much about the source code. This helps in thoroughly testing the application.
Second, this methodology has made it possible to optimize the code by helping in uncovering the hidden errors and removing them.
Third, it provides an opportunity for introspection.
Another plus point is that the testing can be started with just one developed unit at hand. We don’t have to wait for the whole program to be ready with GUI. Majority of the paths are covered in white box testing.

Disadvantages: Just as white box testing has advantages; it has its own minus points too.
The first major disadvantage is that it gets very complex once you start with it. This happens so because every path in the program has to be tested and so all the paths have to be identified which can become very time consuming and difficult indeed. For this the programmer as well as the tester must have a great deal of knowledge in this level of detail.
Second major disadvantage is that white box testing consumes too much time. In every project it is not possible to testing each and every single path. It is certain that some of the paths will go unnoticed. This is why the white box test cases are very complex and can only be implemented with a thorough knowledge of the application and code. Maintenance of the test scripts proves to be a burden as the complexity increases and changes have to be made in the implementation.

This testing methodology requires many testing tools which might not be available at hand instantly. The work of a mechanic is analogous to white box testing i.e., the programmer examines the source code as to see why it is not working.
First the tester or the programmer must analyze the source code with the help of comprehensive software documentation, code and the samples.
Second, the tester needs to think for the ways or methods by which he can disrupt the normal functioning of the application or what are the input factors that can cause the program to go awry? Now based up on these assessments, the white box testing techniques can be implemented. The assessments have to be made carefully for white box testing to be successful. In simple words white box testing is just a means for the verification of the source code.
The logic and the structure of the code must be known to the person who is testing it. Logical decisions are implemented. Each logical decision tests a different path. In white box testing programmers can only substitute for testers i.e., they only can test the application. If other testers are hired, they will take some time in understanding the source code of the program or need a high degree of technical knowledge.

Testing: A brief summary of white box testing

We all know white box testing by many other names as well such as glass box testing, clear box testing, structural testing, transparent box testing etc. This testing methodology is used for testing the internal structure of the software or to be precise to test how the application actually works and whether it works as desired or not. This is just the opposite of black box testing that is responsible for testing the system’s functionality only, not getting into the internal structure. The test cases for the white box testing are designed based up on the programming skills and internal perspective from which the system is viewed. From the various inputs, some are selected for exercising all paths through the program and determining whether they are producing the desired outputs or not.
A number of errors can be discovered using this testing methodology. But it is not capable of detecting the parts of the specification that have not been implemented or are missing. The following testing techniques are included under the white box testing:

- Data flow testing
- Path testing
- Decision coverage
- Control flow testing
- Branch testing
- Statement coverage

This testing methodology is used for testing the source code of the application using the test cases that are derived using the above mentioned techniques. All these techniques are used as guidelines for the creation of an error free environment. Any fragile piece of code in the whole program can be examined using white box testing. These techniques act as the building blocks for white box testing and its essence lies in carefully testing the source code so that no errors occur later on. Using these different testing techniques all the paths through the source code that are visible can be exercised and an error free environment can be created. But the tester should have the ability to know which path of the code is being tested and what the output should be. Now we shall discuss about the various levels in white box testing:

- Unit testing: White box testing done during this phase ensures that code is working as desired before it is integrated with the units that have already been tested. This helps in catching the errors at an early stage so that later it does not becomes difficult to find them when the units are integrated and the system complexity increases.
- Integration testing: Test cases at this level are meant to test how the interface of various units interact with each other i.e., the system’s behavior is tested in an open environment. During such testing any interaction that is not familiar to the programmer comes to light.
- Regression testing: This phase uses the recycled test cases from white box testing.

For the white box testing to be effective the tester should understand the purpose of the source code so that they are tested well. The programmer must understand the application so that he/ she can create effective and appropriate test cases.
The programmer working on preparing the test cases needs to know the application very well in order to ensure that he / she can prepare cases to handle every visible path for testing purposes. With the source code having been understood, this understanding of the course code lets the team use this analysis of the source code for preparing test cases. And for making these test cases, these are the steps that need to happen in white-box testing:
- Input, that may include functional requirements, specifications, source code etc.
- Processing unit, that carries out the risk analysis, prepares a test plan, executes tests and gives results.
- Output, that provides the final result.

What are characteristics of fuzz technique? What are uses of fuzz technique?

Fuzz testing or fuzzing as it is commonly known as, is another type of very popular mixed software testing methodology i.e., it can be either implemented as a black box testing technique or white box testing technique or even as grey box testing technique, though the latter case being very rare. 

- It also provides the options for running it manually or by some semi automated or fully automated processes. 

- Unexpected, invalid or random data is what all is fed as input data values to the test cases created for the fuzz testing of a software system or application. 

- After the arrangement of the test input data values for the test cases, the whole software system or application is subjected to an inspection to determine if at all it is experiencing any problems like crashes, memory leaks or failure of the code assertions that are nothing but built in defects in the software system or application. 

- Fuzz testing or fuzzing is actually a testing methodology that is categorized under the category of the security testing and is often used for security checks of a software system or application. 

Fuzz testing if further divided results in to two sub categories as described below:

1. Generation based fuzz testing: 

This type involves creation of entirely new set of input data based on the model input data and

2. Mutation based fuzz testing: 

This type is concerned with the generation of new set of data by mutating the existing samples.

Fuzz testing lays more emphasis more on the network protocols and the file formats of the software system or application than the other aspects of the system. But this does not hold necessary that only the network protocols and file formats should undergo fuzz testing, other aspects of the input data for the system can also be subjected to the fuzz testing. Common input for the fuzz testing basically includes:

1. Key board events
2. Mouse events
3. Environment variables and
4. API calls sequence.

The unusual input forms are:

Such as those being mentioned below can also be subjected to fuzz testing:

1. Shared memory
2. Contents of the data base of the software system or application and
3. Inter leaving of threads (precise)

The input data which manages to cross the trust boundary of the software system or application catches the attention of the testers more than any other happening during the testing. The origination of the fuzz testing is traced back to the University of Wisconsin and discovered by the professor Barton Miller. 

Uses of Fuzz Testing

The fuzz testing has emerged as a software testing technique with more than one uses:

1. For the testing of large projects having a budget just enough for the development of the test tools it can be employed as black box testing technique.
2. It is a very feasible and affordable software testing technique and has a high benefit- to - cost ratio.
3. The fuzz testing proves helpful in providing a sample stating the behavior of the software system or application generated in a random manner.
4. It very effectively demonstrates the exception handling capability of a piece of the source code of the program without crashing.
5. Fuzz testing gives an assurance for the maintenance of the overall quality of the software system or application rather than just acting as a testing tool for finding bugs.
6. Fuzz testing can also be used as a substitute for the formal methods employed for exhaustive testing of the software system.
7. One can rely on fuzz testing for determining the reliability of the software system as an application of static analysis, code audits or partial rewrites.

What is meant by code coverage? What are different types of code coverage?

Code coverage is one of the important measures that comes across while carrying out the software testing. 

What is Code Coverage?

- Code coverage is the measure that gives the degree up to which the source of a software system or application has been tested. 

- Code coverage earlier was recognized as one of the methods that were developed for carrying out a systematic software testing. 

- The term code coverage was first referenced under the “communications of the ACM” in the year of 1963 by Maloney and miller.

- Code coverage is measured using a specialized tool called code coverage analyzer for instrumenting the binaries in order to add all the tracing calls and run all the automated test cases against the software system or application under question. 

- A good code coverage analyzer will not only give you the degree of the coverage but will also allow you to dig deeper in to the data and check out for the lines of the code that are executed during a particular test. 

- Today we have a large number of code coverage criteria that amount to the total code coverage. 

Types of Code Coverage

- The software system or application that is under testing is adorned with certain options that run under a particular environment. 

- Each and every function of the program that is executed is traced back to the function calls in the source code which allows the easy checking of the rarely accessed, never accessed parts of the program under normal conditions. 

- This gives the software testers an assurance that the most important conditions of the source code have been tested. 

- The outcome is then subjected to analyzation for sorting out of the code that has been analyzed. 

- The process is a combination of several code coverage methods that aim to provide a manageable and rigorous set of regression tests. 

- The output of the code coverage analyzation can be used by the engineers for devising the test cases and configuration sets that in turn will increase the code coverage relating to the vital functions. 

- Mostly two common forms of the code coverage are used by the software testers namely:

1. Line coverage or statement coverage: 

- This type of the code coverage reports regarding the execution foot print of the software testing in terms of the lines of code which are executed for the completion of the test.

- The statement coverage provides you with the benefit of the identification of the bugs that arise in the constructs of the control flow of the source code like consecutive switch labels and compound conditions. 

- The major setback of this code coverage type is that even though it seems like you have achieved 100 percent of the code coverage, there might be 100s of wicked and uncaught bugs.

- This disadvantage of the statement or line coverage is over-comed by a more advance form of code coverage called the branch coverage.

2. Edge or branch coverage: 

This type of code coverage reports regarding the branches or the code decisions that like statements are executed for the completion of the tests.

- Both of the above mentioned types of the code coverage provide the result in the form of code coverage metric in terms of percentages. 

- Out of the two, the branch code coverage is considered to be quite comprehensive that the equivalent percentage of the statement code coverage.

- It is also very easy to determine the number of branches that have to be tested since the Boolean decisions have only two outcomes i.e., either true or false. 

What is the concept of penetration testing tools?

We all are quite familiar with what is a penetration test or a pen test. Every kind of software testing technique makes use of certain tools, so does penetration testing. 

This article is focused up on the tools that are meant for carrying out the penetration testing. Before moving on to the discussion about the tools, let us buck up with some concepts of penetration testing. 

About Penetration Testing

- Penetration testing gives a measure of the security of the software system or application or a computer network. 

- This is done by the simulation of the attacks as from the outside malicious attackers. 

- The attacker can also be an insider. 

- The attackers are classified in to outsiders and insiders on the basis of the approach of their access to the software system or application. 

- The attackers not having any authorized access to the system are called as outsiders and those who have any extent of authorized access to the system are called insiders. 

- The first step in the penetration test is the identification of the potential vulnerabilities of the system by carrying out an active analysis.

- These vulnerabilities are a consequence of the improper configuration of the software system or they may occur also because of flaws in the hardware and software components of the system. 

- Some of the technical counter measures may also revoke these vulnerabilities.

- The penetration is performed in the way that a potential attacker might follow to attack the system. 

- After the identification of these vulnerabilities, these are brought to the notice of the owner of the system. 

- These potential vulnerabilities are then coupled with a proper assessment of their potential impacts on the system as well as organization using several effective penetration tests. 

- Some technical counter measures are then designed to reduce their impact on the system. 

There are several reasons that make the penetration testing way more valuable. Now coming to the discussion regarding the penetration testing tools, since there are many ways in which the penetration testing can be carried out, there are several types of tools that can be employed for the penetration testing.

Approach used in Penetration testing

- Depending up on the amount of knowledge the tester has about the software system or application, either the black box approach or the white box approach is followed. 

- If the tester has less knowledge of the system, he/ she is likely to follow the black box approach.

- On the other hand if he/ she has ample amount of knowledge then the white box approach is used. 

- Accordingly the tools are chosen i.e., black box testing tools for black box approach and similarly white box testing tools for the white box approach. 

- It is required that the location and the extent of the system to be tested is determined properly before starting the testing. - For the white box approach the tester needs to know about the critical aspects like the IP address of the system and source code. 

- If the amount of knowledge is intermediate between the amounts required for the black box and white box approaches, then the grey box testing approach is allowed. 

- This involves the intermixing of the white box and black box testing techniques. 

- Both the white box testing tools as well as black box testing tools can be employed here. 

- All these three approaches have their own merits and demerits which are often debated.

- These tools are deployed for the creation of the hostile environment for the testing of environment.

Types of Penetration Testing Tools

1. Port Scanners

2. Vulnerability Scanners

3. Application Scanners

4. Web Application Assessment Proxy

What is meant by penetration testing?

The term “penetration testing” is not so rare and perhaps many of us familiar with this type of testing. In this piece of writing we have discussed the penetration testing in more detail. 

About Penetration Testing

- Penetration testing is another testing methodology adopted for testing the security of a computer network or system against the malicious attacks. 

- Penetration testing evaluates the security level of the computer network by bombarding the network with false simulated attacks as malicious attacks from outside as well as inside attackers.

- The outside attackers do not hold any authorized access to the computer system or network but the inside attackers do have than access but only to a certain level. 

- The whole process of the penetration is based on an active analysis.

- This active analysis assesses all the potential vulnerabilities of the computer network or system that are merely a result of its poor security level as well as configuration level.

- Apart from this, the known and unknown flaws form both the hardware as well as software system contribute to these vulnerabilities rather than only operational weaknesses. 

- This active analysis is carried out from the view point of a malicious attacker and is all about the active exploitation of the recognized vulnerabilities. 

Steps in Penetration Testing

- First step in the penetration testing is always the identification of the vulnerabilities. 

- The identified issues and vulnerabilities are then brought to the notice of the whole development team.

- A number of penetration tests are then carried out on that particular system along with the coupling of the information with the active assessment of the risks associated with the computer system or network. 

- A whole lot of effective procedures are designed to reduce the affect of these vulnerabilities. 

Advantages of Penetration Testing

There are several other reasons why the penetration testing holds good:

1. It is effective for the determination of the feasibility of the similar vectors of attack.
2. Helps in the identification of the vulnerabilities which possess a very high risk when the exploitation of a combination of low level risks is done following a particular sequence.
3. Proves quite effective in the determination of the vulnerabilities that cannot be detected with the help of application vulnerability scanning software or automated testing processes.
4. Assists in the assessment of the measure of the operational and business impacts of the attacks on the computer network or system.
5. Successfully tests the effectiveness of the network defenders in detecting and responding to the attacks.
6. Provides the evidences in support of the investments that need to be made in the security field of the computer system or network.

Penetration testing has been recognized as an important component of the security audits. The penetration testing can be carried either way round i.e., either through the black box testing route or through the white box testing route. 

The path to be taken is decided by the amount of knowledge that the tester has about the system or network under testing. If the tester has got a brief knowledge then the black box testing route is followed else the white box testing techniques are preferred. 

Another thing to be determined before starting the testing is the location of the system that has to be tested and also its extent. 

Penetration testing if carried through white box testing

For following the white box approach to penetration testing, the testers needs:

- to have the full knowledge of the system infrastructure,

- to have the full knowledge of the source code, 

- to have the full knowledge of the IP address and

- to have the full knowledge of the network diagrams etc. 

In some cases the grey box approach to penetration testing can also be followed based on how much information is available. Black box approach is useful for simulating an outsider attack whereas the white box approach can simulate and insider attack.  

What are automated unit tests?

Automated unit tests can be thought of as the basic tools with the help of which the unit testing is carried out on the individual modules of the software systems and applications.

Without these automated unit tests, one cannot think of carrying out unit testing since it takes up too much of time. These automated unit tests test the whole unit or the module along with the associated below mentioned things:

1. Its control data
2. Usage procedure
3. Operating procedures and so on.

Since the units or the modules form the smallest tokens of any software system or application, hence the unit testing forms the first step of any software testing life cycle.

What are automated unit tests?

- In procedural programming, these automated unit tests usually cover up the individual procedures or functions.

- In object oriented programming an automated unit test may cover an entire interface for example a class, individual method and so on.

- The automated unit tests are usually developed by the developers, programmers sometimes but more commonly they are created by the white box testers because they have so much of knowledge about that particular software system or application under test.

- They are created simultaneously with the software development process.

- All of the automated unit tests created for a software system or application are quite different from one another and they are assisted by a variety of separate elements like those mentioned below:

1. Method stubs
2. Mock objects
3. Fake objects
4. Test harnesses and so on.

- The above mentioned elements help the automated unit tests in testing a module in isolation.

- The automated unit tests though created by the white box testers, are executed by the software developers since testing this way ensures that all the specifications and the requirements of the software system or the application are fulfilled and the program executes in the desired direction.

- The concept of the automated unit tests came in to existence to formalize the implementation of the automated unit testing as a part of the build automation which would have been a hectic task if carried out manually.

- The automated unit tests are aimed at isolating each and every part of the program and check whether they are working properly or not.

- An automated test being strict provides a written contract that is to be satisfied by the piece of code for which it is meant.

Benefits of Automated Unit Tests

In simple words, we can say that it affords many simple benefits!

- Automated unit tests have been known for years to find problems earlier in a software development life cycle.

- Most of the agile software development methodologies like the scrum and the extreme programming (XP) involve the creation of these automated unit tests before the production of the actual code.

- Later the code is written and is run through the test.

- If it passes, then the code is declared to be error free.

- With these automated unit tests, the program code can still be refactored and checked again with the same automated unit test till the code is considered to be complete.

- These automated test cases facilitate the quick checking of all the functions and modules for quick identification and fixation of the problems if any encountered.

- These automated unit tests have a characteristic which is that they reflect the way in which the code is intended to execute.

- Depending up on the test coverage of these unit tests and development practices in use, the automated unit tests can be made more accurate.

- Apart from all this, the automated unit tests do a better job when it comes to the reduction of the uncertainty in the program.

What is the entry and exit criterion for unit testing?

First let us brief up ourselves with what the concepts of the unit testing and then we shall proceed further in discussing about the entry and exit criterion for unit testing.

About Unit Testing

- Unit testing is a self explanatory term.

- It involves the testing of the smallest units or modules of the software system or application in order to determine whether or not they are working properly and in the desired manner.

- Since this testing methodology is employed for testing the smallest individual unit of the software system or application, hence it got the name “unit testing”.

- If the program or application has been developed using procedural programming oriented language then the entire module is treated as units in the unit testing or else in general, the individual procedures and functions are the units.

- For the applications and softwares in which objected oriented programming has been implemented, the entire application’s interface.

- The test cases for the unit testing are created by the white box testers since they have an in depth knowledge of the software system or application.

- Sine the units are tested in isolation with the other units and independently, hence the test cases created for unit testing are also independent in nature.

Unit Test Case

- A unit test case can be assisted in unit testing by using substitutes like those mentioned below:
1. Mock objects
2. Methods stubs
3. Test harnesses
4. Fakes and so on.

- The test cases for the unit testing are both written and executed by the software developers.

- This ensures that the source code is at par with the design and architecture of the software system or application and works as accordingly as specified.

- The test cases can be implemented in two ways i.e., manually or through the use of automation tools as a part of build automation.

Entry Criterion for Unit Testing

- The functional specifications requirements of the software system or application under test need to be frozen.

- The technical design specifications need to complete and approved. They should have been released.

- The system specifications document also should be complete, approved and released.

- No issue should be pending in the query issue register regarding the requirements under the unit testing.

What else unit testing involves?

- Apart from just testing the modules, the unit testing involves the verification of the requirement specifications in regard with the finalized design and functional specifications requirements.

- For harnessing the maximum benefits of the unit testing, all of the requirements specifications should be signed off.

- Unit testing is said to be complete when the source code is complete and that too according to the specifications.

- All the design specifications should meet the design standards and the entire unused variable, code files should have been removed from them.

- The code documentation including the commenting and AOT documentation must also be complete and approved.

- To put it simple the code should be in such a state so as to be released to the customers or the clients.

Exit Criterion for Unit Testing

- All the units have successfully passed the unit test.

- The code is complete according to the requirements.

- No elements and features are missing.

- The possible errors and warnings have been resolved. This criterion is optional since most of the times it’s not possible to remove all of the errors.

- Code optimization for all the three tiers has been done.

- There is no error in the performance optimization.

- All label files have been created.

- All unused files have been removed.

- Redundant code is removed.

Multiple Condition Coverage is a white box testing technique - Explain?

There are so many types of the code coverage and the multiple condition coverage is also one of them and that is what has been discussed all about in this article.

What is code Coverage?

Code coverage as we all know is a process that aims at the following purposes:

- Creation of the additional test cases for increasing the coverage.
- Looking up for such areas of the software system or application code that have not been exercised by any of the test cases.
- Determination of the quantitative measure of the extent of the code coverage which can be thought of as an indirect measure of the software quality.
- Identification of the redundant test cases which does not affect the code coverage.

The whole process is a time consuming one and thus is helped by an automating tool called the code coverage analyzer.

What is Multiple Condition Coverage?

- Multiple condition coverage is a modification of the condition coverage type of the code coverage.

- As the term “multiple condition coverage” itself suggests that it implements almost all the possible combinations of the conditions in order to check whether those possible combinations do occur in the execution of the test cases or not.

- A large number of the test cases are required for carrying out a full multiple condition coverage of a decision and these are supplied by the truth tables of the logical operators obtained for that particular decision.

- The multiple condition coverage offers an added advantage in the languages such as C, C++, Java and so on which have short circuit operators which is that it helps in implementing a thorough testing.

- It in turn makes the multiple condition coverage in such languages very similar to the condition coverage.

Disadvantage of Multiple Condition Coverage

But, one should not think that the multiple condition coverage is void of any disadvantages.

- It does have one which is that the deciding up on the minimum number of test cases required for the coverage proves to be very difficult job.

- The problem worsens as the complexity of the boolean expressions implemented in the decisions increases.

- Another disadvantage adds to the problem of the test case designer which is that the minimum number of the test cases required for the coverage varies substantially from condition to condition having the similar level of complexity.

Lets take an example:

For example, in a decision, first condition may require 5 test cases while the second one may require 11 test cases and both the conditions are having the same number of operators and operands.

In contrast to the condition coverage that includes the decision coverage also, the multiple condition coverage does not include any decision coverage.

Now what about the languages like Pascal and visual basic that do not have short circuit operators? For such languages the multiple condition coverage is implemented in the form of the path coverage for logical expressions.

One point to be noted here is that the advantages and disadvantages remain the same in the path coverage form also since in the path coverage the number of each additional logical operator is doubled the number of the required test cases.

Multiple condition coverage takes in to consideration two or more conditions and then makes them in to a group and then they are tested. "Since the multiple condition coverage requires a very detailed knowledge of the software system or application’s code in order to understand all the decision aiming statements in that and in turn decide for the minimum requirement of the test cases, therefore it has been categorized under the white box testing techniques."

Loop testing is a white box testing technique - Explain?

Loop testing is also one of the white box testing techniques and thus requires a very deep knowledge about the software system or application. Loop testing methodology has been designed exclusively for the checking of the validation of the iterative constructs which are nothing but the loops.

Types of Loop Constructs
These loop constructs are 4 types as mentioned below:
1. Unstructured loops
2. Simple loops
3. Nested loops and
4. Concatenated loops

Tests applied to different Loop Constructs
Now we shall define some of the tests that can be applied to the above mentioned types of loop constructs under the context of the loop testing:

1. For unstructured loops only one thing is possible which is that they should be redesigned in order to form a structured construct and then can be tested accordingly.

2. For simple loops a number of allowable passes through them is specified first and then the following tests are applied:

(a) Skipping of the entire loop.
(b) Making only one pass through the loop.
(c) Making two passes through the loop.
(d) Making “p” passes through the loop where p is the maximum number of passes.
(e) Making “n-1”, “n”, “n+1” passes through the loop.

3. For nested loops simply the testing approach of the simple loops is extended but, the number of the test cases increases geometrically as per the number of the nested loops and the level of nesting. Usually the following steps are followed:

(a) The inner most loop is the starting point for the testing.
(b) All other loops are set to minimum possible values.
(c) Simple loop tests are conducted for the inner most loop and the outer loops or the nesting loops are kept in their minimum values only till the testing of the inner most loop is complete.
(d) For the excluded values more tests are added.
(e) Now once the testing of the inner most loop is complete, this loop including all the other nested loops are set to typical values and the testing moves outwards. The other nesting loops are held with their minimum values.
(f) The testing in this manner continues until and unless all the loops have been tested.

4. For concatenated loops also the approach that has been defined for the testing of the simple loops can be used but only if the either loops are independent of each other i.e., if the loop counter for one of concatenated loop is 1 and it is used as the executing value for the other loop, then the two loops are said to be dependent on each other and hence the simple loop approach cannot be followed for them.

More about Loop Testing

- It has been observed so many times that most of the semantic bugs preside over the loops.

- It becomes difficult for the path testing also to commence since there are so many paths generated via a loop and an infected loop leads to infected paths which makes even further difficult to track the bug.

- Some of testers believe that it is just enough to test the loop only two times but this is not a good practice.

- A loop should be tested at the following three instances:
a) At the entry of the loop
b) During the execution of the loop and
c) At the exit of the loop

- Loop testing is aimed at testing a resource multiple numbers of times by executing it under a loop and this whole process is controlled by a diagnostic controller.

- However, one rule has been defined for the loop testing which is that the user can interact only at the entry and exit of the loop and nowhere in between.

Data flow testing is a white box testing technique - Explain?

A program is said to be in active state whenever there is some data flow in the program. Without having the data flowing around the whole program, it would not have been possible for the software systems or application to do any thing.

Hence, we conclude that data flow is an extremely important aspect of any program since it is what that keeps a program going on. This data flow also needs to be tested like any other aspect of the software system or application and therefore, this whole article is dedicated to the cause of the data flow testing.

What is Data Flow Testing?

- Data flow testing technique has been categorized under the white box testing techniques since the tester needs to have an in depth knowledge of the whole software system or application.

- Data flow testing cannot be carried out without a control flow graph since without that graph the data flow testing won’t be able to explore any of the unreasonable or unexpected things i.e., anomalies that can influence the data of the software system or application.

- Taking these anomalies in to consideration, it helps in defining the strategies for the selection of the test paths that play a great role in filling up the gaps between the branch testing or statement testing and the complete path testing.

- Data flow testing implements a whole lot of testing strategies chosen in the above mentioned way for exploring the events regarding the use of the data that occurs in a sequential way.

- It is a way determining that whether or not every data object has been initialized before it used and whether or not all the data objects are used at least once during the execution of the program.

Classification of Data types
The data objects have been classified in to various types based up on their use:

- Defined, created and initialized data objects denoted by d.
- Killed, undefined and released data objects denoted by k.
- Used data objects in predicates, calculations etc, denoted by u.

Critical Elements for Data Flow Testing

- The critical elements for the data flow testing are the arrays and the pointers.

- These elements should not be under estimated since they may fail to include some DU pairs and also they should not be over estimated since then unfeasible test obligations might be introduced.

- The under estimation is preferable over the over estimation since over estimation is causes more expense to the organization.

- Data flow testing is also aimed at distinguishing between the important and not so important paths.

- During the data flow testing many a times pragmatic compromises are needed to make since there exist so many unpredictable properties and exponential blow up of the paths.

Anomaly Detection under Data Flow Testing

There are various types of anomaly detection that are carried under the data flow testing:

1. Static anomaly detection
This analysis is carried out on the source code of the software program without the actual execution.

2. Dynamic anomaly detection
This is just the opposite of the static testing i.e., it is carried out on a running program.

3. Anomaly detection via compilers
Such detection are possible due to the static analysis. Certain compilers like the optimizing compilers can even detect the dead variables. The static analysis itself is incapable of detecting the dead variables since they are unreachable and thus unsolvable in the general case.

Other factors:
There are several other factors that play a great role in the data flow testing and they are:
1. Data flow modelling based on control flow graph
2. Simple path segments
3. Loop free path segments
4. DU path segments
5. Def – use associations
6. Definition clear paths
7. Data flow testing strategies

Basis Path Testing is a white box testing technique - Explain?

As we all know in a typical software system or application, there exist so many possible paths between the entry and exit of the paths and the number of such potential paths is doubled by every decision that comes in the way.

On top of this every case statements adds to the number of the potential paths by multiplying the number of cases with the number of potential paths and this is further incremented by the loops which multiply all the different possible iteration values with the number of potential paths.

So you see with so many paths all over, a single unit of the code makes it extremely difficult to provide a 100 percent coverage to that unit. This can be illustrated by the following example:

- Consider a software program consisting a straight line code having only a single loop that is to be executed 50 times.
- For such a code we can easily make out that it will have 50 paths.
- Like all the other processes, the testing also does not have any unlimited amount of resources.
- There are a limited number of people to perform testing, limited resources, time, efforts and equipment to make up to the successful testing in a stipulated period of time.
- The testing processes always face a challenge of covering all the possible paths without going out of the specified limits.
- But this seems like very difficult to be achieved!

"This all can be possibly achieved if all the redundancy is eliminated from the source code to the maximum extent that is possible and at the same time, adequate coverage must be maintained."

How was basis path testing developed?

- Basis path testing was developed as a hybrid between the two testings namely the path testing and the branch testing.

- The path testing is the testing methodology that has been defined to execute paths all over a computer program whether all or selected and the branch testing is the testing methodology that has been defined for the execution of all the outcomes of every decision in a software source code.

- It can be thought of as a testing methodology that tests every branch and also all the independent paths throughout the software program.

- A path can be a sequence of statements or instructions to the system that starts at one junction and ends at the other and in between it may encounter several other junctions.

- An independent path is a bit different from the normal path in the way that it introduces a new set of processing statements or conditions.

What steps are involved in basis path testing?

It is a path that takes in to account only the paths with the linear combination of the iterations and in no other way. The following steps are involved in the basis path testing:

1. Drawing out of the control flow graph
2. Calculation of the cyclomatic complexity
3. A basis set of paths is chosen
4. Generation of the test cases for the execution of each and every path in the code

Benefits of Basis Path Testing

1. It clearly defines the number of the independent paths and the number of the test cases required.
2. It executes every statement at least once.
3. It executes every condition for both outcomes i.e., true and false at least once.
4. It pays attention to the logic of the program.
5. It facilitates the arbitrary test case design in contrast to the analytical test case design.

Basis path testing is a white box testing technique since one requires an in-depth knowledge of the program.

Explain the concepts of fuzz test technique?

WHAT IS FUZZ TESTING?

- Fuzz testing is the formal name for fuzzing which as we know is another software testing technique that involves playing with the software system or application using all types of possible invalid input test data.

- By the invalid input test data we mean it can be either unexpected data or random data i.e., any other type of test data other than the specified input data type.

- It is not necessary that the fuzz testing should always be automated; it can also be semi automatic though keeping the whole process fully automated consumes less time and effort.

- Semi automatic process is used only when there is manual interference required in the testing.

- After feeding the invalid input data to the software system or application, the behavior of the system is monitored for any exceptions like hanging, crashing or failing of the code assertions that are built in and also check for any memory leaks.

From the above discussions we can easily make out that the fuzz testing has been developed for testing of the security related issues of the software system or application. Till now two types of fuzz testing have been identified namely:

1. Mutation based testing
2. Generation based testing

CONCEPTS OF FUZZ TESTING

- Fuzz testing was developed to complement the negative testing and syntax testing.

- Both of the above types of the fuzz testing can be either employed as white box testing technique or black box testing technique or as a mix of the two techniques i.e., grey box testing.

- Whatever technique may be used, it is focussed up on the common target i.e., network protocols, file formats and so on.

- Though these two are the usually targeted elements, almost any type of input test data for the program can be subjected to the fuzz testing.

- The most common types of input data include:
1. Sequence of API calls
2. Environment variables
3. Mouse events
4. Keyboard events etc.

- Apart from just testing the input test data types, even some elements like shared memory, contents of a data base, interleaving of threads etc can also be tested by the fuzz test.

- But, usually the input that is able to cross the trust boundary of the security of the software system or application is targeted.

- Among all other approaches to fuzz testing, the one which is mostly preferred is the black box approach.

- This approach is mostly employed for the testing of large software projects having a budget for the development of the testing tools.

- Fuzz testing is included in the class of the software testing techniques that offer a high benefit – to – cost ratio.

- Like other software testing techniques, fuzz testing also has a drawback which is that it is able to develop only a sample of the behavior of the software system.

- In some rare cases, if the software passes the fuzz test, it merely indicates that only a part of the software system can effectively handle the unexpected input data types without any problem.

- This tells us that fuzz testing can be considered as an overall quality assurance factor and not merely just a bug finding tool.

- Fuzz testing is also not to be taken as a substitute neither for the formal methods nor for the exhaustive testing.

- It also gives us a gross measurement of the reliability of the software.

- Based on the results of the fuzz testing, it can be decided that which part needs a partial rewrite, static analysis or code audits.

What are different methods and techniques used for security testing at white box level?

It requires a great deal of efforts to harness a good level of security. To obtain good security statistics one has to follow a proper approach to the testing. Like for any other kind of software testing one need to decide for security system also that who will carry out the testing and what approach has to be followed. Carrying out the security testing at the white box level is not at all easy as it is very complex and detailed.

APPROACHES FOR SECURITY TESTING AT WHITE BOX LEVEL
Basically till now two basic approaches have been identified for the security testing at the white box level and these have been mentioned below:

1. Functional Security Testing
- This approach to testing is usually followed by the standard testing organizations. - It deals with the checking of the features and functionalities of the software system or application for determining that whether or not they are working as stated. - This sounds like a very classic approach to security testing.

2. Risk Based Security Testing
- This is a more traditional approach to security testing and is followed usually by the quality assurance staff.
- This approach is quite difficult as compared to the previous mentioned approach.
- The main problem here is of the expertise of the testers since this approach calls for great skills in testing.
- Firstly to design the security tests which can completely exploit the vulnerabilities are difficult to be designed since for this it is required that the tester thinks like an attacker.
- Secondly, the security tests do not exploit the security of the software system or application directly and this causes a problem to observe the outcomes of a security test.

ABOUT SECURITY TESTING AT WHITE BOX TESTING LEVEL

1. A security test carried out without much precaution and logic can cause the whole security testing go wrong and this in turn can lead the software tester to carry out even more complicated test processes to counteract such a situation.

2. Risk based testing requires more skills than experience.

3. Most of the security testing methodologies or techniques that we use at the white box level are traditional and some of them have become out dated.

4. On the other hand the security exploitation techniques used by the attackers have become sophisticated day by day and the traditional methods used to cope these issues are becoming extinct.

5. Security testing at both the black box level and white box level tend to have a better understanding of the software system or application but different approaches are followed at both the levels.

6. The different approach followed by them is decided on the basis of the access of the source code i.e., whether or not the tester is having access to source code.

7. Security testing at the white box level is concerned with the rigorous analyzation of the source code of the software program as well its design.

8. It basically deals with finding the errors in the security mechanism of the software system.

9. In very rare cases it happens that this approach involves the matching of the patterns and automation of the whole testing process by implementing a static analyzer.

10. One peculiar drawback has been discovered for this kind of testing which is that this kind of testing sometimes may report a bug in some part of the software but actually there exists no such bug.

11. But still security testing at white box level using static analysis methods and techniques proves good for some software systems and applications.

12. Risk based testing calls for a lot of understanding of the whole software system.

13. After all, the product security is very much essential to the reputation of the company.

What are Application Testing Methodologies?

First of all, lets be clear with what is application testing actually.
- It is simply the testing of application software. But, it is not so easy to carry out as it sounds like.
- To develop good application software, great efforts and skills are required both of development and testing.
- Testing is needed to check the quality status of application software.
- This is indeed very important for quality assurance and to see that if the application software is meeting the expectations of the consumers or not.

WHY IS TESTING METHODOLOGY IMPORTANT?
- It is obvious that all the aspects of application software cannot be discovered by following just one testing methodology.
- One has to employ many testing methodologies in order to discover most of the hidden bugs and errors.
- Many methodologies have been developed for testing application software.
- Discovery of flaws is the primary aim of any software testing methodology.
- Criticism is yet another aim.

APPLICATION TESTING METHODOLOGIES

1.BOX TESTING TECHNIQUES
- White Box Testing
It includes techniques that are used to test the program or algorithmic structures and working of that particular software application in opposition to its functionalitY or the results of its black box tests.
a) API testing
b) Fault injection
c) Code coverage: Code coverage can be defined as a measure to measure the extent to which the source code of a software system has been tested.
d) Mutation testing
e) Static testing

- Black Box Testing
a) Equivalence partitioning
b) Boundary value analysis
c) Pair wise testing
d) Fuzz testing
e) Exploratory testing
f) Model based testing
g) Specification based testing

- Grey Box Testing
As the grey colour is made from the combination black and white colours, so does grey box testing is made from a combination of both white box testing as well as black box testing.
- Visual Testing
As the name suggests, non destructive testing techniques do not involve vigorous checking of the software structure.
- Unit Testing
- Integration Testing
The units or modules are combined and tested.
- System Testing
- System Integration Testing

- Regression Testing
It basically discovers and unhide the hidden and new errors and flaws.

- Acceptance Testing
There should be some kind of testing that looks in to the contract and verifies whether or not all the requirements have been met. Acceptance testing serves the purpose right. Acceptance is a composition of 3 kinds of tests namely Physical tests, Chemical tests and Performance tests.

- Alpha Testing
The purpose of checking the application software before the release is served by alpha testing on the basis of:
a) Service level agreement or SLA as it is abbreviated.
b)Requirements
c)Specifications
d)Defect rate efficiency (known as DRE in short form).

- Beta Testing
Beta testing is carried out after the successful completion of the alpha testing.
- Performance Testing

- Usability Testing
Usability testing can be defined as a technique which is used in interaction design. This designing is centred around the user and accounts for the evaluation of the software system, application or product by testing it out on the software product users.

- Security Testing
Security testing as its name suggests can be defined as a process to determine that whether or not a software or information system or application is capable of protecting data and keeping it secure.

- Internationalization
Internationalization can be defined as a process of coding and designing a product. This coding is done in such a way that it can perform well almost on any platform after modification for use in different regional standards and languages.
- Localization

What is the difference between the dynamic testing and static testing?

Dynamic testing is also called dynamic analysis. It has always been a part of software testing and is basically used to explain the dynamic behavior of a software application or a program.

- Dynamic testing can now be defined as the testing the response or reaction (physical in nature) of the system to the dynamic variables ( the variables which keep changing with time and are not constant) that have been used in the program.

- To carry out a dynamic test for any program, one has to compile the software, run it and work with it.

- Working with the software involves inputting data values to the variables.

- It also involves checking the output if whether or not it is up to the expectations of the programmer.

- The actual program output is checked against the desired program output. Input and output is checked for validation of the software.

Many methodologies have been developed for dynamic testing. Few have been listed below:

- Unit tests
- System tests
- Integration tests and
- Acceptance test

Idea of dynamic testing is typically based on testing the software during execution of the program and also during its compilation.
- Dynamic testing is totally opposite of static testing.

- The software application must be actually compiled and executed and tested as the dynamic testing is a part of validation process, which is again a part of verification and validation process.

- There are many methodologies that can be used for testing a program dynamically. If you are not comfortable with one technique, you can for the other one.

- Dynamic testing includes functional test techniques which are commonly known as black box techniques.

- They help in designing test cases which are based on the functions of the software application under test and there is no need to consider the details of the software structure.

- These techniques are used to check for input and expected output.

White box testing techniques or structural test techniques are used to check the structural design of the software application for flaws and mistakes.

Static testing can be compared in great contrast to the dynamic testing.

- Static testing can be defined as a form of software testing in which the software is not actually used.

- It is not as detailed as the dynamic testing and it is carried out to determine the sanity of the code, document or algorithm.

- It can be primarily stated as the syntax checking of the program code and manual review of the document or the source code to find the errors and bugs.

- Static testing usually used by the software developers who wrote the code in isolation and techniques such as inspections, reviews and walk through are used.

- If we look at the static testing from the point of black box testing, it involves review of specifications and requirements of the software system or application.

- Static testing forms the verification part of verification and validation process.

- Like dynamic testing, static testing can also be automated.

- Static testing involves programs to be analyzed by a compiler or an interpreter
that’s asserts the syntactic validity of the programs.

- Bugs and errors discovered during the static development are easier and less expensive to fix than later in the development cycle.

- The people who are involved in static testing are basically software developers and testers.

- During static testing you need to have a check list to ensure whether the software system or the application is keeping with the standards of the organization or not?

- These standards are basically for coding, deployment and integrating.

What are different types of white box testing? Part 2

White-box testing or clear box testing, transparent box testing, glass box testing, structural testing as it is also known can be defined as a method for testing software applications or programs. White box testing includes techniques that are used to test the program or algorithmic structures and working of that particular software application in opposition to its functionalities or the results of its black box tests.

White-box testing can be defined as a methodology to verify the source code of the software system if it works as expected or not. White box testing is a synonym for structural testing.

Unit testing and Integration testing is already discussed in previous post.

Types of White Box Testing

- Function level testing:
This white box testing is carried to check the flow of control of the program. Adequate test cases are designed to check the control flow and coverage. During functionality level white box testing simple input values can be used.

- Acceptance level testing:
This type of white box testing is performed to determine whether all he specifications of a software system have been fulfilled or not. It involves various kinds of other tests like physical tests, chemical tests and performance tests.

- Regression level testing:
This type of white box testing can also be called as retesting. It is done after all the modifications have been done to the software and hardware units. Regression level white box testing ensures that the modifications have not altered the working of the software and has not given rise to more bugs and errors.

- Beta level testing:
Beta testing is that phase of software testing in which a selected audience tries out the finished software application or the product. It is also called pre- release testing.