Test Driven Development - Some benefits

According to a study, test driven development involves writing more tests and thus makes programmers more productive. The hypotheses produced from the study were inconclusive regarding code quality and relation between productivity and TDD. Programmers that use the pure test driven development on projects said that they rarely felt like invoking the debugger. Version control system and TDD when used together, makes it easy to revert the code when it fails unexpectedly to the previous form that passed all the tests. Thus debugging here proves to be less productive than reverting.
TDD has more to offer other than just correctness validation. It can also be used for driving the designing process. Since the focus is shifted to the test cases, the programmer can see how the client will be using the functionality. As a result of this programmer’s concern with interface increases. This benefit of TDD is just the opposite of design by contract. It is so because we approach through test cases rather than using preconceptions and mathematical assertions.
With TDD you get the ability to proceed with programming in small steps with as much as you are comfortable with. Thus you are able to concentrate at the present task where your goal is to make the code pass the test. Initially in TDD, we are concerned with error and exception handling. There is a separate process for creation and implementation of these extraneous situations. This way it is ensured that at least one test covers each part of the code. The programmer’s and users’ confidence in the software is boosted.
Even though it is obvious that programming with TDD requires more code than with other techniques, according to a model by Müller and Padberg, implementation time in TDD is shorter. The defects are caught early because of the frequent testing in the development cycle. The errors are thus prevented from turning into expensive and endemic bugs. It also reduces the time period of debugging phase. The code produced with test driven development is more extensible, flexible and modular. This is by virtue of TDD’s methodology of forcing the programmers think in the terms of smaller code units. The benefit of all this is that you  get a cleaner, focused but loosely coupled code.
The modularization of the code to some extent is also affected by the design pattern of the mock objects used. This is so because the pattern requires writing the modules so as to make it easy to switch between versions for deployment and testing reasons. Every code path is covered by the automated tests unless no more code is required for the code to pass the test. For example, if the programmer wants to add an else branch to an if statement, then his first requirement is to write a code that causes that branching. That is why all the tests produced in TDD are quite thorough and even the most unexpected of the changes is detected.
It has been proved experimentally that TDD approach is superior to the tradition test – last approach when it comes to lower CBO (coupling between objects). The experiment also showed that TDD results in better modularization, easier testing and reuse of the already developed products. The effect of the TDD approach on generation of unit tests was measured using the MSI (mutation score indicator) and the BC (branch coverage). These indicate the effectiveness and thoroughness of the unit tests. A medium effect is represented by a mean effect size based up on the meta – analysis of the experiment that was performed. For branch coverage the effect size was medium and therefore taken to be substantive. 

Some white box testing techniques – part 2

This post is a continuation of the discussion on white box testing techniques from part 1 of this article (link).

- Path testing: This is a typical application program that consists of 100s of paths between two points of the program i.e., the entry and the exit. The number of paths goes on multiplying by the number of decisions that are encountered during the execution. This number is again increased by the execution of the loops. Each of these paths has to be tested to ensure there are no bug. This seems quite easy for a straight line having 50 – 100 lines of code but what about a large program? Counting gets difficult. With a limited number of resources (and most software projects have a limited number of resources), all the possible paths have to be covered. Path testing can be made a little easy by not consider the redundant code in the program as far as possible. Path testing is responsible for checking whether or not each and every path is getting executed at least once. Today we also have a new type of path testing, called the basis path testing which is a mixture of branch testing and path testing. Even the path testing makes use of the control flow graph. It involves calculating the cyclomatic complexity. For each & every path, a test case has to be designed.

- Statement coverage: We also know this white box testing technique by other names such as segment coverage or line coverage. This coverage methodology provides coverage only to the conditions that are true. The false conditions are neglected. This helps in the identification of the statements that are often executed in the program. It also helps in finding out the areas of the program where there is no data flow because of no execution. But all the lines of code are examined and executed. Statement coverage is important for verifying whether the code is serving its desired purpose or not. In other words it gives us a measure of the code quality. We can consider it as a partial form of path testing. Disadvantages include: false conditions are left untested; it gives no report about the loop termination and does not consider the logical operators. One line of code might be a full statement, a partial one or blank or may contain many statements. There are nested statements too. The statement coverage has to be measured first and then the test cases should be designed.

- Decision coverage: Another name for decision coverage is “all – edges coverage”. Here all the conditions that result because of a logical decision are considered. It does not matter whether the conditions are true or false. This is just the opposite of statement coverage. The outcome of a decision is nothing but a branch. So the branch coverage gives a measure of extent up to which the decisions in a program have been tested. This is why most of us often get confused with branch coverage and decision coverage and use them interchangeably. But it is more than statement coverage. The decision statements include loop control statements, if statements, if – else statements, switch – case statements i.e., the statements that result in either a true or a false value based up on the logical condition in that statement. This white box testing technique is helpful in validation of branch coverage that is whether or not all branches have been covered. It helps in ensuring that no branch gives an abnormal result that may disturb the operation of the whole program. There are certain drawbacks of the statement coverage as we saw above. These drawbacks are eliminated by decision testing. 

Some white box testing techniques – part 1

In this article we shall give a brief description of the various techniques that are used in white box testing.

- Control flow testing: This is more of being similar to structural testing and is based on the control flow model of the application that is being testing. In control flow testing certain paths among all are selected judiciously so as to provide maximum coverage. The paths are chosen so that a decided thoroughness in the testing process is maintained. This means that paths that have been selected should be enough so as to cover all the statements at least once. It is usually applied to the new software systems that have to be put under unit testing (unit level white box testing). This technique assumes that there are correct specifications, correctly assessed and defined data, no bugs in the program except those in the control flow. The number of control flow bugs is fewer in the programs that are written using object – oriented or structured programming languages. This technique makes use of a flow graph usually called the control flow graph that represents the control structure of the program.

- Data flow testing: Flowing data in a program is what keeps a program active. If there is no data flow, the program won’t be able to perform any operation. The program’s data flow needs to be tested to ensure that it is constant and efficient. Data flow testing also requires a data flow graph. This graph is required to know where and why the data is going or if it is going to correct destination or not. This testing helps in uncovering any anomalies that might restrict the flow of data in a program. Knowing these anomalies also helps in branch and path testing. A number of methods and techniques go in to data flow testing for exploring various events that are taking place in the program, whether right or wrong. It is used for checking whether all the data objects have been initialized or not and makes sure that they are used at least once in the whole program. Arrays and pointers are considered to be the two major elements that play a critical role in data flow testing. These two elements cannot be neglected. Data flow testing might include static anomaly detection, dynamic anomaly detection and anomaly detection through compliers.

- Branch testing: As the name suggests, this testing technique is used for testing all the branches of all the loops within a program. Here branch coverage plays a great role. It has to be made sure that each and every branch in the program is executed once. Thus the test cases are designed in such a way that all the branches are covered. This technique is just for complementing the white box testing at unit testing level. The programmers aim that their test cases must provide 100 percent branch coverage. Also the coverage has to be proper otherwise it can lead to other potential problems such as removal of the code that was actually correct and insertion of faulty code. But providing 100 percent coverage is not possible and we are always left with some bugs and errors that never come to the scene. Branch testing lets you uncover errors which lie in those parts of the program that are least executed or never executed. But there is a potential drawback too; which is; that it is very ineffective in uncovering errors in interactions of structures and decisions. Because of this drawback, the testers usually prefer to go for path testing.

Read more about this in Part 2 - White Box testing techniques.

Ensuring that defects have a Change List and Change Lists have reasons / defects added

There are some processes that are pretty important for software development teams to follow in terms of being track changes and the reasons for the same. What kind of changes do you track ? Well, here is an example (and this post is meant as an example, teams should review what are the kind of items they are tracking and why they would want to track these changes). In this post, I talk about ensuring that all code changes have a mention of the reasons why the code was changed, as well as whenever a defect has been marked fixed, then the defect management software would have a note on the code change list. Why would you want to track changes ?
- It provides some discipline for the development engineer. The knowledge that each change needs to have a reason for the same ensures that the person thinks in more detail about the change.
- The reason for the change ensures that somebody else who is reviewing the change can easily pick up on what the changes were. For example, if there is somebody who is doing code review, then it becomes easy to identify the reason for the change and do the due diligence about the impact. When there is a code change due to a defect, then reviewer can read the defect and even determine whether the code change will really fix the defect or not.
- The tester can look at the Change list mentioned in the defect, and use this to determine whether the change list has been made it to the current build or not. This helps save time, else the other way would be to test for the change and then determine whether the defect fixing change has made it to the build or not, which would take more time.
- The person who made the code change may not be with the group or the organization at a later point of time, and somebody who takes over the code would find to easier to determine the reason why code changes were made by reading the notes and thus determining the reason.
- Searching for a specific item in the code base or the defect management software would become much easier. For example, if the team wanted to find out the exact code change that was done for a particular defect, they could easily do so with this information present in the database and then finding the particular section of code.
- During periods when the team is under lockdown and only controlled changes are made, these changes need to be approved by a different set of people, typically managers and/or leads. They would look at the code base and see the Changelists that have been approved, and then review the defects for which these changes have been made, and then review the actual defect before making a decision on whether the changelist was appropriate or not to be approved. Doing these kind of workflows would be much more difficult if the relevant data was not added to the various tracking systems.

Feeling that the team has peaked too early - done with defects earlier than usual

It is an odd situation to be in. You have a product schedule, and in this schedule, you end up with your feature development work at a certain stage, and then reach a stage where the entire activity involves testing of the product. If you define this in more detail, it is the classical waterfall development model. First there is the requirements phase, followed by the design phase where the requirements are converted into design and architecture followed by the actual coding where the design is converted into code by the development team.  This code is then tested by the quality team to detect defects in this code, which follows a process of detecting defects by the testing team, getting them fixed by the development team and re-testing, and so on, till the defects have been detected to a high degree, at which the product is deemed fit to release.
You can have different development methodologies, whether these be Waterfall, iterative or Scrum, or any other. In my experience, as you move towards getting more and more functionality being coded and available to the testing team, there will come a point in the schedule where all the functionality has been built in and the team has the whole product available to do testing. It is hard to visualize a schedule where the product is being built with more and more functionality being built right till the last stages of the release.
As you have more and more parts of the product being available, you need time to do testing of the product with all the pieces connected, all the workflows being available and so on. You can have features being tested part by part as they are being built, but there will always be workflows which are not possible to test until all the pieces have been connected together.
So, in a typical development cycle, you do an estimation of the time needed for the development phase vs. the time needed for testing post development, with a lot of this estimation being done based on historical records and information, and bake this into the schedule. Accordingly, you plan for the phase where the fresh  development activity has to come to and end, and plan for getting this done in your schedule.
Once you reach that stage, it means that your team has completed all the development activity for building in the features needed in the product, and post this stage, you will only have testing and defect fixing. Seems fine, right. Unfortunately, it could be that you have done a much better job at code review, unit testing and other such activities, which resulted in your code being of a better quality than normal. What this results in is an embarrassing situation where you defect timeline is actually projected to come to an end before the due date; and this is where the manager asks the question about the team actually allocating too much time for testing and less time for feature development; since with these statistics, it is pretty clear that you could have done more feature work.
You can resolve this, although it is hard to squeeze in a new feature after you have completed testing; instead, you need to spend time reassuring the team and the manager that they have done a perfect job, that it is not a shame if you ended up completing the work before the actual timeline (although you need to incorporate this entire situation for the post-mortem and for future planning). Further, there are always important features that you were not able to plan for the current cycle, so it would make sense to actually get a head start on such features after branching the code.

Some definitions of regression testing and what it means ..

What is regression testing?

Regression is a type of software testing that is used for discovering new errors and bugs in the software system. In terminology of regression testing these bugs are called regressions. Regression testing is carried out for the already existing areas of the system whether functional or non – functional. This testing is carried out after any change has been made in the configurations or patches. The basic purpose of regression testing is ensuring that any change or modification made to the software does not affect its functionality or introduce some other new errors. It also checks whether a change in one part of the system had any effect on the other parts of the system. Some common methods of carrying out regressing testing are:

- Re – executing the tests that have already been executed.
- Checking if any change has occurred in the working of the software.
- Checking if the errors corrected earlier have appeared.

Regression testing as such is very time consuming since you have to test everything again and again, but can be performed effectively by selecting a certain number of tests. The number of tests selected is sufficient enough to cover the whole unit in which the change is made. According to a research, it has been found that making a change in one part of the software often introduces errors in other parts. In some of the cases the same errors reoccur because the fix that was made earlier to prevent them gets lost. This happens probably because of errors humans make while revising the code i.e., because of the poor revision practices. Often the fix that we apply for a problem is fragile that it is easily breakable and can get lost. Or sometimes this fix for one part of the software can cause an error in another part.
In some other cases it might happen that during redesigning, the same mistakes might be made that were made during the designing of the original software. Therefore mostly regression testing is considered to be a good practice. It helps in locating and fixing the bug, recording the tests which led to the discovery of the bugs, repeated execution of the tests at regular intervals after some modification has been done in the program. You might be thinking that this can be done even by following the manual testing procedure by means of some programming techniques. But this is better carried out with automated testing as it takes less effort and time when compared to the manual testing.
An automated testing suite consists of various software tools that allow for the automatic execution of the test cases and generate reports. The programmers might set up such automated testing systems for automatically doing regression testing at regular intervals of time. This way the regression tests can be run either after every compilation, once a week or even every night. There are various tools available for automated regression testing such as follows:
- TinderBox
- BuildBot
- Hudson
- Jenkins
- Bamboo
- TeamCity

Regression testing is closely associated with extreme programming as the former is an inseparable part of the latter. Here at each phase of the software development cycle, the entire software package is put through repeatable, extensive automated testing. The software quality assurance team is responsible for performing the regression testing after the development is over. But fixing the bugs at this stage comes very expensive. This potential problem is fixed at an earlier stage by means of unit testing. The test cases written by the programmers are for the verification of the intended outcomes i.e., they are either unit tests or functional tests.