Defect Management: Dividing overall defect reports into separate functional areas - Part 3

This is a series of posts where I look at the creation of a defect status report, in a way that it provides enhanced information to the team and its managers and helps them to make decisions. In the previous post (Count of new incoming defects), I talked about adding more parameters to the defect report that help in information that can let the team know whether the number of defects getting added on a daily basis will enable the team to reach their defect milestones by the required time and date. This kind of data, coming in on a regular daily cycle, helps the team to decide whether their current defect pattern is on the desired path, above it, or below it, and accordingly make the required decisions.
This post will see more details added to the defect report that provides a lot of useful information to the team. The last post talked about the flow of incoming defects that move to the ToFix state. However, there is another type of information that is relevant. Defects that are present in the system are not only owned by the development team. In addition to the defects in core code, there may be defects that are present in the components used in the application. These are defects that are not attributable to the development team, but to the vendors or other groups that provide the components. A number of teams that I know typically track these defects in the defect database, but distinct from the defects with the core team.
The nature of defects that are against external components is different from those in the core code. Even though to the customer it does not matter whether the defect is within the core code or in an external component, the amount of effort required in terms of coordination and communication is entirely different from the other defects that are with the core developmental team. If a defect is with a component that is not owned by the team, the timeline for fixing of the defect may take longer and need persuasion; or there may be a lot of back and forth between the tester and the external component team to study the situation in which the defect occurs (which also includes sending the environment in which the defect occurred to the external vendor - and this has its own cost and restrictions, since if the team is working on a new version of the software, there would NDA issues and IP issues related to sending details of the environment to the external component team), and so on. Another concern could be that that even if such a defect is resolved, it might need a new version of the component, which has its own extra cost about testing the component on its own to check whether it is fine or there are other issues with the same.
As a result, it needs to separate out the incoming defects about whether they belong to the core team or whether they are attributable to people outside the team; and if the proportion of such defects that are outside the core team is increasing, it is a matter of concern to the team, since resolving such defects typically takes much more effort and time.

Defect Management: Dividing overall defect reports into separate functional areas - Part 2

Part 1 (Dividing defect reports into ToFix, ToTest and ToDefer counts) of this post talked about the importance of Defect Management in a software project, and then got into some details about the regular sending out of a report on Total defects, with these defects having been broken down into ToFix, ToTest and ToDefer stats, maintained on a graph over a period of time with daily updates, so that the team and the managers can figure out whether the team is on progress to resolve these bugs.
This post continues on this line, talking about additional items that can be added to this defect chart and metrics to provide more information to the team and determine whether it is on the right track or not. Are all these metrics important ? There is a lot of comments about not over-burdening people with too many statistics, and there are more comments about letting people do their work rather than sending so many statistics that they stop looking at these stats. However, it is also true that the role of the team managers is to look at the broader situation in terms of project status, and the defect management is an important part of this. It is true that the team members should not be burdened with these figures, but for the team managers, it is critical to look at such data.
So, the team looks at the ongoing figures for defects in terms of ToFix over a period of days and tries to determine whether the team is on the right track or not. So what else should you be capturing ? Another metric that can now be added to such a report is about the number of defects that are still incoming. There are primarily 2 ways in which defects can be added to the count of developers:
- New defects that are logged against the development team and which add to their count and to the overall ToFix count
- Defects that have been rejected by the testing team after they have been marked fixed by the developer but there is a problem in the fix (this can vary a lot among different teams and even in a team - a developer could be fixing defects with hardly any returns and there could be another developer who is under pressure and many of whose defects are returned because of some problems). So, whether to determine this kind of statistic and calculate metrics for such a case determines of whether the team is seeing such kind of returns for the defect management.
Once you have these kind of defect counts, it helps in determining the current status of defects and see whether the team is on the right track. So, you have a total count of open ToFix defects, and there is a decline in such a count needed to hit the deadlines. However, for getting to such a deadline, you need the number of incoming defects to be also fitting into this strategy. If there are a large number of incoming defects, then the team will not be easily able to determine whether their ToFix defect count is decreasing by the amount they want to hit their targets, and this then needs a change to the strategy to determine whether the team will get there or not.

Defect Management: Dividing overall defect reports into separate functional areas - Part 1

Handling defects is one of the major efforts that plays an integral role in handling a project schedule and making it successful. I have known multiple teams where the team did not have a good running estimate of their defect count and the defect estimation over the remaining period of time left in the schedule; as a result, when the team was closer to the final stages of the schedule, they found that they had too many defects that made the remaining part of the schedule very tight - which meant that if they were to do an accurate reckoning of their status, they would need to either defer more defects and maybe end up with a product that is lower in quality; or the product would need to extend their timeline / schedule, which has a huge implication  for the team and many other teams that are involved in the release schedule of the product.
How do you avoid this ? The first paragraph of this post points out a huge problem, but the answer cannot be handled in a single post; it can be handled by a single cheesy phrase but which does not provide any solutions - "You need to do Defect Management". Now, let us get down to the meat of this post - this post just takes a specific aspect of defect management - sending a split of the defect counts as per the different areas. This in turn provides a better view of the defect picture to the team and helps in the process of overall defect management.
We wanted to have a system whereby we could track the counts for each of the separate functional areas and yet have the management team have access to these data on an ongoing basis. These also helped the separate functional teams do a targeting of the counts of the defects of their respective functional areas and work towards reducing this count. So, we took the overall data for defects for the product (open defects) and split these into the following areas:
Open defects:
ToFix (these are primarily defects owned by the development team, although there could be defects that are carried by other team - such as where there are defects with components supplied by external teams)
ToTest (these are primarily defects owned by the testing team, although since anybody can file a defect within the team, there may be people other than the testing team who own a defect)
ToDefer (the exact terminology of these defects can be different across organizations; but these are typically defects that are with a defect review committee for evaluation. These can be significant defects that need evaluation by the committee before they are to be fixed, or these can be defects that are not worthy of fixing but the team wants the committee to take a final call, and so on).
What is the purpose of sending out separate stats on a regular basis ? These data, when plotted on a regular graph over a period of time provides a large amount of information. The team and the managers, although they are in the thick of things, sometimes need to see such kind of aggregate information to take a good decision. For example, if the team is in the second part of the cycle and close to the timeline, and yet the ToFix graphs do not show a declining trend, then this is something to worry about. When such a stage happens, I have seen the development team manager doing a major discussion with the team to figure out how to reduce these counts and figure out what is happening. In extreme cases, I have seen the team actually take a hard look at these defect counts and then make a recommendation for extending the schedule (which is not a simple step to take).

How to generate graphs for analyzing the testing process in test director?

The graphs that are created during the test director testing process let you keep a track of the progress of the test plan, test runs, defect tracking, requirements and so on. Such graphs can be generated at any point of time during the process and also from any of the test director modules. The graphs created by the test director are based up on the default setting however they can be customized by the user.

A project consists of data of different types. The graphs that you create using the test director can help you a big deal in analyzing the relationships between these different types of data. Each of the modules of the test director comes with a number of graph generating options. After you are done with generating the graph, you can customize its various properties so it comes out exactly as per your specifications and displays the information you want and in the way you want. 

Now we shall mention the steps following which you can generate a defects graph which will show the summary of the defects by status as well as priority levels. 

Steps for generating Defects Graph

Follow the steps:

1. Click on the defects tab to turn on the defects module of the test director. The defects module will be displayed in the defects grid.
2. Now for choosing a graph go to analysis menu, then graphs, then summary, then group by status option. This will open up a defects summary graph. This graph is grouped by status by default.
3. Next you need to clear the default filter. Clicking on the filter button will do the task for you. The filter dialog box will open up. You will see that the detected by field is set to the current user name by default. Here, click the clear button and the applied filter will be removed by the test director.
4. If you want to define a filter for viewing the defects with high to urgent priority then click the filter condition box for the priority field in the filter dialog box. Clicking on the browse button will open up the select filter condition dialog box’. Then select the required logical expression in the right pane and in the left pan select the level. Click OK to save the settings and close this dialog box.
5. Next for defining a filter for viewing the defects that are not closed click the filter condition box for the status field. Again open the select filter condition dialog box by clicking on the browse button. Select the ‘not’ logical expression and select closed in the left pane. Click OK to close this box and once again click OK to close the filter dialog box.
6. For setting the X axis of the graph select priority on the right side of the window for viewing the number of defects according to priority.
7. Clicking on the refresh button will refresh the graph i.e., a new graph will be displayed.
8. For displaying additional defect details click on a bar segment of the graph. A drill down results dialog box will display the defects related to that bar segment. Close this dialog box by clicking on the close button.
9. There are various graph views available such as the data grid view and pie chart. Clicking on the corresponding options will display the graph as a pie chart, grid and so on.
10. Close the graph and click on the back button to go back to the defects module.