Defect Management: Dividing overall defect reports into separate functional areas - Part 3

This is a series of posts where I look at the creation of a defect status report, in a way that it provides enhanced information to the team and its managers and helps them to make decisions. In the previous post (Count of new incoming defects), I talked about adding more parameters to the defect report that help in information that can let the team know whether the number of defects getting added on a daily basis will enable the team to reach their defect milestones by the required time and date. This kind of data, coming in on a regular daily cycle, helps the team to decide whether their current defect pattern is on the desired path, above it, or below it, and accordingly make the required decisions.
This post will see more details added to the defect report that provides a lot of useful information to the team. The last post talked about the flow of incoming defects that move to the ToFix state. However, there is another type of information that is relevant. Defects that are present in the system are not only owned by the development team. In addition to the defects in core code, there may be defects that are present in the components used in the application. These are defects that are not attributable to the development team, but to the vendors or other groups that provide the components. A number of teams that I know typically track these defects in the defect database, but distinct from the defects with the core team.
The nature of defects that are against external components is different from those in the core code. Even though to the customer it does not matter whether the defect is within the core code or in an external component, the amount of effort required in terms of coordination and communication is entirely different from the other defects that are with the core developmental team. If a defect is with a component that is not owned by the team, the timeline for fixing of the defect may take longer and need persuasion; or there may be a lot of back and forth between the tester and the external component team to study the situation in which the defect occurs (which also includes sending the environment in which the defect occurred to the external vendor - and this has its own cost and restrictions, since if the team is working on a new version of the software, there would NDA issues and IP issues related to sending details of the environment to the external component team), and so on. Another concern could be that that even if such a defect is resolved, it might need a new version of the component, which has its own extra cost about testing the component on its own to check whether it is fine or there are other issues with the same.
As a result, it needs to separate out the incoming defects about whether they belong to the core team or whether they are attributable to people outside the team; and if the proportion of such defects that are outside the core team is increasing, it is a matter of concern to the team, since resolving such defects typically takes much more effort and time.

Defect Management: Dividing overall defect reports into separate functional areas - Part 2

Part 1 (Dividing defect reports into ToFix, ToTest and ToDefer counts) of this post talked about the importance of Defect Management in a software project, and then got into some details about the regular sending out of a report on Total defects, with these defects having been broken down into ToFix, ToTest and ToDefer stats, maintained on a graph over a period of time with daily updates, so that the team and the managers can figure out whether the team is on progress to resolve these bugs.
This post continues on this line, talking about additional items that can be added to this defect chart and metrics to provide more information to the team and determine whether it is on the right track or not. Are all these metrics important ? There is a lot of comments about not over-burdening people with too many statistics, and there are more comments about letting people do their work rather than sending so many statistics that they stop looking at these stats. However, it is also true that the role of the team managers is to look at the broader situation in terms of project status, and the defect management is an important part of this. It is true that the team members should not be burdened with these figures, but for the team managers, it is critical to look at such data.
So, the team looks at the ongoing figures for defects in terms of ToFix over a period of days and tries to determine whether the team is on the right track or not. So what else should you be capturing ? Another metric that can now be added to such a report is about the number of defects that are still incoming. There are primarily 2 ways in which defects can be added to the count of developers:
- New defects that are logged against the development team and which add to their count and to the overall ToFix count
- Defects that have been rejected by the testing team after they have been marked fixed by the developer but there is a problem in the fix (this can vary a lot among different teams and even in a team - a developer could be fixing defects with hardly any returns and there could be another developer who is under pressure and many of whose defects are returned because of some problems). So, whether to determine this kind of statistic and calculate metrics for such a case determines of whether the team is seeing such kind of returns for the defect management.
Once you have these kind of defect counts, it helps in determining the current status of defects and see whether the team is on the right track. So, you have a total count of open ToFix defects, and there is a decline in such a count needed to hit the deadlines. However, for getting to such a deadline, you need the number of incoming defects to be also fitting into this strategy. If there are a large number of incoming defects, then the team will not be easily able to determine whether their ToFix defect count is decreasing by the amount they want to hit their targets, and this then needs a change to the strategy to determine whether the team will get there or not.

Defect Management: Dividing overall defect reports into separate functional areas - Part 1

Handling defects is one of the major efforts that plays an integral role in handling a project schedule and making it successful. I have known multiple teams where the team did not have a good running estimate of their defect count and the defect estimation over the remaining period of time left in the schedule; as a result, when the team was closer to the final stages of the schedule, they found that they had too many defects that made the remaining part of the schedule very tight - which meant that if they were to do an accurate reckoning of their status, they would need to either defer more defects and maybe end up with a product that is lower in quality; or the product would need to extend their timeline / schedule, which has a huge implication  for the team and many other teams that are involved in the release schedule of the product.
How do you avoid this ? The first paragraph of this post points out a huge problem, but the answer cannot be handled in a single post; it can be handled by a single cheesy phrase but which does not provide any solutions - "You need to do Defect Management". Now, let us get down to the meat of this post - this post just takes a specific aspect of defect management - sending a split of the defect counts as per the different areas. This in turn provides a better view of the defect picture to the team and helps in the process of overall defect management.
We wanted to have a system whereby we could track the counts for each of the separate functional areas and yet have the management team have access to these data on an ongoing basis. These also helped the separate functional teams do a targeting of the counts of the defects of their respective functional areas and work towards reducing this count. So, we took the overall data for defects for the product (open defects) and split these into the following areas:
Open defects:
ToFix (these are primarily defects owned by the development team, although there could be defects that are carried by other team - such as where there are defects with components supplied by external teams)
ToTest (these are primarily defects owned by the testing team, although since anybody can file a defect within the team, there may be people other than the testing team who own a defect)
ToDefer (the exact terminology of these defects can be different across organizations; but these are typically defects that are with a defect review committee for evaluation. These can be significant defects that need evaluation by the committee before they are to be fixed, or these can be defects that are not worthy of fixing but the team wants the committee to take a final call, and so on).
What is the purpose of sending out separate stats on a regular basis ? These data, when plotted on a regular graph over a period of time provides a large amount of information. The team and the managers, although they are in the thick of things, sometimes need to see such kind of aggregate information to take a good decision. For example, if the team is in the second part of the cycle and close to the timeline, and yet the ToFix graphs do not show a declining trend, then this is something to worry about. When such a stage happens, I have seen the development team manager doing a major discussion with the team to figure out how to reduce these counts and figure out what is happening. In extreme cases, I have seen the team actually take a hard look at these defect counts and then make a recommendation for extending the schedule (which is not a simple step to take).

Working with vendors: Setting up a process for queries and the back and forth for these queries

I had been working with a development team for a long time, and one of the most frustrating items for the team was about working with people external to the team. The team had been working on multiple versions of the product for a long time, and had a lot of experience on the product (many members of the team had been working on different parts of the product for more than 5 years) and hence had a level of knowledge in some areas that was superior to the product manager (who also had a lot of experience in the product as well). However, the team needs to work with a large number of people who are not so experience. This happens with a number of teams. Typically, teams work with people who do localization of the product, teams work with teams who do documentation of the product, in many cases, some amount of testing is also done by external team.
In such cases, the biggest problem is that the external teams will NOT have the same amount of experience with the product as the product team has, and in many cases, the external team person working on the product has very little or no experience with the product. In such cases, if there is some kind of fast learning program that enables the external team member to learn about the product, it is beneficial, but in some cases, such a help is not possible (primarily because it takes effort to produce such help, and the product may also be changing fast between versions in which this help may need to be updated fairly quickly).
But, consider that whatever be the level of help available, the external team member will take time to come upto some kind of acceptable level, and they will also have a lot of queries on this front. Some queries would be available if there is some kind of FAQ available for the product, but typically FAQ's cover the product as a whole, and may not be available when some external tester starts getting into a feature in detail. In such a case, you will have people who have queries and things will move faster if these queries are settled quickly.
And this is where the frustration comes in. Unless there is some kind of process available for such queries, the team member can feel that these queries come up on a regular basis or get repeated (in fact, when talking to team members, this feeling has been projected again and again and there are repeated request to ensure that these get resolved).
We have tried multiple measures to ensure some kind of process for trapping all these queries, especially with regard to ensuring that the queries are all categorized as per the different features (and this ensures that a database was built up with regard to queries that these external vendors have). First, we tries using a MS-Excel document for capturing these, but then we started running into concurrency problems about exchanging the document back and forth, and who maintains the master copy of the document. And, given that the team was very comfortable with the defect management software and the vendors would also have to start using this software, we laid down a process whereby all the queries were logged as defects against the specific feature where the query was (and with a keyword that identified it as a query from a vendor), these defects were assigned to the respective feature person, and then the query would be responded to from the respective person, and the vendors were also told to ensure that they went through these defects before they raised a query.

Risk Planning - Look at bug curves for previous projects and identify patterns - Part 4

These are a series of posts that I have been writing the process of risk planning for the project. For a software development project, one of the biggest risks relates to the defects that are present during the software development cycle. These risks could be because of the number of defects are more than expected, or defects are of a much higher severity than expected, or that some of the defects are getting rejected with the fixes either being inadequate or totally problematic. Typically, for software or for projects that have been going through several versions, there is a lot of information available in terms of the data for the previous versions of the software projects, with this information being very useful to make predictions about the defect cycle during the current version. However, at the same time, my experience taught me that not everybody even considers looking at previous cycles and figuring out problematic areas that may turn up, and how to handle those. Being to able to predict such problematic points and then figuring out how to resolve them is an integral part of risk planning.
In the previous post (Looking at bug curves for previous projects and identifying patterns - Part 3), I talked about how the bug graph analysis from previous projects indicated that there were a large number of open defects soon after a point when the development team had completed the handover of all the features to the testing. This was essentially because a number of the big ticket features came near this date, and this was the first time that the testing team was able to test these. But this overload of defects was something that was causing a lot of pressure to the team, and it was unhealthy for the project. We had to resolve this, but we would never have even actually realized this problem if we had not mined the data from previous projects.
In this post, I will take another example of what we did using the mining of the data of looking at defects from previous cycles and then doing some analysis of that data. There can be many such cases, and I will take up some of these cases that were important to us in these posts, but you can extrapolate these to as many cases as you like and then gather the data and do the analysis.
One of the quests that we had for our team was to figure out how to decrease the time period that a defect was open with a developer - or more accurately, the time period in which a developer first looked at a defect. One of the biggest problems was that there would be defects which were not looked at by a developer for many weeks because of the load that the developers had, and when they got the time to look many of such defects, the tester did not accurately remember the conditions, or the software application had been changed many times since then because of all the developers checking in their changes, and it was not easy to resolve such defects. In short, the quest was that defect ageing be reduced from the many weeks that it was currently, and everybody agreed that we had to do something. However, before we proceeded to do something, we needed to know the extent of the problem, especially at different parts of the development cycle.
For this purpose, we again needed to refer to the defect database for data for previous versions, set up the query for this, and then get the data. To some extent, we also compared the data for multiple previous versions to see whether there were patterns in the data, and we did find some patterns. The analysis for how to reduce this, and what this involved in terms of changes in terms of defect management was something that took effort, but there were rewards for the same. But the most important part remained that we could only do all this analysis once we had the data for the previous cycles, and this was recognized as an important of risk management.

Read more about this in the next post (Risk Planning - Look at bug curves for previous projects and identify patterns - Part 5)

Risk planning - Look at bug curves for previous projects and identify patterns - Part 3

This has been a series of posts on doing some risk planning by focusing on the defect management side. There is a lot that can be done to improve the defect management in the current cycle by looking at the defect trends from the previous cycles (in most cases, there is a lot that can be learnt from previous cycles; it's only totally new projects where nothing can be learnt). In the previous post (Looking at bug curves and identify patterns - Part 2), I took more of a look at ensuring that the defect database is properly setup for getting the information needed for generating this data. Unless this sort of data can be generated for the current cycle, it cannot be used in the next cycles. Hence, there is a need to make sure that required infrastructure is in place for the same.
In this post, I will assume that the data is there for previous cycles to interpret and hopefully generate some actionable items. In the first post in this series, I had already talked about some examples that can be used such as identifying a phase in the project when there have been more defects rejected in the past and then try to change that. There are clear benefits of doing such kind of work, and the project manager should be able to get some clear improvements in the project.
The way to use this kind of defect data from previous cycles is to be more systematic in terms of analysing it, and then identifying some clear patterns from this analysis that will help to identify improvements. So, one clear way of starting on this line is to take the entire defect curve from the previous cycle (charting the number of open bugs against the time axis, and doing the same for other bug stats as well (consider the example of rate of defect closure, the number of defects that have been closed, the number of defects that were closed by a manner other than fixing, and so on).
Based on these various defect stats, there is a lot information that can be generated that can be used for analysis. For example, we used to find that the number of open defects used to shoot up to its highest figure soon after the development team had completed their work on all the features, and this was basically a number of the features would come to the testing team near the final deadline for finishing development, and this was when the testing team was able to put in their full efforts. They were able to find a large number of defects in this period, and as a result, the peak in open defects would happen around here. This was a critical timeline in the project, since the number of defects with each developer would reach their maximum, and cause huge pressure.
Based on the identification of such as pattern, we put in a lot of effort to see how we could avoid a number of features coming in near the end. We had always talked about trying to improve our development cycle in this regard, but the analysis of the defects made it more important for us to actually make these changes. We prioritized the features in a different way such that the features were we expected more integration issues were planned to be complete much earlier in the cycle. None of this was easy, but the analysis we had done had shown how not changing anything would lead to the same situation of a large number of open defects, as well as the increased pressure on the development and testing team. Such pressure also led to mistakes, which caused more pressure, and so on ....


Read more about this effort in the next post (Look at bug curves for previous projects and identify patterns - Part 4)

Risk planning - Look at bug curves for previous projects and identify patterns - Part 2

In the previous post (Looking at bug curves for identifying patterns - Part 1), I talked about how the defect curves for the previous versions of the software product should be reviewed to look for patterns that will help in predicting defect patters in the current release. As an example, if there was a phase in previous cycles where the pressure caused a greater number of defects to be either partially or completely rejected, then this was worrisome. Such a problem area is something that should be focused on to ensure that these kind of issues are reduced or removed totally altogether, and the benefits can be considerable.
In this post, I will continue on this subject area. One of the biggest problems that teams face is that they are so worked up in terms of what is happening in the current release that they use previous versions for help in estimation, and do not really use defect management and analysis to the level they should. One of the critical focus items in terms of project management and risk planning is about understanding that one looks not only at the current version of the software development cycle, but also at future versions. There is a lot of learning to be had from current releases and one should ensure that we are able to gain from this learning in the next release.
The Defect database should be set in such a manner that while it provides you the functionality to do your defect management for the current release, there should be the ability to store this information. If for example, your defect database is not able to store the data that defects have been rejected, or capture the fact that defects did multiple rounds back and forth between the developer and tester, you are losing out on data that is pretty important. We actually ran into such a situation, where we wanted to determine which defects are going back and forth between the developers and testers, or even between multiple people on the team. This was a way to figure out which defects are taking more time, and it seemed like a good place to start. The concept was that if we could figure out these from the previous cycle, and were able to get a figure on which sets of people do not work well together (too much back and forth between people over a defect is certainly not useful, you would expect people to collaborate and resolve issues rather than doing discussion in a defect).
However, things did not work as well as we expected. The defect management system did not have such a query or something even similar to it. It was possible to get this information once we got access to the tables in the database, but this is not something that is easy or quick to do. We needed to get hold of people who had some expertise in how the database was structured, and also needed access to people who knew how to manipulate the database and get us the report that we wanted. It took a fair amount of time, and in the end, we got what we wanted. The results were interesting. They showed that there was a person in the team who wanted every bit of information to be in the database, even something that was clear could be asked and did not add any value to the defect information. But, the defect was passed back to the other person, and then it would depend on the time and defect load of the other person. However, since the person did not have the defect on himself, normal statistics would not show any problem.
We took some action on this one in terms of some counseling for the person from the manager, in a non-threatening way, and resulted in improvement in terms of defect handling. However, it was difficult to quantify the time saved, but we were satisfied in terms of the results we could see, and felt that there was improvements we made in the defect management, and would help in terms of reducing the load due to defects.

Read more about this effort in the next post (Look at bug curves for previous projects and identify patterns - Part 3)

Risk planning - Look at bug curves for previous projects and identify patterns - Part 1

As a part of the risk planning for projects, defect management is one of the key areas to handle and handle well. A large number of defects may not be a problem if they are expected, but if unexpected, they can cause huge problems to the schedule of a development cycle. As an example, if the number of defects is much higher than expected, there are many problems that occur:
- The amount of time that is required to be devoted to resolving these defects and testing the changes will cause a strain on the schedule.
- And it is not just the effort change, a sudden high number of unanticipated defects will cause the team to wonder about the quality of the work done,
- Most important, will cause a lot of uncertainty about whether all the defects have been found, or are there many more to be found ?
- When more defects are found, the overall cycle of analyzing the defect, figuring out the problem, doing an impact analysis of the change, making the change, getting somebody to review the change, and then testing the change will cause a lot of strain.
- Some of these changes can be big, and will require more effort to validate the fix, and in some of these cases, the team management will decide that they would not want to take a risk and would rather that the defect get passed onto customers.
The above was just an example of what happens when there are a large number of defects that suddenly crop up. However, it would be criminal on the part of the team management and the project manager if they have not already done a study of the kind of defects that come up in different stages of the development cycle. The best way to do this, and do some kind of forecasting, is to look at the bug curves of previous cycles (which obviously cannot be done if the data about defects of previous cycles is not taken at that time, or stored at a later point of time).
We had started doing this for the past 2-3 years or so, and this helped us determine what points of the development cycle had the highest number of defects that were found, as well as closed, and even which were the times when there was the highest chances of defects not being fixed (either being partially fixed or being rejected totally by the tester). Now, even though every cycle would be different, there were some patterns that had a high probability of being repeated (and this even true when one project differs from the other, although it varied from project to project).
Let us take another example. There was a time in the project when defects had a higher chance of being rejected, and a rejected defect can be very expensive in terms of time of both the developer and the tester. As a result, during such stages that had happened in previous projects, we would ensure that there was higher focus on impact analysis and code review, and had even borrowed some senior developers from another project for around a month just for the additional focus on reviews. This paid off, since the number of defects getting rejected went down considerably, which per single defect did not amount for much, but our analysis showed that the total saving of effort because of this additional effort was around 25%.

Read more about this effort in the next post (Look at bug curves for previous projects and identify patterns - Part 2)

Defect handling - Planning how to find bug bugs earlier in the cycle

In every cycle of software development, whether this be product development or working on a project, one of the key items is about finding defects. There can be small defects or big defects, but the plan is always to try to fix these defects. Smaller defects can sometimes be more easy to handle, since they have a lesser impact on the customers, and in several cases, it would even be easy to defer some of these bugs if they are low severity and there is a squeeze on time and resources. However, larger bugs, those that have an impact on functionality or workflows are harder, and so are the ones that are complex or need more time to fix. These are the sort of bugs / defects that a team would find hard to defer or leave alone, and they can be critical to fix.
One of the biggest problems with such defects is the time period in which many of these defects are found. Defects which have a high impact are typically found in the latter part of the cycle, primarily because a lot of the functionality comes to be ready in the latter half of the cycle. If there are a number of features in development, the earlier parts of the cycle see the development of these features. As time progresses, these features start getting into a good shape and the integration points of these features start getting worked on. This is the time when the workflows of the product start coming into shape, and that is when the testing team will be able to check the integration of these features into one another.
The testing effort at this stage is able to detect workflow problems, as well as design flaws where the type and amount of information flowing from one feature to the other may have issues, and not happening as per design. These defects take more time to analyse, and may also need teams from the different feature areas to collaborate to figure out the defects. As a result, the time involved to fix these defects is more, and this gets problematic when the number of such defects found is more than expected, which means that the time that the team has to fix issues may be less. Further, the later that such defects are found, the greater the risk that the fix for such defects can cause other problems; because of such risks, defects found later are at a greater risk of not being fixed.
What do you do ? Some of the issues may be more problematic to solve. Feature development work focuses on the specific features in the earlier part of the cycle, with the focus shifting to integration only later in the cycle, so changing the timelines for this may be more difficult. But, it is possible to do studies to estimate the amount of bugs that will be found (far easier of this is just a new version of the product being developed and there is historic data), and then plan for more time for the same. At the same time, a number of problems typically are found out if there is inadequate time spent during the design and architecture phase and teams should ensure that they are spending the right amount of time on these activities. Further, as the feature is being developed, workflow or integration related flows should be tested, even if integration has not been completed. An example of this can be done is to prepare a software harness which will allow the input and output of data from the various features even if the integration has not been done. Doing this ensures that a number of the defects that are found post integration can be found earlier in the cycle, and this saved a lot of time and effort.

When is a defect not a defect and should be classified as a feature ?

This is an interesting question that causes a large amount of debate in many teams during their development process. A defect is something that needs to be corrected (in terms of the discussion in software teams) - and this is without getting into the exact semantics of the definition of a defect. So, in the regular process, a tester finds a problem that needs to be corrected and is assigned to a member of the development team to fix. The development team member analyses the defect, finds the required change, and then makes the necessary fix, the fix is rolled into the product and can then be tested by the tester (the same one or somebody else) and if it is fixed, the defect is marked as verified and can be closed.
This works most of the time, but there are cases when the process does not work like this. For example, there are times when the defect that is marked in the system, when analysed, is found to have a big change. It could be that the fix for the defect causes a change in a code section or file that is used in a number of areas of the application. When this happens, it is not so easy to make a change, since there can be a large impact. The impact analysis of such a change needs to be more wide-spread, looking at the change that needs to be done in the different areas, the change needs to be thoroughly reviewed, and then once the change is made, a thorough testing needs to happen. However, this is not the case that I am talking about, where the defect is still a defect and not a feature.
The feature change is a different kind of defect. This is when a defect is logged which talks about some change in the workflow. It could be possible that when a defect is logged, the defect was logged on the basis of an interpretation of the tester, or the defect is actually tracking a specific point of the product where the feature definition was missing or flawed. Consider a case where a team has defined a new feature, where all the workflows that were written as a part of the product development were done, and then it was discovered during testing that the specific workflow of one area was not done as it should have been.
So what to do ? Well, whenever such a defect is written, it should be categorized as a feature rather than a defect. A defect means that the focus is more on the fixing of the defect, while if you are looking at the problem from a feature perspective, there will be more attention paid. The product manager, the designer, the feature team which was looking at the feature, all of them should be brought in, and they need to look at what the nature of the change should be, whether this impacts more than just the workflows before and after, and so on. It is also possible that if this is at a late enough stage, this group makes a decision that the feature will be left as it is, or the team recommends the change which is then made. Doing it as a feature will also ensure that other teams that need to look at feature changes such as documentation and localization will be informed as a matter of process. In the end, the treatment of such a change may be finally be done through the defect process, but this is done only after the change has had a thorough look at it.