Getting statistics of build success rates and reasons for failure

During a regular product development cycle, the Daily Build is one of the most important items that have to successfully happen. If the Daily Build has failed, or is only partly successful, or even needs only some partial work to do before using, it can be a source of irritation for the development team. Even though they can use the previous build, the fixes from the previous day would not be there, which is an irritant. And if such build failures happen from time to time, the irritation can build up and cause doubts about whether the daily build can be depended upon to happen consistently.
There can always be reasons for the daily build to have failed - reasons could be some dependency problem that a developer did not verify before checking in their fix, or some hardware failure, or problems with some recent patch (once we had a case where a new Operating System patch was in turn causing the Virus Scan engine to not release one of the build process files, and it was a pain to diagnose that problem), or some other human failure on the part of the build engineer.
If the build failure happens on a periodic basis, status meetings are sure to resonate with worries about how the frequent build failures are impacting the ability of the team in terms of reduced productivity (and this worry can sometimes be enhanced from the actual levels, but it is not easy to make this statement in the status meeting). Further, it is an imperative that each and every build failure be monitored to ensure that the same reason for build failure does not happen again (or there better be a very good reason why the same reason could cause a build failure again).
The way to do this is a simple process. The build engineer can easily setup a process whereby the statistics for every day's build success are tabulated, and since it is an assumption that the reasons for the build failure or delay would have been monitored by the build engineer, these can be added to the stats for every case where there has been such an occurrence.
On a periodic basis, such a report should be sent out to all the people concerned with the build or who have been nominated by the team to track the build (we had created an email alias for the build issues and whoever wanted could become a member of the build alias, and this build report would be sent to this email alias). There were advantages of having such a status:
- It ensured that everybody had accurate figures about the number of times the build was broken or was delayed
- It pointed out issues that could be regular or cause repeated build failures and how they were handled. This one area of the report was a great confidence-builder
- We were able to monitor whether there were improvements in the build success rate (and that was a reasonable expectation)
- It also pointed out cases where the development team engineers caused failures and put pressure on the development manager to tighten the ship on the development side.

Ensuring a regular email with changelist information ..

The daily build is a pretty important item for a development team. There is a need to have a new build of the software on a daily basis, so that defect fixes and code changes make ti to the testing team as quickly as possible. If the build gets delayed for any reason, it means that the development team will have to wait to get these fixes. These delays cause a delay in the actual schedule, and hence most teams have a schedule whereby they get a build early in the morning before the team members have arrived, and also because this means that everything that has been done in the previous day will be available in this build.
However, just because you need a build every day does not mean that you will get it. The process goes that all the changes done in the previous day are all checked in the source safe that the team uses, and based on the last checkin (typically at the cut off time decided between the development team and the build team), the build is fired and takes the required amount of time to process and become a build that is available to the team in time before they come in the next morning.
However, there are a number of things that can go wrong. One of these items relates to whether a checkin done by a developer near to the cutoff time has been incorporated in the build. At near milestone times, there will be a lot of discussion between the development team and the build team to be sure about the actual checkin that is used as the last checkin on which the build has been fired, but on a regular basis, such kind of discussions do not happen. And it is not only the last checkin that can cause a problem, there are other components that go into the application which need to be included in the build and become part of the application. However, all of these are engineering tasks, and sometimes the configuration can go wrong, or the build number that is picked up for one of the components could be wrong (we had a tough case once where a particular functionality was not working as desired, and we found that there was a problem in the execution of a script, which was getting affected due to one of the latest patches, and hence it was picking up the previous version of a component - this was painful to find and diagnose). The other case was when somebody made a great big checkin, but it did not make it into the build while the testing team was under the impression that it was there in the build and start to find issues.
As a result of all these cases, one of the solutions that we had come up with was about ensuring that the build script would also number of the last checkin, and also capture the latest version numbers of all the components that go into the system, and make this available along with the build. Since we also had defect fixes in the defect management system marked with the checkin number, the testing team was easily able to figure out whether the defect fix that they were waiting for made the current build, or came in too late and would only be available in the next build. At times, especially near milestones, the team could even request a new build with this latest checkin to ensure that the feature or defect that they had been waiting for was available to them. Similarly, the version numbers of the components made it very easy to determine whether the component list that was there in the product was the latest.