Subscribe by Email


Friday, May 22, 2015

Getting statistics of build success rates and reasons for failure

During a regular product development cycle, the Daily Build is one of the most important items that have to successfully happen. If the Daily Build has failed, or is only partly successful, or even needs only some partial work to do before using, it can be a source of irritation for the development team. Even though they can use the previous build, the fixes from the previous day would not be there, which is an irritant. And if such build failures happen from time to time, the irritation can build up and cause doubts about whether the daily build can be depended upon to happen consistently.
There can always be reasons for the daily build to have failed - reasons could be some dependency problem that a developer did not verify before checking in their fix, or some hardware failure, or problems with some recent patch (once we had a case where a new Operating System patch was in turn causing the Virus Scan engine to not release one of the build process files, and it was a pain to diagnose that problem), or some other human failure on the part of the build engineer.
If the build failure happens on a periodic basis, status meetings are sure to resonate with worries about how the frequent build failures are impacting the ability of the team in terms of reduced productivity (and this worry can sometimes be enhanced from the actual levels, but it is not easy to make this statement in the status meeting). Further, it is an imperative that each and every build failure be monitored to ensure that the same reason for build failure does not happen again (or there better be a very good reason why the same reason could cause a build failure again).
The way to do this is a simple process. The build engineer can easily setup a process whereby the statistics for every day's build success are tabulated, and since it is an assumption that the reasons for the build failure or delay would have been monitored by the build engineer, these can be added to the stats for every case where there has been such an occurrence.
On a periodic basis, such a report should be sent out to all the people concerned with the build or who have been nominated by the team to track the build (we had created an email alias for the build issues and whoever wanted could become a member of the build alias, and this build report would be sent to this email alias). There were advantages of having such a status:
- It ensured that everybody had accurate figures about the number of times the build was broken or was delayed
- It pointed out issues that could be regular or cause repeated build failures and how they were handled. This one area of the report was a great confidence-builder
- We were able to monitor whether there were improvements in the build success rate (and that was a reasonable expectation)
- It also pointed out cases where the development team engineers caused failures and put pressure on the development manager to tighten the ship on the development side.


No comments:

Facebook activity