Product Development: Planning for metrics

What are metrics ? Metrics are becoming an increasingly important part of capturing all sorts of information about the product and about the interaction of users with the software system. If you are looking for a definition, here it is: A software metric is a measure of some property of a piece of software or its specifications. Let me give you a common place application: Microsoft has touted Vista as their most stable and secure Operating System. To validate whether Vista is indeed a stable operating system across the wide ranging hardwares with their users, they need a way to measure how often the system crashes. And hence, you need a crash reporting log that sends back information to Microsoft whenever the operating system crashes (or if this is not possible, whenever the system recovers from a crash). To include user benefits, when this information is reported, the company could build in a system that would also try to analyse the reasons for the crash and provide a crash (this worked for me when the system was able to let me know that a new monitor driver was available).
There are all sorts of metrics (and we will talk about some of these metrics in the next post), this post is more about planning for metrics during the project kickoff stage. How do you plan for metrics during this initial phase ?
- The product management team and engineering need to work out which are the metrics they want to capture (these could be in the nature of: different functions launched by the user within the application; the average time that the user keeps the application alive; and so on)
- The tool to be used for this purpose needs to be evaluated (this may be a tool user by other applications within the company, or an external tool may be needed for this purpose)
- If the tool needs to be purchased, then the funding for the tool needs to be planned
- Effort estimation for the metrics tool needs to be planned. The effort estimate for the metrics tool needs to include the effort estimate for making the hookups to the tool inside the application dialogs, as well as the effort needed for testing of the application metrics.
- Legal approval. In many cases, a metrics tool that is well integrated with the software application needs to capture user steps and workflows, and may involve privacy issues. Hence, it is important that any such effort to capture metrics has been validated with the legal team, and the Terms of Use / License has been appropriately amended.