Different types of end-user metrics

Typically, metrics in software are meant to mean something related to capturing lines of code, number of defects, or some other measurements related to software quality. User metrics is something entirely different, meant to try to capture data relating to the customers usage of the software. The advantages that you get out of such capturing end user software metrics are:
1. Capturing such metrics helps in validating design principles: Say both product management and engineering feel that the implementation of a particular feature is the defined way to do it, but after looking at user-functionality usage metrics, you find that the usage of the feature is lower, then some re-thinking needs to happen.
2. Helps in capturing problem areas: Suppose there is logic in the system that it reports every time that the application gets into an unstable or incorrect position (for example, if there are some parameters being passed to a functional that was deemed unlikely), then you know that there is additional work to be done in these areas.
3. Finding priority areas: Reviewing user-functionality usage graphs can be help you find which are the functions that are the most used of the application. Sometimes, product teams get surprised when they review the usage metrics for their applications. Supposed you are the team for a photo editing software, and you expect that some of the editing tools would be the most popular; and then you find to your surprise that a simple tool such as a Quick Fix tool is the most popular. Such data helps you determine the areas that are important to users, and what you need to fcous on.
4. Determining poorly designed workflows: If you have metrics in place that determine how long a users takes to complete a particular workflow as well as how many people have abandoned the workflow, you get an idea of how simple vs. badly designed a workflow is. A poor workflow would leave users spending far more time on that workflow, or even pressing 'Cancel' to leave the function in frustration.
5. Determining which features to drop. Sometimes there are features in a software that are built with a lot of hope, and do not live up to the promise. In addition, there are other features that have outlived their usefulness and need to be dropped as part of a periodic cleaning of the application. Such end user metrics can help a lot in determining the features that need to be removed.
6. Determining license usage. In the case of softwares that are available as floating licenses or as enterprise softwares, capturing of usage information helps in determining whether the pricing of such software is correct. Such metrics also help in determining whether there is any violation of licensing terms.

Product Development: Planning for metrics

What are metrics ? Metrics are becoming an increasingly important part of capturing all sorts of information about the product and about the interaction of users with the software system. If you are looking for a definition, here it is: A software metric is a measure of some property of a piece of software or its specifications. Let me give you a common place application: Microsoft has touted Vista as their most stable and secure Operating System. To validate whether Vista is indeed a stable operating system across the wide ranging hardwares with their users, they need a way to measure how often the system crashes. And hence, you need a crash reporting log that sends back information to Microsoft whenever the operating system crashes (or if this is not possible, whenever the system recovers from a crash). To include user benefits, when this information is reported, the company could build in a system that would also try to analyse the reasons for the crash and provide a crash (this worked for me when the system was able to let me know that a new monitor driver was available).
There are all sorts of metrics (and we will talk about some of these metrics in the next post), this post is more about planning for metrics during the project kickoff stage. How do you plan for metrics during this initial phase ?
- The product management team and engineering need to work out which are the metrics they want to capture (these could be in the nature of: different functions launched by the user within the application; the average time that the user keeps the application alive; and so on)
- The tool to be used for this purpose needs to be evaluated (this may be a tool user by other applications within the company, or an external tool may be needed for this purpose)
- If the tool needs to be purchased, then the funding for the tool needs to be planned
- Effort estimation for the metrics tool needs to be planned. The effort estimate for the metrics tool needs to include the effort estimate for making the hookups to the tool inside the application dialogs, as well as the effort needed for testing of the application metrics.
- Legal approval. In many cases, a metrics tool that is well integrated with the software application needs to capture user steps and workflows, and may involve privacy issues. Hence, it is important that any such effort to capture metrics has been validated with the legal team, and the Terms of Use / License has been appropriately amended.