Different types of end-user metrics

Typically, metrics in software are meant to mean something related to capturing lines of code, number of defects, or some other measurements related to software quality. User metrics is something entirely different, meant to try to capture data relating to the customers usage of the software. The advantages that you get out of such capturing end user software metrics are:
1. Capturing such metrics helps in validating design principles: Say both product management and engineering feel that the implementation of a particular feature is the defined way to do it, but after looking at user-functionality usage metrics, you find that the usage of the feature is lower, then some re-thinking needs to happen.
2. Helps in capturing problem areas: Suppose there is logic in the system that it reports every time that the application gets into an unstable or incorrect position (for example, if there are some parameters being passed to a functional that was deemed unlikely), then you know that there is additional work to be done in these areas.
3. Finding priority areas: Reviewing user-functionality usage graphs can be help you find which are the functions that are the most used of the application. Sometimes, product teams get surprised when they review the usage metrics for their applications. Supposed you are the team for a photo editing software, and you expect that some of the editing tools would be the most popular; and then you find to your surprise that a simple tool such as a Quick Fix tool is the most popular. Such data helps you determine the areas that are important to users, and what you need to fcous on.
4. Determining poorly designed workflows: If you have metrics in place that determine how long a users takes to complete a particular workflow as well as how many people have abandoned the workflow, you get an idea of how simple vs. badly designed a workflow is. A poor workflow would leave users spending far more time on that workflow, or even pressing 'Cancel' to leave the function in frustration.
5. Determining which features to drop. Sometimes there are features in a software that are built with a lot of hope, and do not live up to the promise. In addition, there are other features that have outlived their usefulness and need to be dropped as part of a periodic cleaning of the application. Such end user metrics can help a lot in determining the features that need to be removed.
6. Determining license usage. In the case of softwares that are available as floating licenses or as enterprise softwares, capturing of usage information helps in determining whether the pricing of such software is correct. Such metrics also help in determining whether there is any violation of licensing terms.

Product Development: Planning for metrics

What are metrics ? Metrics are becoming an increasingly important part of capturing all sorts of information about the product and about the interaction of users with the software system. If you are looking for a definition, here it is: A software metric is a measure of some property of a piece of software or its specifications. Let me give you a common place application: Microsoft has touted Vista as their most stable and secure Operating System. To validate whether Vista is indeed a stable operating system across the wide ranging hardwares with their users, they need a way to measure how often the system crashes. And hence, you need a crash reporting log that sends back information to Microsoft whenever the operating system crashes (or if this is not possible, whenever the system recovers from a crash). To include user benefits, when this information is reported, the company could build in a system that would also try to analyse the reasons for the crash and provide a crash (this worked for me when the system was able to let me know that a new monitor driver was available).
There are all sorts of metrics (and we will talk about some of these metrics in the next post), this post is more about planning for metrics during the project kickoff stage. How do you plan for metrics during this initial phase ?
- The product management team and engineering need to work out which are the metrics they want to capture (these could be in the nature of: different functions launched by the user within the application; the average time that the user keeps the application alive; and so on)
- The tool to be used for this purpose needs to be evaluated (this may be a tool user by other applications within the company, or an external tool may be needed for this purpose)
- If the tool needs to be purchased, then the funding for the tool needs to be planned
- Effort estimation for the metrics tool needs to be planned. The effort estimate for the metrics tool needs to include the effort estimate for making the hookups to the tool inside the application dialogs, as well as the effort needed for testing of the application metrics.
- Legal approval. In many cases, a metrics tool that is well integrated with the software application needs to capture user steps and workflows, and may involve privacy issues. Hence, it is important that any such effort to capture metrics has been validated with the legal team, and the Terms of Use / License has been appropriately amended.

Product Development - Prerelease planning

During a product development cycle, getting inputs from users is of great importance. It is not so easy for a development team to get inputs so easily. One way is to find focus groups or do other usability studies where groups of users from the desired customer groups are quizzed about their needs and their workflows, and based on that, user workflow design is made. However, such studies are many times one-time only, and it is hard to do such usability studies on a continuous basis to get frequent guidance to the development team. One option for works fairly well in such scenarios is to get a pre-release / beta program where people from the target customer base interact with the team on a regular basis. They would do this under a NDA (Non Disclosure Agreement) where the fear of details of the product being revealed to the public are far reduced.
A pre-release program needs to be planned thoroughly, ideally when the project planning is happening. Some of the factors that needed to be planned as a part of this are:
1. At what stage should the pre-release program start ? Right at the start of requirements detailing, where pre-release participants can review snapshots of how the workflow would look like and provide details ? If this can be achieved, then it would make sense to do this through the development cycle. The team can get quick feedback of user reaction to proposed designs.
2. What is the number of participants that should be part of this program ? This is a question for the product team, but given that only a percentage of people signed up for a pre-release program actually provide useful feedback over a period of time, the team should plan to sign up more people than required.
3. What are the features that should be exposed to users through a pre-release program ? Features should be selected that are new, features where there are extensive usability issues, and features where there is a lot of doubt about the proposed implementation.
4. How should the users be exposed to features ? The normal process is that in the initial stages, pre-release users would be exposed to mockups that detail the workflow of the features, and as time goes by and builds start getting made with the features being implemented, the pre-release users would also start getting exposed to these builds and features and can test them with their own perspective and provide useful feedback.
5. Actual process for taking feedback: Typically when pre-release users get going, they can generate a significant amount of feedback over the cycle of the product development cycle, in terms of feedback on features, new feature requests, and bugs. All of them need to have their own channels for capturing such feedback; with people on the product team being deployed for monitoring the feedback, a separate process for making sure that bugs logged by users move into the bug tracking software used by the product team and the pre-release users are able to view the progress of these bugs.
6. How many ratings of the product: It is always useful to get users to rate the product along with allowing the pre-release users to provide open-ended feedback of the features in the product. If you do this a number of times in the cycle, it provides the product team with a view of which features most appeal to the users; it also give the product management team a list of features that can be taken for the next version.
There would be many other advantages as well, please provide these via comments when you know of other advantages of a robust pre-release system. For example, getting the pre-release system in place allows testing of the product across a broad range of devices and operating systems.

Product Development - Actual Kickoff of Development Effort

During every product development cycle, there are 2 distinct stages that you reach in the cycle. There is an initial phase when the planning happens, and there is a later stage when the actual work happens with respect to the actual design and development effort. These are very broad level definitions, with many details and finer points.
- Planning stage: This is the stage where the team defines with broad strokes, the details of what the product will look like in terms of features. The general direction of the product, the features that will make it to the release, the schedule of the release, all of these are decided in this stage. In addition, if the company is structured in such a way that the configuration group, installer and release teams, internationalization teams, and other central technology groups are separate teams, then they will need to be signed up.
- Implementation stage: In this stage, the teams get working on actual implementation of features. By this time, the product team has already decided on the features, their priority, and schedule. The core of the development phase, which can be translated into requirements breakup and detailing, user design, engineering design, test case and test plans, coding, testing, and release all happen in this stage.
In between these 2 distinct phases of the project, there is a need to get an actual kickoff. If the product is such that it needs an approval from executive management, and also to get all the supporting team on the same discussion stage, a typical kickoff meeting is called. In this meeting, the project manager / program manager gets all the stakeholders (including the product team, management, executive sponsors, supporting teams, product management, etc) onto the same room and presents the schedule, final feature list, commitments of support, languages, pre-release plans, issues and risks and so on.
Such a meeting typically takes around a month to plan, since you need to get a meeting time right which is convenient for all the attendees, you need to prepare a presentation to run in the meeting, and so on. A kickoff meeting is the actual time when the sponsors say yes to the product development phase proceeding, at the same time, there is also a chance that the executive management team may need clarifications or may even ask the team to go back and review their plans.

Product Development - planning a minor / dot release - Challenges

In the last post, I had talked about why a minor (dot) release is needed, as well as some of the reasons as to why doing a dot release is an inconvenience. However, if the decision has been made to do a dot release, then it is necessary to understand the process of planning and executing a dot release, including some of the difficulties (challenges) that emerge in such a minor release.
What are some of the activities that need to be done ?
1. You need to finalize the changes that need to be a dot release. If this release is because of some known changes, then the changes need to be analysed, and the engineering response (and design) needs to be worked out.
2. If the changes are still unknown, for example, if your release is failing some security tests or some certification, then you need to figure out what can be done. If you take an example where the earlier release is failing some new certification norms (and for those who know how certification works, it can be a lot of effort to prepare the infrastructure and execute all the cases. As an example, you may need to take the help of tools for certification, and those tools may need an upgrade of memory. In other cases, the amount of testing required may be huge, and the actual calendar time needed may stretch the schedule of the dot release.
3. For companies where a lot of work is handled by support teams (configuration, build, release, internationalization, etc), the required overhead of handling all those teams and getting their support takes both budget allocation and time.
4. Too many minor releases causes cynicism in the market about the initial stability of the software release. For example, Microsoft releases many service packs for its software, and there are many people who do not migrate onto a newer version until they see 1 or 2 service packs because they would rather wait for some of the important bug fixes.
5. You run into issues where there are multiple releases for around the same version (with say versions 8.0, 8.1, 8.2, etc for a product). Once you have multiple versions, you get into support issues whereby issues are different for these versions, and support may be a nightmare.
6. Newer dot versions, in many cases need to be deployed to the retail channel (replacing the software already in the retail channel), and to the web store (including to many online software sellers who resell the software). All of these steps involve huge logistical nightmares and / or costs.
7. Branching strategies. Getting a dot (minor release) in process needs major configuration issues (getting branches in place, especially when work is also going for the next release), and takes a fair amount of explanation to both the Development and Quality teams.

Of course, if you folks can suggest more issues that make minor releases a challenge, please add comments.

Product Development - what is a minor / dot release

For those of you unacquainted with the concept of a minor release (I will also call it a Dot release from time to time), it is a release that you do after the main release has been done. If that did not make too much sense, let me try again ! Suppose you have released version 8.0 of Microsoft's Visual Studio to great fanfare, and with much expectation that this is 'the' release, perfect in all ways. Once you release 8.0, the overall grand plan would be to sell 8.0, and have the team working on developing 9.0 with no constraints.
Let me tell you a little secret. As you move closer to the release date of a product version, you (and in fact, most of the team members), start entering a time of partially restrained panic and impatience. It is also a normal decision that as you get closer to the release date, bugs that would have been fixed by the project team even a couple of months back will not be fixed now, unless they are deemed critical. The primary reason for this customer unfriendly move ? Simple Murphy's Law - 'If anything can go wrong, it will' ! Any bug fix has a cost - the fix needs to be reviewed, and then the quality team needs to test the impacted area of the bug. If something goes wrong with the bug fix earlier in the cycle, you have time to fix the impact. However, as you get closer to your release date, this buffer time is no longer available. As a result, almost every product team takes a call that as you get closer to the release date, bugs need to be more and more critical before they can be fixed. Thus, the product team may opt to not fix bugs that could potentially impact customers, but if the bug is found say a week before release, the bug would be classified based on whether you would stop the release for such a bug. Sadly, very few bugs make the criteria of being a release-stopper.
Once you have released version 8.0, there will be bugs that will come in from the field (including bugs that the product team had decided should not be fixed). Also, there may be other changes that are required in the release, and which cannot wait for version 9.0. And thus the stage is set for a generation of release of Visual Studio 8.1 (now you know where the term dot comes from, since this minor release is actually 8 'dot' 1). Such releases follow a few basic points:
1. Bugs (whether older bugs, or new bugs generated by customers) are evaluated as to whether they are deemed important enough to fix; these are the only bugs that are selected for inclusion in the dot
2. Typically, when a dot release is planned, the release supplants the earlier release. So, following the above example, new customers will get release 8.1 of Visual studio vs. release 8.0 that was released earlier
3. A dot will typically have all the activities of a full release, the only advantage being that the time frame is drastically reduced. A reverse is that the inter-group coordination that had more time available has to be compressed into a shorter time frame
4. A dot inconveniences everybody, since it takes much more attention to do everything in a shorter time period; further, it takes away resources (including team management attention) from working on the next release
5. Reaction time is much less than in a normal release

Product Development - Rolling out the patch

The previous post had thoughts on what are some of the conditions under which a software patch is typically required. Once the decision has been made to make a patch, there are a set of activities that are needed to be done. A patch typically is a miniature version of a complete product development cycle, not with all the activities, but certainly having many of the steps that one needs to carry out in the typical cycle. What are some of these steps that are needed to be done ?
- Decide on which of the fixes need to make it to the patch: When preparing a patch, there is always the temptation to fix all the issues that have been found after the release. However, beware of feature creep. You should only include fixes on the patch that are deemed critical. Putting other fixes means that the time period for the patch becomes longer, it means that QE needs to out more effort to test the patch, and so on.
- Make sure that all the teams are signed up for this patch. For a moderately sized software company, there will be multiple teams (with their specialized functions) that are needed to make the patch. These include the development team, quality testing team, configuration team that is responsible for the installer and release, localization team (if there are multiple languages involved - this is another decision point, whether the patch needs to be rolled out for all languages), marketing teams to ensure that patch publicity plans have been rolled out, web teams (if the company website is maintained by another team, they need to involved so that they can schedule the patch rollout date into their plans)
- Decide the schedule of the patch. Even though this is an abbreviated release, it will still take time, and the further this time pushes into the schedule for the next release, the more impact that this patch will have in terms of creating problems for the next release.
- Decide on branching strategies. Typically teams need to make sure that the source code repository has a proper branch created to handle the new branch for the patch. In most cases, the entire team will not be involved in making the patch, and so the work being done by the rest of the team for the new version will be on the main branch, and the work being done by the engineers working on the patch will be on the separate branch. A strategy also needs to ensure that the defects being fixed on the patch will be integrated with the main branch once the work on the patch is done.
- At the time the work on the patch is being carried out, if the patch is due to some major customer issues or some security issues, then the marketing and management of the team need to decide whether to release news of an upcoming patch. Typically, news of patches are not released earlier, but sometimes such news can help in calming down customers who would otherwise be worried.
- Development work for the fixes that go into the patch will need to be done. This typically involves a thorough engineering investigation into the issues that need to be fixed as well as working with outside partners if the issues needs such cooperation
- Once the fixes are made, they need to be thoroughly tested by the QE team on the different platforms on which the software is supported
- Now the patch is ready, and needs to be rolled out through the different media that can be used. The patch can be mentioned on the company web site, on the product page, in customer support forums, to product users directly if there is the ability to send software users a notification inside the product, via email to all the registered users, show as an update to users if the app has the ability to do an automatic check for updates.

The next post will talk about minor (dot) releases and the difference between a patch and a dot release.