These series of a posts are about analytics, the measure of information related to a user. The last post in this series (Measuring data relating to user information - Part 6) was about an error in data collection as well as the pitfalls of having a strategy that depended on business decisions being made just on the basis of data collected through analytics. Analytics should be one of the pillars of a decision making strategy, with market research and other factors also contributing to how and why a decision should be made. If you jump into the field of making decisions without having a proper strategy relating to decision making, then there is a good chance that the decision making could go down a path that is faulty or inaccurate.
In some of the previous posts, I have been taking some examples that show where decisions with regard to the product can be taken on the basis of data collected from the product in the hands of customers. In this post, I will take another example of the same - a case where there were certain reports from the customers and the product management was not sure about whether the feedback was proper, and looking for more evidence to substantiate or confirm the problems. So, for this example, there was feedback from several channels that talked about how customers were perceiving that the quality level of the product was not as good as previous versions and this was manifested in more cases of crashes.
Now, this is good feedback, but can you take this as gospel truth. On the surface, this seems like clear feedback that you should recognize and take action accordingly. So, there should be some kind of investigation that would cause you to behave differently from how you have behaved after previous releases, since that will be in customer interest. You would need to commit more effort to investigation and solutions to quality problems; and even though this might seem to be in customer interest, this is a cost to the ongoing product development. And there is the contra view - with more means of expressing discontent such as user forums, community groups and the same, there is the possibility that the quality level is the same as that of previous levels, it's just that the information collection systems are catching more of this discontent.
Now, you are stuck. Both sound good, but you need to take a decision one way or the other. This is where data collection from user systems works good. One of the first items you should be capturing is where information about whether the user application has closed normally, or whether the application has closed abnormally (such as in a crash or when the user was forced to terminate the application after it was hanging) and also capture this information for operating systems that are supported by the application. Further, you would need to do this for the different features and dialogs that are present in the application. Once you have been capturing this information, there is a lot you can do in terms of determining how often a feature or the entire application crashes in the hands of the users, where does the crash happen (this will need a lot of development effort though to determine what causes the crash in the application). This will also help to determine whether the frequency of crash is more than in previous versions of the application release.
Once you have such a data and the data has checked out to be accurate in some respects (for example, if your testing team is getting crashes in reasonably similar areas, then it helps to confirm this data to a large extent), you can make product level decisions. If that means that you need to spend some time on product stability and quality, then you need to do so; otherwise if the quality level seems fine, then you know that the information you are getting from the customers needs to be handled through regular support mechanisms and does not need the development team to spend extra effort.
In some of the previous posts, I have been taking some examples that show where decisions with regard to the product can be taken on the basis of data collected from the product in the hands of customers. In this post, I will take another example of the same - a case where there were certain reports from the customers and the product management was not sure about whether the feedback was proper, and looking for more evidence to substantiate or confirm the problems. So, for this example, there was feedback from several channels that talked about how customers were perceiving that the quality level of the product was not as good as previous versions and this was manifested in more cases of crashes.
Now, this is good feedback, but can you take this as gospel truth. On the surface, this seems like clear feedback that you should recognize and take action accordingly. So, there should be some kind of investigation that would cause you to behave differently from how you have behaved after previous releases, since that will be in customer interest. You would need to commit more effort to investigation and solutions to quality problems; and even though this might seem to be in customer interest, this is a cost to the ongoing product development. And there is the contra view - with more means of expressing discontent such as user forums, community groups and the same, there is the possibility that the quality level is the same as that of previous levels, it's just that the information collection systems are catching more of this discontent.
Now, you are stuck. Both sound good, but you need to take a decision one way or the other. This is where data collection from user systems works good. One of the first items you should be capturing is where information about whether the user application has closed normally, or whether the application has closed abnormally (such as in a crash or when the user was forced to terminate the application after it was hanging) and also capture this information for operating systems that are supported by the application. Further, you would need to do this for the different features and dialogs that are present in the application. Once you have been capturing this information, there is a lot you can do in terms of determining how often a feature or the entire application crashes in the hands of the users, where does the crash happen (this will need a lot of development effort though to determine what causes the crash in the application). This will also help to determine whether the frequency of crash is more than in previous versions of the application release.
Once you have such a data and the data has checked out to be accurate in some respects (for example, if your testing team is getting crashes in reasonably similar areas, then it helps to confirm this data to a large extent), you can make product level decisions. If that means that you need to spend some time on product stability and quality, then you need to do so; otherwise if the quality level seems fine, then you know that the information you are getting from the customers needs to be handled through regular support mechanisms and does not need the development team to spend extra effort.
No comments:
Post a Comment