Tracking platform usage and making decisions based on that

Even in today's data based world, if you are an analytics expert, you can't expect to be totally popular, or that people will welcome you with hands outstretched. There are many aspects of a product development cycle that would benefit from integration with a data analytics cycle - generating the questions, collecting the data, and the extremely important task of generating the output information and conclusions (and a wrong conclusion at this point can have negative implications for the product - putting in effort in wrong or unimportant areas). However, consider the case where there is support for using analytics and there are resources dedicated for ensuring that requisite changes to the product can be made (based on the analytics output).
One major area where the team needs to gather information is on the area of platform support. Platform could mean the Operating System (MacOS, Windows, others), Versions (Windows XP, Windows 7/8, etc) as well as browsers support (versions of Chrome, Internet Explorer, Firefox, Opera, etc). These can make a lot of different in terms of the amount of effort required for product development. For example, on different versions of Windows, the versions of systems files are different and it is a pain to support these different versions. There can be small changes to the functionality of the system based on these files, and in many cases, the product installer would actually detect the Windows version and install the required versions of these systems files. If you could find out that the number of people using these different versions, and find out that one of these versions is being used by a small number of consumers, then the decision to drop that particular operating system may be taken.
So, this is one of the areas in which analytics can work. The simplest way to do that is to make it part of the functionality of the product, where the product, on the user's computer, dials home and provides information such as the platform on which the product has been installed. Once this kind of data has been collected and a proper analysis is done, then the product team can look at this information and factor that into the feature list for the next version of the product. The team will need to decide on the time period for which the data would be captured, and also some of the benchmarks which will decide whether data from the analytics effort needed to be used for making decision (for example, the data seems to be such that people feel that the data is widely different from public perception).
However, this is also dependent on how much the team can depend on this data and the analysis; after all, even a small variation during the analysis can result in information that has levels of inaccuracies in it. But, it is necessary that the team spends effort in the analytics effort, since the payoff from using accurate data analysis and interpretation is very high.

Analytics - Measuring data relating to user information - Part 7

These series of a posts are about analytics, the measure of information related to a user. The last post in this series (Measuring data relating to user information - Part 6) was about an error in data collection as well as the pitfalls of having a strategy that depended on business decisions being made just on the basis of data collected through analytics. Analytics should be one of the pillars of a decision making strategy, with market research and other factors also contributing to how and why a decision should be made. If you jump into the field of making decisions without having a proper strategy relating to decision making, then there is a good chance that the decision making could go down a path that is faulty or inaccurate.
In some of the previous posts, I have been taking some examples that show where decisions with regard to the product can be taken on the basis of data collected from the product in the hands of customers. In this post, I will take another example of the same - a case where there were certain reports from the customers and the product management was not sure about whether the feedback was proper, and looking for more evidence to substantiate or confirm the problems. So, for this example, there was feedback from several channels that talked about how customers were perceiving that the quality level of the product was not as good as previous versions and this was manifested in more cases of crashes.
Now, this is good feedback, but can you take this as gospel truth. On the surface, this seems like clear feedback that you should recognize and take action accordingly. So, there should be some kind of investigation that would cause you to behave differently from how you have behaved after previous releases, since that will be in customer interest. You would need to commit more effort to investigation and solutions to quality problems; and even though this might seem to be in customer interest, this is a cost to the ongoing product development. And there is the contra view - with more means of expressing discontent such as user forums, community groups and the same, there is the possibility that the quality level is the same as that of previous levels, it's just that the information collection systems are catching more of this discontent.
Now, you are stuck. Both sound good, but you need to take a decision one way or the other. This is where data collection from user systems works good. One of the first items you should be capturing is where information about whether the user application has closed normally, or whether the application has closed abnormally (such as in a crash or when the user was forced to terminate the application after it was hanging) and also capture this information for operating systems that are supported by the application. Further, you would need to do this for the different features and dialogs that are present in the application. Once you have been capturing this information, there is a lot you can do in terms of determining how often a feature or the entire application crashes in the hands of the users, where does the crash happen (this will need a lot of development effort though to determine what causes the crash in the application). This will also help to determine whether the frequency of crash is more than in previous versions of the application release.
Once you have such a data and the data has checked out to be accurate in some respects (for example, if your testing team is getting crashes in reasonably similar areas, then it helps to confirm this data to a large extent), you can make product level decisions. If that means that you need to spend some time on product stability and quality, then you need to do so; otherwise if the quality level seems fine, then you know that the information you are getting from the customers needs to be handled through regular support mechanisms and does not need the development team to spend extra effort.

Analytics - Measuring data relating to user information - Part 6

This is a series of posts that talk about the use of analytics in your software application. My experience is more in the nature of desktop applications, but a lot of what has been written in the posts earlier is also related to analytics for web application; there may be some differences, but the need for analytics is the same, and the decisions that can be made on the basis of analytics are the same. Some of the guidelines and warnings are also the same; in short, whether you are working on desktop applications or web applications, the tools may be different, but there is a strong need to ensure that you have designed a strategy for the same, and not doing this on an adhoc basis. In the previous post (Analytics - Measuring data relating to user information - Part 5), I talked about a problem where the team had made a strategy to collect data, but there were not enough people to actually analyze the data and take decisions based on such decisions.
However, there are some pitfalls when it comes to analytics, and taking decisions based on that. There is a joke about a person who would scream for data for every decision, whenever there would be a need for any decision or the planning for taking some decisions, there would be a hunt for data, and if the data was not present, then there were high chances that the team would be sent for such data. This is a joke, but I have seen managers who get too data-oriented. This may be anathema to those who are firm proponents of analytics, but there can be 2 problems with an analytics oriented approach.
- The data may be incorrect
- There may be so much emphasis on data, that it crosses a limit and common sense is lost

Sometimes these 2 problems can also intersect, but let's take each of them separately. You cannot just wish for data to happen - this is a very obvious statement, but it goes to the heart of the problem. We had a situation whereby we were collecting some data sent by a particular dialog in the application, and the data was coming in beautifully. A full release went by, and nobody thought much of the code in the particular function. However, in the next release, there was a defect in that section of the dialog that also affected the data that was collected, and the developer who was debugging that area came across something puzzling. It turned out that the data collection code did not enter one of the areas in which the application went into, which we speculated was around 15% of the time in customer interactions, but we had no real data. The net result was that we understood that our data for that particular dialog was understated by a percentage, but we did not know that particular percentage accurately. Hence, any decision that we made on studying the data from that dialog had a margin of error that was unacceptable. We reviewed the test cases and their execution from that particular time when the code was being written, and realized that because of a paucity in time, the testing for this particular part was not done as it should have been. The learning from all this was that data could be incorrect even with the best of efforts. And this takes us to the next para, although not directly.
Basing business decisions on data analysis can be great if you have the correct data, and can be suicidal if your data is incorrect. Further, when important decisions are being taken, it is important that there be some sort of confirmation, or that data is used to confirm some decision rather than being the driver of the decision making. So, suppose the business end of the application wants to run a campaign based on their observing of the market information they are getting, analytics could be of great help in confirming some of the assumptions that the team is making as a part of this decision making. But, using only analytics as the base on which to make decisions, or creating an environment for the same is not recommended.
Even when collecting data, there should be a thorough analysis of the data and the data collection methods to ensure that the data that is collected is correct; in fact I had a colleague who was in favor of analytics but had also been burnt before. His advice was simple - when you are getting data from analytics, assume that the data is wrong and then prove that it is right and then use it.

Read the next post in this series - Measuring data relating to user information - Part 7

Analytics - Measuring data relating to user information - Part 5

This is a series of posts relating to the measurement of user information and analyzing the data related to that user information. In a previous series of posts, we have looked at some examples of data measurement, and how to use this data analysis for decision making. In the previous post (Measuring data relating to user information - Part 4), I looked at the other side of gathering data. One can go overboard in collecting data and start collecting information which would get the product and the organization in legal trouble and face protests from consumers. One always need to ensure that the data being collected has been cleared by the legal team or others authorized to ensure that the data collection does not go beyond what has been legally permitted.
This post will cover a common experience that people face when in the business of collecting user information and then trying to make some sense out of it. Even large companies face a similar problem - the problem of not doing anything based on this data. Seems very strange, but this happens a lot. Recently I had a discussion with an analytics consultant. He has been in the field for a decade and a half now, working for organizations and working independently. The biggest problem he faced was of getting people to commit for either getting enough resources to do the data collection, or even more, of analyzing this data to make informed decisions.
I have seen this problem myself. The software applications that we worked on had a large amount of instrumentation done in order to collect data on many different parameters. This was done after careful design by a team comprising the product manager, developers and testers, with design of what to do with the data once it is collected after the product has been released. However, it should be well understood as a part of the design for analytics that there needs to be effort to analyse the data and make sense out of the data. And this is where the biggest problem was. Resourcing was always a constraint, since analytics was never a priority over features for the product, and unless there is a change in this attitude, things will never change.
So, what used to happen ? All the data would be collected, would be a huge database of information that we were collecting for each version (and which would grow over a period of time while we made additions to the data collection techniques for the features in the application), but at the most, we would take one set of data and try to make sense out of that.
The team was also able to see this, and hence, the next time a discussion would be called for trying to to figure out what more data needs to be collected, what are some of the reports that would be useful, there would be less interest from the team in trying to be involved in the analytics for the ongoing version. This was getting to be a big problem, but there was no clear solution to this. The pressure of resourcing vs. feature work was not going to go away, and unless analytics was seen on priority with other feature work, things would not really change. And this is where teams and especially their managers need to be more immersed in what all can be done through analytics, primarily the advantages for business decision making.

Read more about this in the next post. (Analytics - Measuring data related to user information - Part 6)

Analytics - Measuring data relating to user information - Part 4

This is a series of posts that focus on analysis and its importance in today's world. In the previous post (Measuring data relating to user information - Part 3), I talked about using the data from previous versions of the software to determine trends, and to use those trends to make major business decisions, and also use this data to complement trends seen from other sources of information.
In today's world, analytics can play a major role. I still remember an article (read the article) which described the power of analysis in today's world, and how it can ever surprise non-industry people. The same article also highlighted the power and problems posed by Analytics in terms of privacy problems. If you read this article, you would be whacked by what data analysis can reveal, and also be shocked to some degree about what data reveals and whether you are comfortable with this kind of information about you being deducted.
And this is the main content of this post. If you are starting to collect data about your customers from within your software, you need to be sure that you are not overdoing it. Once you start designing what all data you are going to collect, you need to ensure that there are hard lines that set the boundaries for the data you are collecting. If you are using a component, you need to ensure that the component respects the same kind of data privacy constraints that you are using.
This might seem like going too excessive, but keep in mind that privacy is a big deal. Most software development teams are not equipped to determine as to what is proper or not. This was brought painfully clear to me when we let the development team design the analytics capturing process, including all the information that was supposed to be captured, and then, when we met the legal team, they junked more than 25% of the data that we wanted to capture. We did not like it, but there are certain boundaries that are required. The opposite is not worth talking about. You could go beyond the privacy guidelines and even implement them in your product and release them, and then there is a chance that somebody detects that you are capturing some information that is deemed as personal, and you are stuck. So, it is always recommended that once you get into the area of capturing information for analytics, make sure that it has been confirmed by somebody who is a privacy expert, which could be a legal person, or could be somebody else.
There are even more worries, especially when it comes to your product selling across geographies. A country or region may have different privacy standards when it comes to collecting information from the user's machine (for example, the European Union has far tighter guidelines when it comes to privacy) and you need to ensure that you are not falling foul of one region by following the standards of another. For example, a privacy guideline of a region would be to insist that the users know about the information that it to be collected, as well as have given their permissions for the same.
But, this is not to say that you should start getting scared about capturing user information. If you have a verified system of guidelines and are following those, you should not be worried about this. Make sure that you are doing your best to capture the relevant information, and you can learn far more about your users preference than you expected, which will help you make the best decisions about your product.

Read more in the next post (Measuring data relating to user information - Part 5)

Analytics - Measuring data relating to user information - Part 3

This is a series dealing with analytics, and the advantages that it brings to the product team. However, any movement into the area of analytics requires a lot of careful thought, and needs time to be spent on the design of a strategy around Analytics. You cannot just say that you want analytics, and put in place some strategy. But when you do get your Analytics strategy right, there are a lot of benefits that are possible. In the previous post (Analytics - measuring data related to user information - Part 2), I talked about a scenario where a team wants to find out the video formats being used by its users in the application, and the benefits of making decisions based on this knowledge rather than making guesses (which may be right, or could be wrong). In this post, I will write more about the usage of analytics while making decisions.

Consider the previous post, which talked about which video formats are popular. However, consider that you need to make decisions about the future, which means that you need to do much more analysis about the data you are getting. So, if you have been in the game for many versions now, don't just look at data for the previous release. Instead, if you have been gathering such data for the past many releases, you need to make an effort to review this data for the past many releases in order to figure out the best possible method ahead. So, even though in the last post, we only reviewed the proportion of video formats that were in use, a better analysis would have looked at the proportion of video formats that the users have been using in the past few versions.

Over a period of time, such analysis would, in most cases, reveal trends that would be useful for the designers of the product, as well as the product managers to know. Till you would have done such an analysis, the way for you to learn such data would be by looking at industry data as well as research done in the forms of surveys and information from users through other means, but all of this is indirect data. Analysis of analytics data allows you to get confirmation on such information, and can help you make decisions that is also backed by hard data. 

A possible example that shows the value of such analysis would be about the usage of mobile devices. So, the product manager would have seen that there is a higher trend worldwide about using mobile devices for capturing such data, and then look at analysis of the data from the past many versions that talks about the source of the videos being used by the consumers of the applications. Consider a case where the trend shows a movement towards mobile devices being the source, but the trend is slow, only going up from around 12 % to 16% in the past 2 years. The question in front of the product manager was about diverting resources to producing a mobile version of the application, but that would require a large amount of code change and architectural and design work, and hence would have an impact on the current release. The other option would be to plan for a mobile version only in the next release, which would have a lower impact on the current release. Based on this data, the Product Manager might decide that although there is a lot of attractiveness in terms of having a mobile version, the data does not suggest that there is an emergent need to create a mobile version in the current release, especially with the costs of doing so. Instead, one can wait for such a release.

Taking such a decision is critical for the application, but being able to take such a decision without having data on consumer usage would mean a decision that is more like a guess-estimate, where some information tells that you can you take a decision, but the amount of data that you have is not adequate to produce a high level of comfort in taking such a decision.

We will continue this series on the usage of Analytics in the next post on this series.

Analytics - Measuring data relating to user information - Part 2

In the first part of this series (measuring data related to users - Part 1), I started out by outlining more details about what analytics is, what is the kind of information that can be captured from users, some kind of information that should not be captured based on privacy guidelines, and what you can do with this kind of information. In this post, I will continue more on this line and provide some more examples of what can be done (the purpose of this series of posts is to describe more about what can be done with analytics through some real life examples that lets you know what to do through analytics).
Let us take the example that we use a lot, a greeting card application that allows the user to use their own photos or images in addition to standard greeting card background photos, allows them to use their own audio and videos, or lets them record them same from the camera and microphone on their computers, and also allows them to add their own text of greetings. The final collected greeting can be sent via email, or through social networks.
Now, the application designers are trying to figure out tweak related to the videos that users upload from their own machines. There are numerous video formats that users can be having with them, since there are many different capture devices. You could be shooting the small video clip using a mobile phone (that too can have a different video format depending on the manufactures of the mobile phone), you could have shot the video using a tablet, could have shot it using a still camera, or have shot it using a video camera.
The size of the video that has been shot depends on the shooting device, depends on whether the user has reduced the resolution of the video in order to reduce the size, or in some cases, decided to use the same video that would have been uploaded to Youtube (which means that the video would have been converted to a FLV video format). Now, for most of these formats, these videos cannot be used just like that. Coders / decoders need to be user for this purpose, and even though there are some open source solutions, there would be commercial software that could be used for this purpose.
The decision about whether to use open source or commercial software could depend on the number of users who would be using such a software. So, as part of the data gathering in previous versions of the software, it could be determined as to which are the video formats that users are using, and then based on this data collection, the proportion of video formats used can be determined. If it turns out that the number of users using a particular format is more than a certain proportion, the product team would determine that it would make sense to use the commercial video encoder/decoder rather than use the open source one. The advantage that users get out of a better software component be greater than the cost advantage of using an open source software. But unless you are getting such information through analytics, any decision you take would be flawed, based on a hunch rather than information.

Read the next post in this series (Measuring data related to user information - Part 3)

Analytics - Measuring data relating to user information - Part 1

For any desktop product, capturing data relating to the computer systems of their user base is very important. For those who are not conversant with the idea of analytics or the necessity of capturing such information, it would make sense to ensure that analytics forms a part of their overall product strategy; but before that, it is necessary for them to understand what is analytics and why it should play a role in the overall strategy of a product team.
As always, I will try to use layman terms in this post rather than use technical jargon. So, analytics is very simple - it means capturing information about their consumers (for example, this could mean that you capture information about the number of times the customer has launched the application, you can capture information about the processors of the customer machine, the Operating system version, whether they are using Windows or Mac, and so on). Of course, there are privacy laws that are in place, and you need to ensure that you are not capturing information that allows your customers to be identified (for example, if you are capturing information about the folder in which the user is storing data, this may in turn be storing information in a folder that captures the user name; further, most advocates of privacy laws would be very hesitant about capturing the serial number of the user).
Capturing of such information is possible through the use of code and functionality, which captures events during the user interaction with the software application (such as the number of times that the user has launched a dialog window, the times that the user keeps a particular functionality open, the user workflow during the process flow, and so on). One prime use that I have seen is about capturing the number of times a dialog shut down improperly (crashed) and the same for the number of time the software application crashed as well.
Once the designers of the application decide on the information that needs to be captured, code has to be written for passing the same information through the internet to a tracking mechanism on the website of the software maker. Now, it needs to be kept in mind that this information is sent to a database to the proper tables, and further, keep in mind that such information can be very voluminous. This may be a few KB's of information for every user, but when you start dealing with thousands of users, or even millions of customers (such as for large application such as Microsoft Office and Adobe Reader), the amount of data can be huge and the database needs to be ready to handle such data.
In addition to the design of the database of capturing the information, the next step needs to be related to processing of this captured information. So, even though you have captured a large amount of data, there needs to be effort put in for analysing such data. The ideal set of people for analyzing such data is typically the people working on relevant features. So, suppose the data analysis for a particular feature needs to happen, the analysis should be done by the same team that is working on the feature, since the team has the expertise to figure out what the information is, and also what to do with the data.

Read more about this in the next post - Measuring data related to user information -  Part 2