Communication: Informing people when there is a problem, and when they depend on you ..

You are providing a service to some clients, and they have come to totally depend on you. Like the post before this one, if you are providing analytics information to external teams, you have a lot of responsibility on your head. They would come to depend on you and if you are doing a good job for them, more and more, they will not think about any problems at your end and will depend on you to inform them if you are running into any problem. And that places an awful lot of responsibility on your end. Towards this purpose, you have to have a strategy to figure out what to do if something goes wrong at your end, and informing your clients quickly should be a part of such a strategy,
We recently had a big shock when it came to our analytics program, where we were using an external component. We were integrating such a component, and passing it parameters from within our code. What this ended up doing was passing data from our user base to the database maintained by the external team, and they would let us mine this information through a web front end, and they would also help us if we needed some new queries or new reports.
All this was perfect, and we did not bother anymore to check whether there could be a problem of any kind (so part of the problem was at our end before of a lack of checking for variables that could impact the program). We were looking at some data pursuant to the launch of a new version of the application, and in the second to fourth week of the launch, we saw data that indicated that the number of people using the older version of the program had reduced. This enthused the Product Management and other senior stakeholders, and a decision was made to target the older users through an email campaign sent to those users who were on our email list and had purchased previous versions of the program. This campaign would build on this conversion and offer some benefits if they did an upgrade, the idea being that if there are people doing an upgrade, it would help to push even more people.
This campaign was made ready in record time, and pushed live for a week. Like any other campaign, this program had some large costs because of the resources used, and the infrastructure cost for tracking of the email campaign, including measuring the conversion numbers. However, at the end of the week, it was very surprising that the conversion rate did not show any uptick; and in a coincidence, a day later we received an email from the analytics team that they were suspecting some configuration issues on their side, and they have probed, and finally figured out that some data had been lost, which also overlapped the weeks that made us decide that people using the older versions had dropped.
This was like a big blow, and since the analytics team was part of the same organization, we bit back the strong words, put in the issue for being more careful the next time that we did a campaign based on data. However, senior stakeholders did not hold back and had an interesting discussion with the head of the analytics team about not sending out a message to the various client teams when they suspected some data issues. The analytics team wanted to be sure before sending out a message to the client teams, but it was a learning to all of us. Since none of us really knows when another team is in a critical position and are dependent on us, as soon as we run into some problem, it is imperative to quantify the risk position and send it out the client teams that are dependent on us. This helps them avoid business decisions that are impacted if the data turns out to be problematic; even if it was a false alarm, there is some loss, but in general, the loss might be less than making a decision based on data that is incorrect.

Analytics - Measuring data relating to user information - Part 5

This is a series of posts relating to the measurement of user information and analyzing the data related to that user information. In a previous series of posts, we have looked at some examples of data measurement, and how to use this data analysis for decision making. In the previous post (Measuring data relating to user information - Part 4), I looked at the other side of gathering data. One can go overboard in collecting data and start collecting information which would get the product and the organization in legal trouble and face protests from consumers. One always need to ensure that the data being collected has been cleared by the legal team or others authorized to ensure that the data collection does not go beyond what has been legally permitted.
This post will cover a common experience that people face when in the business of collecting user information and then trying to make some sense out of it. Even large companies face a similar problem - the problem of not doing anything based on this data. Seems very strange, but this happens a lot. Recently I had a discussion with an analytics consultant. He has been in the field for a decade and a half now, working for organizations and working independently. The biggest problem he faced was of getting people to commit for either getting enough resources to do the data collection, or even more, of analyzing this data to make informed decisions.
I have seen this problem myself. The software applications that we worked on had a large amount of instrumentation done in order to collect data on many different parameters. This was done after careful design by a team comprising the product manager, developers and testers, with design of what to do with the data once it is collected after the product has been released. However, it should be well understood as a part of the design for analytics that there needs to be effort to analyse the data and make sense out of the data. And this is where the biggest problem was. Resourcing was always a constraint, since analytics was never a priority over features for the product, and unless there is a change in this attitude, things will never change.
So, what used to happen ? All the data would be collected, would be a huge database of information that we were collecting for each version (and which would grow over a period of time while we made additions to the data collection techniques for the features in the application), but at the most, we would take one set of data and try to make sense out of that.
The team was also able to see this, and hence, the next time a discussion would be called for trying to to figure out what more data needs to be collected, what are some of the reports that would be useful, there would be less interest from the team in trying to be involved in the analytics for the ongoing version. This was getting to be a big problem, but there was no clear solution to this. The pressure of resourcing vs. feature work was not going to go away, and unless analytics was seen on priority with other feature work, things would not really change. And this is where teams and especially their managers need to be more immersed in what all can be done through analytics, primarily the advantages for business decision making.

Read more about this in the next post. (Analytics - Measuring data related to user information - Part 6)

Analytics - Measuring data relating to user information - Part 4

This is a series of posts that focus on analysis and its importance in today's world. In the previous post (Measuring data relating to user information - Part 3), I talked about using the data from previous versions of the software to determine trends, and to use those trends to make major business decisions, and also use this data to complement trends seen from other sources of information.
In today's world, analytics can play a major role. I still remember an article (read the article) which described the power of analysis in today's world, and how it can ever surprise non-industry people. The same article also highlighted the power and problems posed by Analytics in terms of privacy problems. If you read this article, you would be whacked by what data analysis can reveal, and also be shocked to some degree about what data reveals and whether you are comfortable with this kind of information about you being deducted.
And this is the main content of this post. If you are starting to collect data about your customers from within your software, you need to be sure that you are not overdoing it. Once you start designing what all data you are going to collect, you need to ensure that there are hard lines that set the boundaries for the data you are collecting. If you are using a component, you need to ensure that the component respects the same kind of data privacy constraints that you are using.
This might seem like going too excessive, but keep in mind that privacy is a big deal. Most software development teams are not equipped to determine as to what is proper or not. This was brought painfully clear to me when we let the development team design the analytics capturing process, including all the information that was supposed to be captured, and then, when we met the legal team, they junked more than 25% of the data that we wanted to capture. We did not like it, but there are certain boundaries that are required. The opposite is not worth talking about. You could go beyond the privacy guidelines and even implement them in your product and release them, and then there is a chance that somebody detects that you are capturing some information that is deemed as personal, and you are stuck. So, it is always recommended that once you get into the area of capturing information for analytics, make sure that it has been confirmed by somebody who is a privacy expert, which could be a legal person, or could be somebody else.
There are even more worries, especially when it comes to your product selling across geographies. A country or region may have different privacy standards when it comes to collecting information from the user's machine (for example, the European Union has far tighter guidelines when it comes to privacy) and you need to ensure that you are not falling foul of one region by following the standards of another. For example, a privacy guideline of a region would be to insist that the users know about the information that it to be collected, as well as have given their permissions for the same.
But, this is not to say that you should start getting scared about capturing user information. If you have a verified system of guidelines and are following those, you should not be worried about this. Make sure that you are doing your best to capture the relevant information, and you can learn far more about your users preference than you expected, which will help you make the best decisions about your product.

Read more in the next post (Measuring data relating to user information - Part 5)

What are Address Binding, Dynamic Loading and Dynamic Linking?

In this article we shall discuss about three interrelated concepts namely address binding, dynamic loading and dynamic linking.

1. Address Binding: 

- There are two types of addresses for the computer memory. 

- These are called the physical address and the logical address. 

- A physical memory location is allocated to a logical pointer by address binding process.

- This is actually nothing but associating the physical address and the logical address with each other. 

- Sometimes logical address is also referred to as the virtual address. 

- This concept is an important part of the memory management. 

- Operating system is responsible for carrying out address binding on behalf of the applications and programs that need an access to the memory. 

- A program cannot be executed without bringing it to the main memory. 

- The instructions of the program have to be bound to right address spaces in the physical memory. 

- Address binding is simply a scheme for performing this job. 

- It can be thought of as something similar to address mapping. 

- Address binding can be carried out at any of the following times:

Ø  Compile time

Ø  Loading time

Ø  Execution time

- In execution time binding, whenever the program requires access to memory, it has to go through a register called the relocation register and is similar to the base register. 

- Then the offset is added. 

- But in binding during the loading time, same thing is done but every time this register need not be evaluated. 

- The addresses are mapped at the time of loading the program in to the memory. 

- If there is a change in the base address, the whole program has to be reloaded.

2. Dynamic Loading: 

- This mechanism is very useful for a program as it helps it do the following things:

Ø  Loading library in to the main memory.

Ø  Retrieving the address of the variables and routines that are contained in the library.

Ø  Accessing those variables and executing those routines.

Ø  Unloading the library.

- Dynamic loading is very much different from the load time linking and static linking. 

- Dynamic loading allows a system to start up even of the libraries are absent. - It also helps in discovering the absent libraries and then gaining the additional functionality. 

- Dynamic loading is a very transparent process since it is the operating system that handles it. 

- Main advantages are firstly, it helps in fixing the patches at once without having the need for re-linking them and secondly, it provides protection to the libraries against modification that is not authorized. 

- Dynamic loading find its major use in the implementation of the software plugins.

- It is also used in the implementation of the computer programs where requisite functionality is supplied by the different libraries and user has the freedom to select the libraries he/ she wishes to provide.

3. Dynamic Linking: 

- This is an important part of the binding process. 

- The purpose of the dynamic linking is resolving the references or symbols and links to the library modules. 

- This process is carried out by a linker program. 

- This programs searches for a set of library modules in some given sequence. 

- This process takes place during the creation of the executable file. 

- The resolved references may be addresses of the jump calls and the routines. - These may in different modules or in the main program.

- Dynamic linking resolves them in to relocatable address or fixed address through allocation of the memory to each of the memory segment of the referenced module. 

Analytics - Measuring data relating to user information - Part 3

This is a series dealing with analytics, and the advantages that it brings to the product team. However, any movement into the area of analytics requires a lot of careful thought, and needs time to be spent on the design of a strategy around Analytics. You cannot just say that you want analytics, and put in place some strategy. But when you do get your Analytics strategy right, there are a lot of benefits that are possible. In the previous post (Analytics - measuring data related to user information - Part 2), I talked about a scenario where a team wants to find out the video formats being used by its users in the application, and the benefits of making decisions based on this knowledge rather than making guesses (which may be right, or could be wrong). In this post, I will write more about the usage of analytics while making decisions.

Consider the previous post, which talked about which video formats are popular. However, consider that you need to make decisions about the future, which means that you need to do much more analysis about the data you are getting. So, if you have been in the game for many versions now, don't just look at data for the previous release. Instead, if you have been gathering such data for the past many releases, you need to make an effort to review this data for the past many releases in order to figure out the best possible method ahead. So, even though in the last post, we only reviewed the proportion of video formats that were in use, a better analysis would have looked at the proportion of video formats that the users have been using in the past few versions.

Over a period of time, such analysis would, in most cases, reveal trends that would be useful for the designers of the product, as well as the product managers to know. Till you would have done such an analysis, the way for you to learn such data would be by looking at industry data as well as research done in the forms of surveys and information from users through other means, but all of this is indirect data. Analysis of analytics data allows you to get confirmation on such information, and can help you make decisions that is also backed by hard data. 

A possible example that shows the value of such analysis would be about the usage of mobile devices. So, the product manager would have seen that there is a higher trend worldwide about using mobile devices for capturing such data, and then look at analysis of the data from the past many versions that talks about the source of the videos being used by the consumers of the applications. Consider a case where the trend shows a movement towards mobile devices being the source, but the trend is slow, only going up from around 12 % to 16% in the past 2 years. The question in front of the product manager was about diverting resources to producing a mobile version of the application, but that would require a large amount of code change and architectural and design work, and hence would have an impact on the current release. The other option would be to plan for a mobile version only in the next release, which would have a lower impact on the current release. Based on this data, the Product Manager might decide that although there is a lot of attractiveness in terms of having a mobile version, the data does not suggest that there is an emergent need to create a mobile version in the current release, especially with the costs of doing so. Instead, one can wait for such a release.

Taking such a decision is critical for the application, but being able to take such a decision without having data on consumer usage would mean a decision that is more like a guess-estimate, where some information tells that you can you take a decision, but the amount of data that you have is not adequate to produce a high level of comfort in taking such a decision.

We will continue this series on the usage of Analytics in the next post on this series.

Define the Virtual Memory technique?

Modern operating systems come with multitasking kernels. These multitasking kernels often run in to the problems related to memory management. Physical memory does not suffice for them to execute the tasks assigned to them because of being fragmented. So they have to take some additional from the secondary memory. But they cannot use this memory directly. Virtual memory offers a solution to this problem. 

What is Virtual Memory technique?

- Using this technique makes the fragmented main memory available to the kernels as a contiguous main memory. 

- Since it is really not the main memory but just appears to be, it has been named as the virtual memory and this technique is called the virtual memory technique. 

- Since, it helps in managing the memory, it is essential a memory management technique. 

- The main storage gets fragmented because of many programming and processing problems. 

- The main memory available to the processes and the tasks is virtualized by the virtual memory technique and then it appears to the process as a contiguous memory location. 

- This memory is a global address space. 

- Virtual address spaces such as these are managed by the operating system. 

- The real memory is assigned to the virtual memory by the operating system itself. 

- The virtual addresses of the allocated virtual address spaces are translated in to the physical addresses automatically by the CPU. 

- It achieves this with the help of some memory management hardware specially designed for this purpose. 

- The processes continue to execute uninterrupted as long as this hardware properly translates the virtual addresses in to real memory addresses properly. 

- If it fails in doing so at any point of time, the execution comes to a halt and the control is transferred to the operating system. 

- The duty of the operating system now is to move the requested memory page to the main memory from the backing store. 

- Once done with this, it then returns the control again to the process that was interrupted. 

- It greatly simplifies the whole execution process. 

- Even if the application would require more data or code that would fit in real memory, it does not have to be moved to and fro between the backing store and the real memory. 

- Furthermore, this technique also offers protection to the processes that are provided distinct address spaces by the isolation of the memory allocate to them from other tasks.

- Application programming has been made a lot easier with the help of the virtual memory technique since it hides the fragmentation defects of the real memory. 

- The burden of memory hierarchy management is delegated to the kernel which eliminates the need for the explicit handling of the overlays by the program. 

- Thus each process can execute in an address space that is dedicated to it. 

- The need for relocating the code of the program is obviated along with using relative addressing for accessing the memory. 

- The concept of virtual memory was generalized and eventually named as memory virtualization. 

- Gradually, the virtual memory has become an inseparable part of the architecture of the modern computers. 

- For implementing it, dedicated hardware support is absolutely necessary. 

- This hardware is built in to the CPU in some sort of memory management hardware. - If required for boosting the performance of the virtual memory, some virtual machines and emulators may employ some additional hardware support. 

- The older mainframe computers did not have any support for the virtual memory concept. 

- In virtual memory technique, each program can solely access the virtual memory.

Analytics - Measuring data relating to user information - Part 2

In the first part of this series (measuring data related to users - Part 1), I started out by outlining more details about what analytics is, what is the kind of information that can be captured from users, some kind of information that should not be captured based on privacy guidelines, and what you can do with this kind of information. In this post, I will continue more on this line and provide some more examples of what can be done (the purpose of this series of posts is to describe more about what can be done with analytics through some real life examples that lets you know what to do through analytics).
Let us take the example that we use a lot, a greeting card application that allows the user to use their own photos or images in addition to standard greeting card background photos, allows them to use their own audio and videos, or lets them record them same from the camera and microphone on their computers, and also allows them to add their own text of greetings. The final collected greeting can be sent via email, or through social networks.
Now, the application designers are trying to figure out tweak related to the videos that users upload from their own machines. There are numerous video formats that users can be having with them, since there are many different capture devices. You could be shooting the small video clip using a mobile phone (that too can have a different video format depending on the manufactures of the mobile phone), you could have shot the video using a tablet, could have shot it using a still camera, or have shot it using a video camera.
The size of the video that has been shot depends on the shooting device, depends on whether the user has reduced the resolution of the video in order to reduce the size, or in some cases, decided to use the same video that would have been uploaded to Youtube (which means that the video would have been converted to a FLV video format). Now, for most of these formats, these videos cannot be used just like that. Coders / decoders need to be user for this purpose, and even though there are some open source solutions, there would be commercial software that could be used for this purpose.
The decision about whether to use open source or commercial software could depend on the number of users who would be using such a software. So, as part of the data gathering in previous versions of the software, it could be determined as to which are the video formats that users are using, and then based on this data collection, the proportion of video formats used can be determined. If it turns out that the number of users using a particular format is more than a certain proportion, the product team would determine that it would make sense to use the commercial video encoder/decoder rather than use the open source one. The advantage that users get out of a better software component be greater than the cost advantage of using an open source software. But unless you are getting such information through analytics, any decision you take would be flawed, based on a hunch rather than information.

Read the next post in this series (Measuring data related to user information - Part 3)