Supporting previous versions of the software (Part 6)

The previous post (Supporting previous versions of the software (Part 5)) talked about a team trying to make a decision about whether to drop support for a previous version of the software or not. If you are a user who is impacted by dropping support for the version of the software you are using, you would think that the team has taken such a decision without thinking about users. However, such decisions are not taken without a detailed discussion, and maybe after thinking about this decision at multiple levels before the final decision on dropping support is taken. However, it can be the exact reverse decision that could happen. With some serious defects having risen, the team has proposed that these cannot be fixed easily and also because these problems are for an older version; however, when a decision is finally taken, the decision is the exact opposite. The support for the previous version is not removed, and infact, the team is told that with these severe defects, there needs to be a fix provided to users.
With this decision, it can provide quite a challenge for the team to provide the dot release / patch for the release, but not impossible and hence they have to plan for it. There are a number of parameters that need to be considered, evaluated and a solution found for all of these. Some of these are (this is not an exhaustive list, if you see more parameters, please let me know through comments):
- Whether to release a dot or patch: What is the difference ? The terminology could vary, but typically consider 2 different kinds of releases ? One is where you have a simple small executable that just updates the few files required for the fix, while the other is a complete release that essentially replaces the application that they have on their machine. The reasons for deciding which method to use is also complex; it depends on the number of files to be updated, as well as the actual mechanics of issues such as upgrades, dependencies, etc. This decision is typically an engineering decision.
- Number of resources to be used for the release: In most cases, teams do not get additional resources for handling such updates unless they are doing this on a regular basis. So, some amount of work estimation needs to be done to determine the number of resources needed, as well as the schedule on which they will be needed.
- Support from external teams: For any product release, there is support needed from additional teams such as testing, installer and release, documentation, etc. Even if there is a small update to be provided, there is a need to ensure that these external teams are in place and committed to the effort, This can be a challenge sometimes.
- The timeframe for this effort needs to be decided: This can be a challenge sometimes. Typically a dot or a patch needs to be released pretty quickly, and there is a need to optimize to ensure that the existing engineering work for the current version is not impacted. This schedule can be a challenge, since the priority will remain the work for the existing release, and yet there needs to be an update.
- Update problems / engineering issues: Whenever an update / patch / dot release for a previous release needs to be released, it needs to be checked with the application installed on the user's machine. This includes ensuring that the application continues to work fine with the update, including the standard checks. This means that the team needs to install previous versions of the application and even replicate the operating system version that some of the customers might be having.
- Informing the users through support teams, forums and so on. There needs to be a campaign to inform users about the impending update release and ensure that feedback needs to be taken quickly when the update is released.

Making sure that all methods are used to inform users about patches or dot releases ..

A patch or dot release is typically done by a product team for defects or some improvements that the team would have made over the past period of time. Most people are familiar with the concept of Service Packs released by Microsoft for Office, Windows, and other tools that it has. So a service pack would typically take a number of improvements and defect fixes and make them available for users to download and install on their machines. A patch release in turn is meant for defect fixes that have been deemed critical enough to provide a fix to users. A service pack is typically meant to be a large patch, combining a large number of fixes and optimizations for the application. A dot release is a slightly different animal. Typically, in my experience, a dot release is when the code of the previously released version of the application is taken, the fixes are all applied to the source codes, and a new version of the application is released. This new version of the application is now ready for use by customers; if the released version of the application was 7.0, now the team can take the 7.0 code, add the fixes to the code, and then release a version called 7.1. This new version, with the fixes, replaces the previous version 7.0, and is the version that is available to consumers.
So, now that you have a new version, or a patch, or a service pack, how do you get this to your users. You would want all your users to take the new version, for multiple reasons:
- If you have fixes for problems available in this release, then with customers adopting such a release, there will be lesser need for support for such issues.
- If you get all your customers to adopt such a release, then it is easier to provide technical support. If you have customers having different versions of software such as 7.0 or 7.1, then it gets more complicated to provide support for such cases. In such cases, it would be easier if all customers have the 7.1 version.
- When you provide a new version post 7.1, it would be easier if you have most or all customers on the version 7.1. When a new version 7.2 is releases, there would be need to provide support for customers to upgrade from versions 7.0 or 7.1. When you have software such as Windows or Adobe Reader, on which there can be a number of service packs or patches released, this kind of upgrade support can make things difficult.

Now that we have the background, how can you increase the chances of more people taking the new patch or release ?
- Spread the information about new releases on the company site. As an example, if you take a large company such as Adobe, it has a section of the site called Updates (link). On this site, all the downloads and patches for different products released by the company is listed. Other large companies also follow a similar practice.
- Use social networking to spread the word. Most popular products have their presence on Facebook or Twitter, and companies should release information on these updates along with details of the major changes, and how to install the application.
- Seed support services such as consumer forums, user forums, customer support services, with information about the updates; for customer support, calls on major defects that are fixed with these updates should tell customers about the same.
- Nowadays, most major products have an update service, and the update should provide information to the customers about the desirability of the update and why the customer should do an update of the product.
- And now for a more forceful approach. Over a period of time, in order to ensure that everybody was on the update, product teams have actually terminated support for those users who are not using the latest patch. As an example, Microsoft a couple of years back stopped providing support for those users who were not on the Service Pack 2 of Windows XP.
All these techniques are important to increase the proliferation of updates among the application users.

Dot / Patch release: Estimating the features where there is a change required

One of the previous posts I wrote about was that of a Dot / Patch release (Estimation of dot / patch release) where there was an effort to show what are some of the reasons for a dot release or a patch release and the broad level points about how to estimate the amount of overall effort needed as well as the generation of schedule needed for the release. This needs to be done even if you already have a hard date for the release of the patch / dot release, since if the generated schedule is going far beyond the required date, then there is something wrong which would need to be solved (possibly by adding more resources, although that cannot be done in every case).
This post talks about actually doing an estimation of the changes required by working on the different features to see the overall impact of the change. Consider a case where a dot release has to be made for the application, with a change made in one of the features for a major defect. Now, the dot release has to be made in a couple of months, with the exact release date being decided based on the estimates for the different areas of the release (development effort, testing, localization, etc). One of the starting points for this estimation effort is figuring the effort needed for development, and for that, there needs to more detailed investigation of what areas need development effort.
How do you do this ? The first and most important starting point is to ensure that you have an experienced developer to do the investigation and estimate. The actual effort can be done by somebody in the development team, but the investigation should be done by a senior and experienced developer. How do you start:
- First, make sure that the developer has been given details of the change needed in terms of requirements
- Give the developer some time to discuss with other team members about the areas which are impacted.
If this is a core area, then the developer would need to go across the application to determine the impact. For example, there may be a need in the user security framework, and that would spin across all the application. If the change is localized to a specific area, then it is easier to make the effort estimate of the change. There is no great complexity in this area, as long as the developer does a thorough job.
- The senior developer should spend time with the actual developer who is going to work on the release and make sure that he has enough understanding of the changes required, and depending on the skill of the developer assigned, make sure that the correct estimate is given (which may included a required buffer).
- The developer also needs to ensure that he has spent time with people from the localization development team and the installer / release teams so that he has an idea of the amount of time needed from their side and can also include these in the effort estimate.
- The developer needs to spend a fair amount of time with the testing team so that they have a good understanding of all the changes that are going to happen in the release as well as the actual impact of the change including all the areas of the application that are impacted. 

Being able to estimate effort needed for smaller releases (patch / dot release)

One of the most difficult tasks to estimate are the smaller projects that a team needs to do. Consider a long project, the typical new version release of a software (for a typical long example, consider the schedule for the release of a new version of Microsoft Office - is it always typically more than 1 year); now even when the team is busy doing this release, there will be always be the need for doing a dot release or a patch at the same time (or even multiple such releases). Why would this happen ?
- There could be a security issue for which an urgent patch needs to be released,
- There could be some new feature that is meant to be pushed through before the next release
- Because the release schedule is long, defects that are found during the next few months are collected and released in an interim release that is then released
- And this is one of the most interesting reasons for a release - I know several people who will not take a new version of a Microsoft until a service pack has been released, since that is when it would have stabilized - so this puts pressure on a company to do a release within a few months

With all this, who does these interim releases ? For products where there is already a definite schedule of interim releases, a small sub-team can be setup to take care of these releases. Such a team will soon gain expertise in doing these smaller releases, and can work on the main release if there are no interim releases ongoing. However, for teams where there is no definite schedule of release of these interim releases, it does not make sense for a team to put dedicated folks to work on the product. Even more, in my experience, when people are assigned to such interim releases, there is a need to rotate the people working on such releases, since these are seen as essential but maintenance, not with the excitement of working for something new.
Now, once people are assigned to do the project, there is a need to figure out the schedule for such a release. However, in most of these cases, the end date is already fixed - the Product Manager already has a date in mind about when these releases need to be released. But, the team still need to define the estimate and a probable schedule in mind:
- Whether this be a dot release or a patch (a patch is a small set of files that is downloaded and can be installed, a dot release is typically the full installer with a few changes in the files). The advantage of having a dot release is that it can replace the original installer
- Define the change that is happening (this typically means the files and features that are being changed, and the estimated impact of this change)
- The amount of time needed for the development team to make the required changes
- The amount of time needed for the testing team to test the area and surrounding areas (this would also include the installer created for the release)
- Typically most products have language versions, and those need to be created and tested when a dot release or a patch is made. So, the estimate for the amount of time needed for these activities also need to be incorporated into the schedule.

Together, all of these can be woven into a schedule for the interim release. If it turns out that the deadline given is less than the projected schedule end, then you need to push back. For a small release, it is next to impossible to meet the project with the required quality principles, unless sufficient time is given. We once had a situation where a particular variation in the installer made a small error in the registry, which then prevented those who installed from being able to install another update, and there was a lot of backlash over that - we had to release another patch for that one.

Read more about the estimation done by a senior member of the development team.

Product Development - planning a minor / dot release - Challenges

In the last post, I had talked about why a minor (dot) release is needed, as well as some of the reasons as to why doing a dot release is an inconvenience. However, if the decision has been made to do a dot release, then it is necessary to understand the process of planning and executing a dot release, including some of the difficulties (challenges) that emerge in such a minor release.
What are some of the activities that need to be done ?
1. You need to finalize the changes that need to be a dot release. If this release is because of some known changes, then the changes need to be analysed, and the engineering response (and design) needs to be worked out.
2. If the changes are still unknown, for example, if your release is failing some security tests or some certification, then you need to figure out what can be done. If you take an example where the earlier release is failing some new certification norms (and for those who know how certification works, it can be a lot of effort to prepare the infrastructure and execute all the cases. As an example, you may need to take the help of tools for certification, and those tools may need an upgrade of memory. In other cases, the amount of testing required may be huge, and the actual calendar time needed may stretch the schedule of the dot release.
3. For companies where a lot of work is handled by support teams (configuration, build, release, internationalization, etc), the required overhead of handling all those teams and getting their support takes both budget allocation and time.
4. Too many minor releases causes cynicism in the market about the initial stability of the software release. For example, Microsoft releases many service packs for its software, and there are many people who do not migrate onto a newer version until they see 1 or 2 service packs because they would rather wait for some of the important bug fixes.
5. You run into issues where there are multiple releases for around the same version (with say versions 8.0, 8.1, 8.2, etc for a product). Once you have multiple versions, you get into support issues whereby issues are different for these versions, and support may be a nightmare.
6. Newer dot versions, in many cases need to be deployed to the retail channel (replacing the software already in the retail channel), and to the web store (including to many online software sellers who resell the software). All of these steps involve huge logistical nightmares and / or costs.
7. Branching strategies. Getting a dot (minor release) in process needs major configuration issues (getting branches in place, especially when work is also going for the next release), and takes a fair amount of explanation to both the Development and Quality teams.

Of course, if you folks can suggest more issues that make minor releases a challenge, please add comments.

Product Development - what is a minor / dot release

For those of you unacquainted with the concept of a minor release (I will also call it a Dot release from time to time), it is a release that you do after the main release has been done. If that did not make too much sense, let me try again ! Suppose you have released version 8.0 of Microsoft's Visual Studio to great fanfare, and with much expectation that this is 'the' release, perfect in all ways. Once you release 8.0, the overall grand plan would be to sell 8.0, and have the team working on developing 9.0 with no constraints.
Let me tell you a little secret. As you move closer to the release date of a product version, you (and in fact, most of the team members), start entering a time of partially restrained panic and impatience. It is also a normal decision that as you get closer to the release date, bugs that would have been fixed by the project team even a couple of months back will not be fixed now, unless they are deemed critical. The primary reason for this customer unfriendly move ? Simple Murphy's Law - 'If anything can go wrong, it will' ! Any bug fix has a cost - the fix needs to be reviewed, and then the quality team needs to test the impacted area of the bug. If something goes wrong with the bug fix earlier in the cycle, you have time to fix the impact. However, as you get closer to your release date, this buffer time is no longer available. As a result, almost every product team takes a call that as you get closer to the release date, bugs need to be more and more critical before they can be fixed. Thus, the product team may opt to not fix bugs that could potentially impact customers, but if the bug is found say a week before release, the bug would be classified based on whether you would stop the release for such a bug. Sadly, very few bugs make the criteria of being a release-stopper.
Once you have released version 8.0, there will be bugs that will come in from the field (including bugs that the product team had decided should not be fixed). Also, there may be other changes that are required in the release, and which cannot wait for version 9.0. And thus the stage is set for a generation of release of Visual Studio 8.1 (now you know where the term dot comes from, since this minor release is actually 8 'dot' 1). Such releases follow a few basic points:
1. Bugs (whether older bugs, or new bugs generated by customers) are evaluated as to whether they are deemed important enough to fix; these are the only bugs that are selected for inclusion in the dot
2. Typically, when a dot release is planned, the release supplants the earlier release. So, following the above example, new customers will get release 8.1 of Visual studio vs. release 8.0 that was released earlier
3. A dot will typically have all the activities of a full release, the only advantage being that the time frame is drastically reduced. A reverse is that the inter-group coordination that had more time available has to be compressed into a shorter time frame
4. A dot inconveniences everybody, since it takes much more attention to do everything in a shorter time period; further, it takes away resources (including team management attention) from working on the next release
5. Reaction time is much less than in a normal release

Product Development - Doing a patcher or a minor release

When doing the planning for a new version of the product (not typically applicable when creating a new product), it sometimes becomes necessary to consider the case of doing a patcher or a minor release (also known as dot release).
Picture this scenario - your product development team has already started planning for the features that will be implemented in the new cycle. They are trying to get an initial SWAG (literally means a Wild Ass Guess, but is typically used to define when experienced development managers and engineers do a rough guess of the amount of resources it will take to develop a given feature), are working with Product Management and User Design to get some more details of the initial feature requirements such that a more accurate estimate (but still rough estimate) is available. In the middle of this, if they are suddenly confronted with the need to plan for doing a patch or a dot (minor) release, it can take away a fair number of resources and affect the schedule for the next version.
So why does a team end up doing a patch or a minor release (dot release) ? Let's talk about a patcher first.
Reasons for doing a patcher:
- Easiest reason. After releasing the product, you discover a bug (through internal testing or from customers) that is liable to affect a number of customers. So, suppose you are doing a printing application, and you find that printing of Excel documents does not happen properly in common cases, this is something that cause you to do a patch. There is a high priority that a number of customers will get affected by such a problem.
- Crashes. A slightly more difficult situation where a number of customers have reported problems where the application suddenly crashes. On diagnosis, you find the root cause of the problem and decide that the risk of customers getting the crash is not very low, or there is a high risk of important data loss when the crash does happen. This is very important for certain classes of applications such as financial and enterprise applications.
- Customer / Tech support issues. There are a variety of issues that are not important enough to warrant a patcher on their own, but together they are causing problems with a high frequency of customer support issues and tech forums posts. Because all of these have a cost (and a lowered reputation because of many customer posts on forums is not easy to recover from), it is sometimes important to admit that there were mistakes made and release a patch.
- Device dependencies. Support you are a maker of a photo software, and many new cameras are released (or you discover problems with some existing cameras), it is important to project an image that you are responsive to developments in your field and are providing updates to your customers.
- Adding some important new operational functionality. This is a slightly tricky area since the recommendation is not implement new functionality in a patch, but if you have some important new functionality that needs to be implemented in as many existing customer apps as possible, then this is something that seems permissible. For example, if you get a new component that allows you to provide customers with a new and easier update mechanism, it makes sense to provide such a functionality through a patch available for wide download and use
- Dependencies on other software. Nowadays most large software applications also internally use other software components. For example, video players will use external codecs, CD burning software will use device burning codecs from Nero or Sonic, and many other such examples. If you find that your dependency is going to cause problems unless you install a newer version of the external component, a simple way is to incorporate such changes in a patcher.
- Competition. Your competition is going to release a new version of their software that gives them a competitive advantage, and your development process means that you cannot release a new version quickly enough. A workaround is to make the changes in a patch and make that available to your customers.

This post is turning out to be longer than I thought, so the process for deploying patches as well as the development of minor releases will be covered in the next post ..