Bug fixing: Ensuring that there is regular contact with outside users

Bug fixing is an integral part of the product development process, and unless the bug finding and bug fixing process starts tapering down near the end of the product development cycle, you are in deep trouble. Near the end of the cycle, is one of the most stressful time for the product development team. After all, when you are nearing the end of the development cycle, any new defect that comes through can cause huge problems for the team and for the schedule. If the defect was serious, then there are many issues that come to light. One needs to figure out whether the area was tested well enough before during the earlier parts of the schedule, as well as figure out what the risk was in terms of making the fix of the defect. If the fix can impact a large enough area, then the team might well want to not make the fix and take the defect. If a serious defect fix has been made near the end of the schedule, then there is a compromise that needs to be made in terms of the amount of time available for testing of the fix.
But this post was not about this problem. The problem is more of a derived problem, In my experience with multiple development cycles, I have seen defects come near the end, or become more serious near the end, where the defect was seen earlier with external users (these are users who are not part of the core team, but could be users who are part of a restricted pre-release or a public release). The challenge comes in terms of recognizing these defects as valid defects.
The advantage of outside users is the amount of diversity that they have. People outside the core team who are testing the product do a lot more adhoc testing, trying out different combinations that the core team would not be testing. In addition, the diversity of equipment that outside users have far strips the diversity of equipment available with the core testing team - they would have more types of machines, different sets of equipment, and so on. This provides an advantage that the core team would be well equipped to use.
At the same time, there is a cost associated with the pre-release users. Some of the defects that they find are already known to the core team, other defects may not be able to be replicated by the core testing team, and yet others are significant defects that the core team takes up for fixing. For a number of defects, it may have been critical for the outside user, but the core team would make a choice that either the defect was not been able to be replicated, or the area was not of a high priority to be taken up for fixing.
However, this is where we had problems and had to make changes. Near the end of the cycle, we would find some defects during the final stage of adhoc testing that had been found by outside users but which the team had dismissed. The amount of impact to the schedule and increasing the stress level of the team was one of the byproducts. To control this problem, we decided to make a lot of effort to evaluate the defects raised by the outside users, including remote control of their machines to try and find the defect cause, spending much more time to analyze the defect, and we had some success in this kind of effort that we were putting. We were able to figure out some of these more serious defects and this in turn reduced the chance of getting some of these defects near the end of the cycle, and also had the byproduct where the outside users felt that their defects were getting more serious attention and produced some great defects.

Soliciting features from your pre-release / beta testers to get them more involved ..

An active set of pre-release participants can be very useful for a product team. Typically why does a team use a pre-release program ? Following are some of the reasons:
- Pre-release participants provide external validation of the features of the application, verifying whether the features that have been built into the product are deemed useful for potential customers. When a team builds a feature, they do so on the basis of market research or other inputs, and then through a process of design and working with the user interface team, they design a solution that is supposed to meet the needs of customers. You can show the workflow to research user bases early, but the validation is only when people get their hands on the product and test it out. And this is extremely valuable, since this sort of validation comes in early enough that there can be changes made in the product to incorporate such feedback.
- Many of these pre-release participants would have been there for multiple versions of the product, and as a result, they have a good knowledge of the product with the advantage that they also have knowledge of actual customer workflows, and hence can quickly suggest some great ways of improving the features. It is upto the product team to take such improvements or not, but I have seen cases where this kind of interaction with the pre-release participant helped the product team learn about some workflows that they were not aware of before. This kind of knowledge is invaluable and the product team should ensure that such participants remain committed to contributing to the pre-release program.
- If you consider the case of video products or imaging products or products that are used for writing to CD/DVD, there are a huge range of devices out there. It is practically impossible for a product team to obtain as well as test these range of devices but when you have a large number of pre-release participants, you can be sure that the total number of devices that you are able to test with is more than what you could obtain on your own, and this gives increased confidence on the part of the product team in the product. As another example, we had a case whereby the product was importing video clips, and there are a large number of video formats that can be used. In one specific format, the embedding of audio with the video clip was crashing the application, and this was not happening with other formats. We would never have found this issue, it was reported by a pre-release participant.

With all these, what else can you do with a pre-release participant. I had earlier mentioned that you need to engage with these valuable testers, and engage with them early. I have seen teams using these participants earlier in the program, even working with them during the feature requirement and feature workflow design process, and even making modifications or tweaks based on the inputs that were received. As a result, there were valuable improvements in the features as well as ensuring that the pre-release participants became very involved with the product. A spin off was that many of them became more active in the user forums and helped some of the more simple customer issues.

Infrastructure: Testing, VPN and network issues relating to external teams

When dealing with testing or development by external teams, there are a lot of infrastructural issues to deal with, and in a number of cases, teams have not prepared some kind of process documentation to take care of such needs and processes. Some of the external teams that can be involved in the process of development or testing for a software application include the following:
- External teams that are working as an extension of the testing team. This is getting more and more common, whereby the testing team can be expanded as necessary by adding vendors to the testing resources. However, when the these external vendors need to start their work, they need access to the testing infrastructure such as test cases repository and the defect management software.
- External teams working on the localization (conversion of the software into different languages). Such teams typically work on both development and testing of the application in different languages and need access to the source repository, as well as the testing infrastructure (test plan and case repository, defect management software, and so on).
- Teams working on the documentation of the product typically need access to the defect management software, since there can be several issues that need to be documented which are typically present in the defect management software, and the documentation team would be provided access to the defect management software.

For these 3 teams and others working in a similar area, if they are located outside the organization, then there may be a necessity for them to be provide the required VPN access to get access to all these tools. In modern secure organizations, such access policies would go through an approval process, and to ensure that such a process is accelerated, it makes sense to prep the approving authorities about these requests for approval.

- External testers / pre-release testers. Such pre-release testers typically do not need any access to source code repository or the testing infrastructure. However, these testers do need a platform where the defects they have logged get into the defect management software in an easy and transparent way (and where the testers are not expected to do any additional work - in some cases, it is as simple as providing them a web page where they can enter the defects that they have found, and some additional parameters that testers  are expected to enter but pre-release testers are not expected to enter are entered by default).

For all these teams, there may be other access issues that teams may not have thought about. Typically when a team has any kind of feature that provides access to an online feature, such features are available in staged servers that need some kind of setting or special access. For ensuring that all the teams listed above and similar teams do not get stuck, it is required to plan for such an access. We had a problem in the past whereby an important online feature was rolled out to the press reviewing team, but there was a big screw up relating to the access for these features, and the team was not able to provide access for around 3 days, in which time, some of the press withdrew - a big disaster. So, a great team has plans for all these before they look to execute such features.