But this post was not about this problem. The problem is more of a derived problem, In my experience with multiple development cycles, I have seen defects come near the end, or become more serious near the end, where the defect was seen earlier with external users (these are users who are not part of the core team, but could be users who are part of a restricted pre-release or a public release). The challenge comes in terms of recognizing these defects as valid defects.
The advantage of outside users is the amount of diversity that they have. People outside the core team who are testing the product do a lot more adhoc testing, trying out different combinations that the core team would not be testing. In addition, the diversity of equipment that outside users have far strips the diversity of equipment available with the core testing team - they would have more types of machines, different sets of equipment, and so on. This provides an advantage that the core team would be well equipped to use.
At the same time, there is a cost associated with the pre-release users. Some of the defects that they find are already known to the core team, other defects may not be able to be replicated by the core testing team, and yet others are significant defects that the core team takes up for fixing. For a number of defects, it may have been critical for the outside user, but the core team would make a choice that either the defect was not been able to be replicated, or the area was not of a high priority to be taken up for fixing.
However, this is where we had problems and had to make changes. Near the end of the cycle, we would find some defects during the final stage of adhoc testing that had been found by outside users but which the team had dismissed. The amount of impact to the schedule and increasing the stress level of the team was one of the byproducts. To control this problem, we decided to make a lot of effort to evaluate the defects raised by the outside users, including remote control of their machines to try and find the defect cause, spending much more time to analyze the defect, and we had some success in this kind of effort that we were putting. We were able to figure out some of these more serious defects and this in turn reduced the chance of getting some of these defects near the end of the cycle, and also had the byproduct where the outside users felt that their defects were getting more serious attention and produced some great defects.