For teams that have setup a smoke test of their regular builds, the benefits are such that they may not even understand about why such a post needs to be done; after all, it seems to be obvious that if you put a smoke test to do a quick check of the regular build (in our case, it was a daily build), there are multiple advantages. The most prominent advantage is that as soon as the build is available, the smoke test can be triggered to be run automatically, which in turn will take the build through a series of tests and determine whether there is a problem that needs to be resolved.
Sounds great, right ? After all, there are all advantages to having such a smoke testing of the regular build, but a number of teams do not use it, and at the same time, there are some additional effort that does need to be added in the calculation. Let me lay out the case which we used to have. Because of the size of our code and the amount of synchronizations that needed to be done with the source safe, we would have a daily build that would kick off in the night around 11 PM and typically get done around 7 AM. Now, the problem we used to have was that it was not possible to get anybody to be there to check whether, even if the build did get formed, whether the build would even launch and whether there were any major components or workflows that were broken.
When somebody from the testing team would typically get in around 9 AM, they would check the build and then provide their feedback. Our earnest hope used to be that there no problems revealed in this check. However, once in a while, the morning verification would reveal some problems that needed to be fixed, and it would mean that the build for that day would be late or would even have to be canceled, which had some impact on the defect find / fix rate.
The ideal solution was to have a smoke test run on the daily build. Now, the problem was that thought it sounded like an ideal solution, there were some incremental efforts that we had to calculate:
- Somebody had to write the smoke test cases and integrate that with the build system to provide an email output to several members of the team about the success or failure of the smoke test
- As and when we made some changes to any of the major UI or the workflows, we would need to update the smoke test to incorporate these changes. Sometimes these would get missed, and then you would find that the Smoke test stopped in the middle of some feature, and then it would take effort to do the diagnostic of what the problem was.
- The smoke test did yield some false positives as well. The team would then have to spend some time looking for the break in the build, and also examine whether this was a genuine catch, or whether the smoke test malfunctioned, and the problem could be ignored.
- The tools for running the smoke test could be expensive and the expenditure may need to be approved by somebody who does not understand as to why there is a need for such a software, so some amount of negotiation and discussion would be needed.
- Suppose the smoke test would start running at 7:30 AM and run for around 45 minutes, somebody from the team would need to be in place or looking at the report from the smoke testing at the completion time of the smoke test, and hence there would be a rotating chair where team members would be assigned this role for a specific period.
Sounds great, right ? After all, there are all advantages to having such a smoke testing of the regular build, but a number of teams do not use it, and at the same time, there are some additional effort that does need to be added in the calculation. Let me lay out the case which we used to have. Because of the size of our code and the amount of synchronizations that needed to be done with the source safe, we would have a daily build that would kick off in the night around 11 PM and typically get done around 7 AM. Now, the problem we used to have was that it was not possible to get anybody to be there to check whether, even if the build did get formed, whether the build would even launch and whether there were any major components or workflows that were broken.
When somebody from the testing team would typically get in around 9 AM, they would check the build and then provide their feedback. Our earnest hope used to be that there no problems revealed in this check. However, once in a while, the morning verification would reveal some problems that needed to be fixed, and it would mean that the build for that day would be late or would even have to be canceled, which had some impact on the defect find / fix rate.
The ideal solution was to have a smoke test run on the daily build. Now, the problem was that thought it sounded like an ideal solution, there were some incremental efforts that we had to calculate:
- Somebody had to write the smoke test cases and integrate that with the build system to provide an email output to several members of the team about the success or failure of the smoke test
- As and when we made some changes to any of the major UI or the workflows, we would need to update the smoke test to incorporate these changes. Sometimes these would get missed, and then you would find that the Smoke test stopped in the middle of some feature, and then it would take effort to do the diagnostic of what the problem was.
- The smoke test did yield some false positives as well. The team would then have to spend some time looking for the break in the build, and also examine whether this was a genuine catch, or whether the smoke test malfunctioned, and the problem could be ignored.
- The tools for running the smoke test could be expensive and the expenditure may need to be approved by somebody who does not understand as to why there is a need for such a software, so some amount of negotiation and discussion would be needed.
- Suppose the smoke test would start running at 7:30 AM and run for around 45 minutes, somebody from the team would need to be in place or looking at the report from the smoke testing at the completion time of the smoke test, and hence there would be a rotating chair where team members would be assigned this role for a specific period.
No comments:
Post a Comment