Build acceptance - Ensuring that one person certifies the build as accepted

During the regular product development cycle, the build (typically daily) is the most important input to ensure that the process of build testing, defect fixing and incorporation of the fix into the product, and testing again continues. If the build does not come on a regular basis, defects are not available for the testing team to fix on a regular basis; and it is essential that defects get into the product at a regular basis. If not, team developers start getting out of phase with respect to files being changed during defect fixes by multiple people, and more effort needs to be spent to resolve these dependencies.
Most teams typically like to get the daily product build early in the morning, before the developers and testers are in position. Such a technique ensures that when the team arrives, the build is already available and ensures the maximum utilization of time. However, as a part of this, it is also essential that the build be ready and be fully fit, not have major defects. For this purpose, it is essential that the build be verified before it is started to be used by the team.
A lot of teams have built automated smoke tests that take the build when it is available, launch it and do a quick major tests, and if everything goes right, an email could be configured to go out to a specially setup email list which would ensure that the right people knew that the build was correct. This works in most cases, but there can be cases where there are false negatives. For example, there is a small change in UI or some other feature which was not incorporated into the smoke test, then the smoke test would give a failure, but the build could have been successful (and one can think that such a case may not happen, where the information about a change would always be built into the smoke test, it can happen and has happened).
In other cases, a person needs to evaluate whether a partly successful smoke test and see whether the new build need to be built or the existing build can be used, with some assumptions. For example, one particular feature need not be fully working, but everything else could be working. Now why would the team want to use a build where not everything was working fine. Well, because there is a cost to make a new build. Building a new build could take time, many hours in some cases, and till that time, the developers and testers would have to be using an older build, which may be fine in the early stages, but gets more problematic as you move along the schedule, since such delays in getting defects fixes in the build may increase the cost and impact.
To take such calls needs to be done by a person on the team; and this needs to be specially assigned, since such a person would need to review the build and take a call before the other team members start working on the build, which would mean that the person would need to get on the job much earlier than normal.

Ensuring a regular email with changelist information ..

The daily build is a pretty important item for a development team. There is a need to have a new build of the software on a daily basis, so that defect fixes and code changes make ti to the testing team as quickly as possible. If the build gets delayed for any reason, it means that the development team will have to wait to get these fixes. These delays cause a delay in the actual schedule, and hence most teams have a schedule whereby they get a build early in the morning before the team members have arrived, and also because this means that everything that has been done in the previous day will be available in this build.
However, just because you need a build every day does not mean that you will get it. The process goes that all the changes done in the previous day are all checked in the source safe that the team uses, and based on the last checkin (typically at the cut off time decided between the development team and the build team), the build is fired and takes the required amount of time to process and become a build that is available to the team in time before they come in the next morning.
However, there are a number of things that can go wrong. One of these items relates to whether a checkin done by a developer near to the cutoff time has been incorporated in the build. At near milestone times, there will be a lot of discussion between the development team and the build team to be sure about the actual checkin that is used as the last checkin on which the build has been fired, but on a regular basis, such kind of discussions do not happen. And it is not only the last checkin that can cause a problem, there are other components that go into the application which need to be included in the build and become part of the application. However, all of these are engineering tasks, and sometimes the configuration can go wrong, or the build number that is picked up for one of the components could be wrong (we had a tough case once where a particular functionality was not working as desired, and we found that there was a problem in the execution of a script, which was getting affected due to one of the latest patches, and hence it was picking up the previous version of a component - this was painful to find and diagnose). The other case was when somebody made a great big checkin, but it did not make it into the build while the testing team was under the impression that it was there in the build and start to find issues.
As a result of all these cases, one of the solutions that we had come up with was about ensuring that the build script would also number of the last checkin, and also capture the latest version numbers of all the components that go into the system, and make this available along with the build. Since we also had defect fixes in the defect management system marked with the checkin number, the testing team was easily able to figure out whether the defect fix that they were waiting for made the current build, or came in too late and would only be available in the next build. At times, especially near milestones, the team could even request a new build with this latest checkin to ensure that the feature or defect that they had been waiting for was available to them. Similarly, the version numbers of the components made it very easy to determine whether the component list that was there in the product was the latest.

Critical to do a smoke testing of the regular build - saves time and effort

For teams that have setup a smoke test of their regular builds, the benefits are such that they may not even understand about why such a post needs to be done; after all, it seems to be obvious that if you put a smoke test to do a quick check of the regular build (in our case, it was a daily build), there are multiple advantages. The most prominent advantage is that as soon as the build is available, the smoke test can be triggered to be run automatically, which in turn will take the build through a series of tests and determine whether there is a problem that needs to be resolved.
Sounds great, right ? After all, there are all advantages to having such a smoke testing of the regular build, but a number of teams do not use it, and at the same time, there are some additional effort that does need to be added in the calculation. Let me lay out the case which we used to have. Because of the size of our code and the amount of synchronizations that needed to be done with the source safe, we would have a daily build that would kick off in the night around 11 PM and typically get done around 7 AM. Now, the problem we used to have was that it was not possible to get anybody to be there to check whether, even if the build did get formed, whether the build would even launch and whether there were any major components or workflows that were broken.
When somebody from the testing team would typically get in around 9 AM, they would check the build and then provide their feedback. Our earnest hope used to be that there no problems revealed in this check. However, once in a while, the morning verification would reveal some problems that needed to be fixed, and it would mean that the build for that day would be late or would even have to be canceled, which had some impact on the defect find / fix rate.
The ideal solution was to have a smoke test run on the daily build. Now, the problem was that thought it sounded like an ideal solution, there were some incremental efforts that we had to calculate:
- Somebody had to write the smoke test cases and integrate that with the build system to provide an email output to several members of the team about the success or failure of the smoke test
- As and when we made some changes to any of the major UI or the workflows, we would need to update the smoke test to incorporate these changes. Sometimes these would get missed, and then you would find that the Smoke test stopped in the middle of some feature, and then it would take effort to do the diagnostic of what the problem was.
- The smoke test did yield some false positives as well. The team would then have to spend some time looking for the break in the build, and also examine whether this was a genuine catch, or whether the smoke test malfunctioned, and the problem could be ignored.
- The tools for running the smoke test could be expensive and the expenditure may need to be approved by somebody who does not understand as to why there is a need for such a software, so some amount of negotiation and discussion would be needed.
- Suppose the smoke test would start running at 7:30 AM and run for around 45 minutes, somebody from the team would need to be in place or looking at the report from the smoke testing at the completion time of the smoke test, and hence there would be a rotating chair where team members would be assigned this role for a specific period.

Explain Automation of Smoke Testing?

Smoke testing though being a very useful, quick  software testing methodology, if not carried out in an automated way can take up a lot of time and efforts. 

Need for Automation

There is one more thing to manually carry out smoke testing which is that if later if you come to know that the testing was being carried out in wrong direction then it will surely give you a headache to perform whole of the testing once again and again if required. 

Why automation of smoke testing useful?

- Automation of smoke testing proves to be very useful in saving time and efforts.

- Smoke testing is considered to be one of the most effective ways that are used for the validation of the changes that are made to the software system or application program code at a high level before a detailed testing is undertaken on the new build. 

- Carrying out smoke tests can help a lot to stabilize the builds and verify that there are no major problems in the software system or application.

- The reliability factor of the smoke testing determines the success of the smoke test i.e., the more you can rely up on the smoke test for finding major problems, the more you can cut down the costs and increase the quality measures of the software product or artifact.

- Smoke testing provides a reliable, scalable solution that can be quickly and easily be automated. 

- One of the best things about automating the smoke test is that it does not require any programming.

- Smoke tests can be quickly written and automated by testers of any skill level within few minutes.

What functions are provided by the tools used for automating smoke tests?

There are so many tools available for automating the smoke tests. With these tools you can perform the following functions:

1. With these tools smoke tests can be written and automated within few minutes and without requiring any programming.
2. Using these tools all the builds can be validated before any changes are incorporated in to them.
3. These tools are quite helpful when it comes to stabilizing the whole build process and verifying the readiness of a build for further full scale QA testing.
4. With these tools quick tests can be conducted ensuring that the basic functionality is still intact and is working properly.
5. The overall quality of the product can be improved since the problems can be detected early.
6. Costs can be reduced ensuring that the team is having sufficient time for developing the product.

How a smoke test is automated?

- Smoke testing is usually preceded by quality assurance.

- As soon as the errors are discovered, it requires less efforts and money to get fixed.

-In automated smoke test, a continuous build process is deployed that automatically performs a smoke test each time a build is finished developing. 

- With such a methodology, the developers come to know easily if the recently developed build caused the problem. 

- Depending upon the configuration of the build tool and the software system or application, the process of implementing the automated smoke tests will vary despite of the basic steps being followed the same. 

- After the successful completion of the build, some set up steps are performed before the application is tested. The steps may include:

1. Copying files to the appropriate places.

2. Setting up the data base tables.

3. Starting a server.

4. Installing licenses.

- Next step is to obtain all the QA files required for the smoke test and running the smoke test. 

- The report of the smoke test is saved and final step involves clean up including the steps:

1. Stopping a server
2. Emptying data base tables and
3. Deleting files.

What are the advantages of smoke testing?

Smoke testing is one of the quick and dirty tests that we have in the software testing fields because of which the major functions of the software system or applications can be worked out without bothering about the finer details of the code and implementation. 

When and how smoke testing is performed?

- Smoke testing is basically performed when the software system or application is thought to be absolutely new.

- After the build is received, it is run and here the smoke testing comes in to play and involves checking whether the build is stable or not.

- A smoke test is not a singular test but rather a series of tests that are carried out on the software system or application before the actual full scale testing of the application is commenced.

- It would not be wrong if you say that the smoke testing is type of non- exhaustive test. 

- Furthermore, in some cases the smoke testing also plays the part of the acceptance test that is run prior to the introduction of a new build to the main structure of the software system or application and before the regression or integration process.

In this article we will be discussing about the advantages of the smoke test but before this let us see what all are its characteristics:

1. Smoke test is scripted either as an automated test or a manually written test.
2. It is designed to focus on all the parts of the application like a cursor can do.
3. It is shallow.
4. It is wide.
5. Ensures the working of the most crucial functions of a program.
6. Ensures that the build is not broken.
7. Verifies readiness of a build to be tested.
8. It is not a substitute for actual functional test.

Advantages of Smoke Testing

Now let us chalk out the advantages of the smoke testing:

1. Carrying out smoke testing at various stages reduces the problem of integration. The risk of integration is minimised. Most of the teams fear facing this risk that a project in which they have to integrate or combine code up on which they have been working individually and it may not work well. At this stage only the incompatibility of the software system is discovered. If the integration takes place earlier than smoke testing then the debugging process will take a lot of time and may require re- implementation and re- designing of the whole system. In most of the cases, projects have been cancelled due to errors during the integration. With the daily smoke tests, the integration errors can be reduced and runaway integration problems can be prevented.
2. If the smoke test has been designed properly, it can detect errors and problems at an early stage.
3. Since the smoke test detects the majority of the problems at an early stage, much time and efforts are saved.
4. With smoke testing, the risk of low quality is reduced.
5. With the daily smoke tests, the quality problems can be prevented from taking the control of the project.
6. The smoke test can uncover major problems and defects as a consequence of wrong configuration.

Basically what happens in smoke testing is that the whole system is exercised from end to end and those errors and problems are stressed that cause the functioning of the whole system to stop. The smoke test is not so exhaustive but it does expose out major problems of the software system or application under test. The smoke testing at all stages ensures the working of the major functionality and keeps a check on the stability of the build. 

What are different characteristics of smoke testing?

Smoke testing is considered to be one of the preliminary software testing techniques to further testing that are intended to reveal simple failures that are simple enough to be a cause of a software project to be rejected. 

In the term “smoke testing”, smoke is used as a metaphor. All the test cases that are known to provide coverage to most important functionality of a component of the software system or application are selected and grouped under a set and later are run. This is basically done to ascertain that the most crucial functions and features of the software system and application are working as desired or not! 

One can understand smoke testing better by having a look at the below mentioned questions:

1. Does the program run?
2. Does clicking the start button do anything?
3. Does the program run?

Goals of Smoke Testing

- The primary goal of the smoke testing is to determine if the problem or fault in the software system or application is so worse that any further testing on that software system or application will be a waste. 

- To put it simply, the smoke testing can be considered to be a cheap and less time consuming way to broadly cover the features and functionalities of the software product in a limited period of time. 

- By carrying out smoke testing, you can easily find out if any of your software system’s or application’s key feature or functionality is broken at some point and so your development team would not spend further time on it fixing or recreating it. 

People often confuse between the two similar terms i.e., smoke testing and build verification test. No doubt both are one and same the only thing when the smoke testing is performed on a build it called build verification test.

When should smoke testing be performed?

- One of the best practices of software testing is that the smoke testing should be performed everyday without fail. 

- Furthermore, the smoke testing can also be performed by the testers before accepting a module or build for further testing.

- Some testing has been listed as the most cost effective method for the identification and fixation of the defects in a software system or application after the code reviews by the software giant Microsoft Corporation. 

- The smoke testing is deployed as a process for the validation of the changes that have to be in the code before they are passed on to the source control in the Microsoft Corporation. 

- The smoke testing has to be carried out either automatically or manually that is purely the choice of the tester.

- When the smoke testing is carried out manually, the process goes on normally but when the automated tools are used, the tests that have to be carried out are initiated during the same process through which the build is generated. 

Smoke tests have 2 types as mentioned below:

1. Unit tests: Smoke tests under this category are known to exercise the sub routines, object methods and individual functions and so on.
2. Functional tests: These types of smoke tests are known to exercise the complete program along with various inputs.

Both the above mentioned type of smoke testing tools together make up a third party product that falls out of the compiler suite. 

A scripted series of program inputs form a function test and some may also have an automated mechanism. On the other hand the unit tests may be formed out of the separate functions that lie within the code. They can even be a driver layer that might be linked to the code. The smoke testing in software can be compared to the smoke testing in hard wares.