Critical to do a smoke testing of the regular build - saves time and effort

For teams that have setup a smoke test of their regular builds, the benefits are such that they may not even understand about why such a post needs to be done; after all, it seems to be obvious that if you put a smoke test to do a quick check of the regular build (in our case, it was a daily build), there are multiple advantages. The most prominent advantage is that as soon as the build is available, the smoke test can be triggered to be run automatically, which in turn will take the build through a series of tests and determine whether there is a problem that needs to be resolved.
Sounds great, right ? After all, there are all advantages to having such a smoke testing of the regular build, but a number of teams do not use it, and at the same time, there are some additional effort that does need to be added in the calculation. Let me lay out the case which we used to have. Because of the size of our code and the amount of synchronizations that needed to be done with the source safe, we would have a daily build that would kick off in the night around 11 PM and typically get done around 7 AM. Now, the problem we used to have was that it was not possible to get anybody to be there to check whether, even if the build did get formed, whether the build would even launch and whether there were any major components or workflows that were broken.
When somebody from the testing team would typically get in around 9 AM, they would check the build and then provide their feedback. Our earnest hope used to be that there no problems revealed in this check. However, once in a while, the morning verification would reveal some problems that needed to be fixed, and it would mean that the build for that day would be late or would even have to be canceled, which had some impact on the defect find / fix rate.
The ideal solution was to have a smoke test run on the daily build. Now, the problem was that thought it sounded like an ideal solution, there were some incremental efforts that we had to calculate:
- Somebody had to write the smoke test cases and integrate that with the build system to provide an email output to several members of the team about the success or failure of the smoke test
- As and when we made some changes to any of the major UI or the workflows, we would need to update the smoke test to incorporate these changes. Sometimes these would get missed, and then you would find that the Smoke test stopped in the middle of some feature, and then it would take effort to do the diagnostic of what the problem was.
- The smoke test did yield some false positives as well. The team would then have to spend some time looking for the break in the build, and also examine whether this was a genuine catch, or whether the smoke test malfunctioned, and the problem could be ignored.
- The tools for running the smoke test could be expensive and the expenditure may need to be approved by somebody who does not understand as to why there is a need for such a software, so some amount of negotiation and discussion would be needed.
- Suppose the smoke test would start running at 7:30 AM and run for around 45 minutes, somebody from the team would need to be in place or looking at the report from the smoke testing at the completion time of the smoke test, and hence there would be a rotating chair where team members would be assigned this role for a specific period.

Ensuring that builds are posted in the locations where people are located

In today's world, applications are developed all over the world, in the sense that if you look at a product development team, it will have members located all over the world. So, I have worked on several products where some developers are based in a location in India, some are based in Germany, and some others are based in multiple locations in the United States. Now, one can argue that this would be confusing and not an ideal state, but the fact remains that because of reasons related to costs, acquisition of companies (I once had a colleague who was absorbed into the company as part of one product, and then assigned to another product based on a resourcing decision - with the associated issue being that the original product was being developed out of his location, but for the new product where he was now associated, he was now a remote worker), and the need to meet the geographic needs of talented people who would otherwise not join, teams can get geographically dispersed. So, let us take it for granted that such teams exist, and consider some of the problems that can happen with such teams. We will talk about coordination & communication issues in a separate topic, but one major issue with teams working on large products is the sheer size of getting the builds to them on a regular basis. Consider teams that work on applications such as MS Office, Photoshop, CorelDraw, and many other such products, the sheer size of the application builds can be upwards of 1GB+.
When you are working on an application development program, it is important to work on the newest version of the application. Typically for most product development applications, this means that a new build will be generated every day, available for the developers and testers. This allows the testers to test with all the fixes that have been made the previous days, and also ensures that developers have the same confidence that their fixes are in, and are being tested regularly. Now this sounds fine when everybody is located in the same geographic location (I mean there are other issues related to ensuring that the build is a solid build, that build failures are detected early, and so on), but gets more challenging when members of the development team are located in different locations. Given the size of some applications (as I mentioned above, they can be 1GB or more sometimes), even with very fast connections, it can take time to ensure that these builds are delivered to the remote locations within a short period that they are available in the location where they are generated. And when you consider locations such as those in India, the availability of fast broadband lines is not of the same assumed speed as those available in the United States (or which can actually be more expensive). So, there needs to be a lot of thought given to how these builds will reach the different geographic locations. All this rules out the possibility of transferring builds manually to the different geographic locations; instead what needs to be done is to add a sequence of commands in the build script near the end that check for whether the build has been made, that the sequence has not showed any obvious errors (and if possible, incorporate some smoke tests to ensure that the build launches and works fine) and then does an automatic ftp transfer to default directories on the other geographic locations. The reason for incorporating smoke tests is to ensure that you do not start sending out builds to the other locations that are not possible to use. Even in such cases, you will find that once in a while, there will be failures and build will not be available, but in most cases, you will find that you needs will be met.
However, there is a complication that can happen about setting the timing of the build delivery across multiple locations, and I will cover that in a forthcoming article (link to next article - TBD).

What are different tools used for smoke testing?

The success of a software testing methodology depends a lot on the quality and type of tools that are being used to carry out that particular test. 

In this article we have discussed about the tools that are used for smoke testing. One can find so many tools around that are available for automating the smoke tests. 

Functions performed by these tools

Using these tools a lot of functions can be performed few of which have been mentioned below:

1. Using these smoke testing tools one can easily and quickly write the test cases and scripts as well as automate them and guess what? These tools do not require any programming.
2. With these smoke testing tools, the validation of all the builds can be carried out easily before any changes are introduced in to them.
3. Nothing comes as handy as these smoke testing tools when one needs to stabilize the whole build process.
4. The available tools make it easy the verification of a build to determine its readiness for further full scale testing.
5. The smoke testing tools help in conducting lightening fast tests determining whether or not the major functionalities are intact and still working as expected.
6. The tools help in reducing the drudgery of the testers as well as the developers by helping in the early detection of the problems and errors.
7. These tools help a great deal in cutting down the overall development costs by significant margins providing the team with sufficient time period for developing the software system or application.

Tools for Smoke Testing

- So many tools are available using which you can perfect your tests like advanced web recorder, object recorder, editor and so on. 

- The tools provide so many actions to be performed for editing the tests if it is required to make any changes in the software system or application. 

- Such an approach to testing i.e., using tools allows flexibility between the test and testers and eliminates the need of creating new tests. 

- There are tools called “work flow designers” that come with an intuitive graphical representation and using which you can easily create as well as manage the work flows of high level of the entire suite of your smoke tests. 

- There are testing tools available that help you introduce check points in the application so that at any point your software application functionality can be further verified. 

- There is one tool available for testing methodologies that is perhaps the most sought after testing tool. 

- This tool lets you convert a test script in to an executable code and then run it on multiple systems without installing testing application on each and every machine. 

- This feature of this tool facilitates an automation that never existed before. 

- Any of the test scripts can be run and with any inputs operating from any data base or application software and almost in every kind of environment. 

- Some reporting and log tools have been introduced which possess advance reporting features like:

1. Time lines
2. Report customization tools that are quite easy to use.
3. Report dash board
4. Audit trail capability (this feature provides you with a complete data base of all the events that took place in the automated testing environment.)
5. Visual logs

- With a tool having all such features a tester can easily analyze the reports with visual aids and fix them quickly.

- These tools provide the testers with a more complete view of the smoke test runs. 

- Though smoke testing can be carried out manually also it is advisable to use automated test tools which initiate tests by a similar process that generates the build itself.

What are different characteristics of build verification test?

Build verification test is often abbreviated as BVT and can be defined as a set of tests that are carried out on all the builds that are newly built in order to verify if those builds are testable or not before they are transferred to the testing team for their further testing. 

Generally, the test cases used in build verification test are considered to be the core functionality test cases which are used to keep the stability of the software systems or applications in check and regulate their testing thoroughly. 

The whole process of build verification test takes a whole lot of efforts and time if carried out manually and therefore the whole process is usually automated. If a build fails the build verification, then the same build is again returned to the developer to fix the faults. 

There are other names also by which the build verification test is known as mentioned below:

1. Smoke testing
2. Build acceptance testing or BAT

In a typical build verification test, there are two aspects that are exclusively tested and are mentioned below:

1. Build acceptance
2. Build validation

Few basics of Build Verification Tests

1. Build verification tests are a sub set of tests that are used for the verification of the main functionalities.
2. Some build verification tests are created on a daily basis and some builds are daily tested and if those builds fail the build verification test, they are rejected and returned back to their developer for making the fixes and when the fixes have been done, a new build is released and is gain subjected to the build verification test.
3. The build verification test has an advantage that it saves the precious efforts of the testing team that are required for setting up a test and testing a build whenever there is a break in the major functionality of the build.
4. The test cases of the build verification test should be designed very carefully so that they provide the maximum possible coverage to the basic functionality of the build.
5.  A typical build verification test is run for 30 minutes maximum and not then this limit.
6. The build verification testing can also be considered as a type of regression testing that is done on each and every build that is new.

Aim of Build Verification Test

- The primary aim of the build verification test is to keep a check on the integrity of the whole software system or application in terms of its build or we can say modules.

- When several development teams are working together on the same project, it is important that the modules that they are developing individually have got good ability for integrating with each other since this is very important. 

Till now so many cases have been recorded in which the whole project failed miserably due to a lack of integration among the modules. There are some worst cases also in which the whole project gets scraped just because of the failure in the module integration. 

- The build release has got a main task i.e., file check in i.e., including all the modified as well as new project files associated with the corresponding builds. 

- Earlier checking the health of the building initially was considered to be the main task of the build verification test. 

- This is called as “initial build health check” and it includes:

1. Whether or not all the files have been included in the release or not?
2. Whether all the files are in their proper format or not?
3. Whether all the file versions and languages have been included or not?
4. Whether the appropriate flags have been associated with the file or not?

Compare Test Driven Development and Traditional Testing?

Till many years the traditional testing was in use until there was a rise of another software development strategy or process called the test driven development. These two development processes are in great contrast to each other. 

This article is focussed entirely up on the differences between the two types of development i.e., the traditional testing and the test driven development. So let us begin by describing the test driven development. 

About Test Driven Development

- The test driven development process is comprised of the repetition of very short development cycles.

- By short here we mean the development cycles in the test driven development are shorter than the usual normal cycles. 

- These development cycles comprise of the following tests:

1. Production of code: This step involves the creation of some code implementing which the code can only pass the test and does not incorporates any new functionality.
2. Execution of the automated tests and observing their success.
3. Re-factoring of code: This step involves the cleaning up of the code.
4. Repetition: The whole cycle is repeated for the improvement of the functionalities. 

- The test driven development was considered to be somewhat related to the test- first programming concepts of the extreme programming long since. 

- After that it came to be as an individual software development process. 

- The test driven development has prove to be quite effective for developing ad improving the legacy code that has been developed using the older development techniques. 

- So many development styles for the test driven development have been identified like those mentioned below:

1. Fake it till you make it
2. Keep it simple stupid or KISS
3. You ain’t gonna need it or YAGNI

- In the test driven development the primary focus is on writing the code which is necessary only to pass the tests to keep the design clean and clear of the fuzz.

About Traditional Testing

 Now coming to the traditional development methodologies or approaches, they are commonly called as the engineering approaches and they were defined at the very beginning of the software sciences. 

- Traditional development methodologies were developed to control the software development processes via a disciplined approach whose build and the stages were predictable.

- In the traditional software development methods, the stages of analysis and design precede the stage at which the software is build. 

- Unlike the test driven development process, these traditional development process are well documented .

- Their main disadvantage is that they are quite difficult to apply because of their complexity.

- Another disadvantage being that the traditional development methodologies are bureaucratic.

- In practical, these traditional development processes often cause a high level of complexity in the software system or application. 

- In traditional approaches there are two main stages namely:

1.      Stage of analysis and

2.      Stage of design

-The foundation of the whole project depends up on these two stages, therefore it is necessary that adequate focus is on these two stages and more and more efforts should be put on these stages. 

- For the project to be successful it is important that the traditional method is applied in the right way.

- Designing is considered to be a highly creative activity. 

- It becomes very difficult to plan and predict these complex methodologies as the level of their creativity increases. 

- The main feature of the traditional development processes is the detailed planning and designing phase. 

- Traditional development holds good when it comes to undertaking very large projects involving higher risk. 

- One more commonly observed thing is that the projects development with the traditional methodologies last longer. 

What is build verification testing technique?

Build verification test is often abbreviated to “BVT”. This test is more commonly known as “build acceptance test”. We can make out from the term itself that this test consists of test cases that are executed on the build of a software product in order to verify that the build of that particular software system or application is testable before the software product is passed on to the testing team for testing.

ABOUT BUILD VERIFICATION TEST
- The main build verification test consists of smaller test cases and these test cases exercise the functionality of the mainstream of the software system or application.
- Any further testing on a build is performed only if it passes the build verification otherwise it is simple rejected.
- In the place of this build, the testing is continued to be carried up on the previous one.
- There should be at least one build that should pass the build verification test.
- Build verification is important to be carried out prior to the commencement of the testing phase.
- It tells the software developers whether the build of the software has been rightly developed or not.
- It also tells if the build is having any major issue.
- A build verification test also avoids time being wasted by carrying out unnecessary testing on the faulty builds.
- A typical build verification test is usually automated.
- When a build is rejected during the build verification test, it is again revered back to the software developer for correcting it.

Build verification test is also called by another name “smoke testing”. In build verification testing two aspects are mainly checked:

1. Build acceptance
2. Build validation


BASIC THINGS TESTER SHOULD KNOW ABOUT BVT

1. A build verification test consists of many smaller sub tests and each one of these tests verifies one functionality of the software system or application.
2. Build verification tests are very effective in saving the efforts of the testing team.
3. The build verification testing methodology can be used to test a build even if its major functionality are not working or are broke.
4. The build verification tests are designed very carefully so as to make them cover a major part of the functionality and features.
5. A certain time limit has been set for the build verification test to 30 minutes.
6. The testing time for running build verification should not exceed this limit.
7. Build verification testing is somewhat similar to the regression testing in the way that it is carried out on each and every build.
8. Build verification test is a way to test the integrity of the software project.
9. It checks whether or not all the modules have been integrated well with each other or not.
10. Maintenance of integrity between the modules is very important since any lag in the cooperation can cause the whole system to go bizarre.
11. Initially, the build verification testing technique was employed to check whether or not all the new features, functionality and modules have been included in the project to be released or not.
12. It also checks for the correctness of the formats of the included files.
13. Even the flags associated with a file are checked.
14. You should be very careful while deciding what all the test cases have to be carried out under the build verification test.
15. Your build verification test must include all the critical test cases and these test cases should not be unstable.
16. Another point to be kept in mind is that you should have an idea of the result of the test cases that you are including in the BVT.

Overview of smoke testing : Characteristics and Advantages

Smoke testing is a quick-and-dirty test that the major functions of a piece of software work without bothering with finer details.
Smoke testing is used when the product is new. Smoke testing is run when the build is received to check if the build is stable. A smoke test is a series of test cases that are run prior to commencing with full-scale testing of an application. It is a non-exhaustive kind of testing.
A smoke test is used as an acceptance test prior to introducing a new build to the main testing process, i.e. before integration or regression.

Characteristics of Smoke Testing
- A smoke test is scripted, either using a written set of tests or an automated test.
- A Smoke test is designed to touch every part of the application in a cursory way.
- It is shallow and wide.
- Smoke testing is conducted to ensure whether the most crucial functions of a program are working, but not bothering with finer details.
- Smoke Testing is applicable when new components gets added and they are integrated with the existing code.
- It ensures that the build is not broken.
- They are also useful for verifying a build is ready to send to test.
- Smoke tests are not substitute for actual functional testing.

Advantages of Smoke Testing
- The integration problems reduces when smoke testing is done at various stages.
- A properly designed smoke test is capable of detecting problems at an early stage.
- If the major problem is detected at an early stage, it saves time and cost.

High-level Best Practice Five(5) in Software Configuration Management

There are six general areas of SCM deployment, and some best practices within each of those areas. The first, second, third and fourth area and there practices are already discussed.

Builds

Builds are necessary to construct software from source files. The high level practices involved with this workspace are :
- In a build, the only inputs that are needed are the source files and the tools that are used. There is no place for procedures or yellow sticks. If setup procedures are there, automate them in scripts. If manual setup is there, document them in build instructions.
- There are chances that the input list is incomplete if the software cannot be build from the same inputs. So, the input list needs to be checked.
- When you are organizing original source files in a directory, you need to ensure that already built objects are kept away and do not contaminate the source files. Built objects (those files that get created during the building process) should be located in a different directory, away from the original source files (which are the files that have been created through a tool such as a code editor, or through notepad, or through code generation tools).
- Developers, test engineers, and release engineers should all use the same and easily available build tools.
- Build should be done often as end-to-end builds with regression testing (“sanity” builds) reveal integration problems introduced by check-ins and they produce link libraries and other built objects that can be used by developers.
- Build outputs and logs, including source file versions, tools and OS version info, compiler outputs, intermediate files, built objects, and test results should be kept for future reference.