What are limitations of smoke testing?

Smoke testing though being quite a helpful software testing methodology has got some limitations which will be discussed in this article. 

Smoke testing is an important software testing methodologies when it comes to the development of very large software projects. "The formal definition of the smoke testing states that it a quick and indeed a dirty software testing methodology that is deployed mainly for testing the major features and functionalities of a piece of software system or application". 

About Smoke Testing

- It is essential to perform smoke testing whenever any changes are implemented in the software system. 

- Originally, the smoke testing for software systems was adopted from the hardware industry. 

- Smoke testing is quite time and cost effective since it lays its primary focus on the components that have been changed recently in order to ensure continued compatibility.

- In addition to all this, a smoke test provides an effective means to confirm that the changed code works as desired and does not hampers the functioning of the whole build. 

- Following this approach, the bugs entering the software system can be immediately fixed. 

- Smoke testing has many advantages that outweigh the limitations but still the limitations cannot be ignored.

Limitations of Smoke Testing

1. Biggest limitation of the smoke testing is that its field of application and usefulness is quite narrow.
2. Smoke testing can only be used when the time frame is small during the introduction of a new functionality or feature in to the software system or application.
3. Though being wide, a smoke test is very shallow.
4. Smoke testing does not take into consideration the fine details of the software system or application.
5. Smoke tests cannot be substituted for actual functional tests.
6. Smoke testing is a kind of black box testing and does not consider the internals of the software system or application.
7. In smoke testing the tester cannot access the source code.
8. In smoke testing the tester has to interact with the system via an interface by giving a variety of inputs and then examining the obtained outputs.
9. Another limitation is that the path coverage provided by the smoke tests is very limited since all the given inputs have to be checked.
10. Smoke tests cannot keep a control on targeting the paths and code segments that might be more error prone than the other segments.
11. Smoke test are a bit difficult to design since they should be designed in such a way that they touch every part of the application software.
12. Usually the smoke testing can be applied only when some new components are incorporated in to the existing software system or application.

More about Smoke Tests...

- Smoke tests can be thought of as preliminary tests that facilitate the further testing of the software system or application. 

- Though the smoke tests reveal simple failure, these failures are good enough to be the cause of the rejection of a prospective and deserving software project. 

- At last we conclude that there exists no such absolute best software testing methodology, each being having its own weakness as well as strength. 

- You cannot expect a testing methodology to fit the test requirements of every application software and conditions. 

- This reason only is enough to conclude that no testing methodology is most important. 

What are different types of smoke tests?

Smoke testing has merged as one of the best general software testing methodology though it is a dirty one and very shallow too! Smoke testing is one of the easiest and quickest black box software testing methodologies that we have today. 

Smoke tests are of two types namely:

1. Functional tests and

2. Unit tests

About Functional Tests

- These type of smoke tests are typically scripts or programs that have been configured to test the group of classes so as to determine whether or not they meet the external requirements.

- One example of functional tests is the screen- driving programs that are used to test the graphical user interfaces (GUIs).

- They are a certain class of black box tests that are based up on the specifications of the software components under testing. 

- The testing is carried out normally i.e., giving the input and observing the output and the internal structure of the system is neglected.

- Functional tests are implanted via the following 5 steps:

(a)  Identifying the functions that are to be performed by the software system.

(b)  Creating the input data that is based up on the specifications of the functions.

(c)   Determining the output based up on the specifications of the functions.

(d)  Executing the test case.

(e)  Comparing the actual and expected outcomes.

- The functional smoke tests form a key component of the software engineering as they help track the growth of the project.

- Using the functional tests, the system can be tested from end to end. 

- Similar to the unit tests the functional tests accelerate the development rate of the project while ensuring its quality. 

- Functional tests are specified by the customers or the customers may provide the developers with user stories based on which the developers may write the functional tests. 

- Errors that are encountered during functional tests are fixed using the unit tests and then again verified by the functional tests. 

- The more the number of functional tests and the more correct they are, the more is the certainty that the software system or application is working. 

- To make this whole process efficient, you need to automate the whole testing process.

About Unit Tests

- For carrying out these types of smoke tests you need a unit test frame work that will create automated unit tests suites. 

- Another thing is that all the classes within the build should be tested. 

- The tests are created before the code is written. 

- The code repository holds the unit tests along with the code that is to be tested. 

- Automated unit tests though are difficult and take a lot of time, yet they offer huge pay back. - It takes a lot of time to produce a good unit test suite. 

- The unit tests enable the following aspects:

(a) Collective ownership: This eliminates the need for individual code ownership since all the classes are guarded by the unit tests.

(b) Re-factoring: This ensures that a change in one part has not hampered the performance of the other parts.

About Frequent Integration

- This is possible only if there is a single universal unit test suite for regression testing and validation. 

- When your unit tests are automated it becomes easy to merge the changes with the latest released version. 

- Whenever a new functionality is added it is required that the unit tests are also changed  accordingly to reflect the functionality. 

How is optimization of smoke testing done?

Smoke testing being one of the quick and dirty software testing methodologies, needs to be optimized. This article focuses on the need of optimizing the smoke tests and how to optimize them. 

Let us first see the scenario behind the need for optimization of the smoke tests that are carried out on the software system or application. 

- In most cases of the development of software systems or applications, the code can be executed either directly from the command prompt or via a sub- routine of the larger software systems or applications. This is called the command prompt approach. 

- The code is designed in such a way that it has the qualities like self awareness as well as it is autonomous. 

- By the code being self aware, we mean that if anything goes wrong during the execution, the code should explain it. 

- Commonly two types of problems are encountered during testing and have been mentioned below:

1. The path was compiled with too much of optimization and
2. The data directory is not pointed out properly by the path.

Steps for Optimization of Smoke tests

- In order to confirm that the code was compiled correctly one should run the test suite properly at least once. 

- The first step towards optimization of the smoke test is to run it and then examine the output. 

- There are two possibilities that either the code will pass the test or it won’t. 

- If it is the second case then there are two possibilities that where your smoke test went wrong:

1. Compiler bugs and errors or
2. The path has not been set properly.

Compiler Bugs and Errors

Let us take up the first possibility, i.e., the compiler bugs and errors. 

-  It is probable that the correct code might not have been produced by the compiler. 

- In some cases of the serious compiler bugs, it is possible that there might be some hardware exception and these kinds of errors are caused mainly by the compiler bugs. 

- In this case the optimization level of the code should be minimized. 

- After this the code should be recompiled and observed again. 

- Optimizing is good for code but if it is of aggressive kind then it will definitely cause problems.

If path is not set properly

- It is obvious that if the code is not able to trace its data files then it will definitely show up some error and this happens because the path has not been set properly. 

- In such cases you need to check which path was it, fix it and recompile the whole code and execute once again.

When does a system or an application crash?

Don’t think that the software system or application crashes only when there has aggressive optimization of the code! 

- Crashes also happens with those programs in which there is no optimization of the code. 

- But in such cases, only compiler errors can be blamed since it happens only if the compiler has not been set up properly on your system. 

- If no program executes, it means that your compiler is broken and you need to talk about this to your system administrator.

How to optimize smoke tests?

- A lot of help comes from MPGO (managed profile guided optimization). 

- The best way to optimize any kind of testing is to maintain a balance between the automated and manual testing.

- You need to run MPGO tool along with the necessary parameters for the test and then run the test. The test now will be optimized. 

- It is actually the internal binaries that are optimized either fully or partially. 

- Partially optimized binaries are deployed only in automated smoke testing.