Automation in Software Testing

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. There can be two types of testing manual and automated. The automation of software testing is becoming more and more popular but at the same time manual testing cannot be ignored. In this article we shall be mostly focusing on the merits and demerits of automated software testing. As chopping and adding of requirements is nothing new in today’s software business and the testing window is getting smaller there is a realization of a greater need for test automation.
The sad part about automation testing is the expectations that the managers have from these testing models. It’s a general belief that automation can help find more bugs which is not true. The efficiency of the test scripts is solely dependent on the efficiency on the test cases that make the test scripts. People expect that since they have introduced automated testing, they can do away with or at least reduce the manual testing which is a “BIG MISTAKE”. As already mentioned it is the test cases that define the efficiency of the automated test scripts and these test cases are written by these manual testers hence if you think of putting away with them the customers might also think of putting away your product.
Continuing regarding the unrealistic expectations that people have with automation testing, part of the blame for these expectations goes to the fact that when vendors give demos of these products they only tell you what you want to hear and not the reality about the difficulties faced when you try to use the tool for your application. Proper planning is essential when you go for the selection of any automated tool. It is very important that the people who are actually going to use the tool get a hands-on experience of the tool. This might sometimes be impractical because of the tight work schedules and time bound deadlines but if applied it’s a very useful technique for the selection.
Many test automation tools provide record and playback features that allow users to interactively record user actions and replay them back any number of times, comparing actual results to those expected. This approach can be applied to any application that has a graphical user interface. However if the developers continuously keep changing the GUI even when it’s not called for it indicates towards lack of process in place and hence the automation test will have to be configured again and again.
People have to be educated about the advantages and limitations of automation testing techniques. It is important that pitfalls of automation testing are properly evaluated to avoid inconvenience at a later stage. It should be taken care of that the selected scripts are compatible to your applications. There is no doubt that automation testing is an asset in the armory of the testing team but without proper knowledge and understanding it can even turn out to be a negative catalyst.

Some reasons why test automation projects fail, and some precautions to take to reduce these chances

One of the most important reasons why test automation projects fail is because not enough planning is give for such projects. In so many organizations, people talk about automation, decide that bringing in software automated testing will solve things for them. One of the prime reasons is that there is a drive towards bringing in software testing automation as a solution, looking at the benefits, but not keeping in mind that an organization needs to also plan carefully for software automation.
So, for example, when a company brings in software automation, it would need to do the following steps, else, it will face problems during the implementation, and could actually reach a situation where the organization considers that the project is a failure.
- Plan the required resources that you need for implementation of software automation, which can be different from the profile needed for testing purposes
- In the initial stages, the amount of effort needed for software testing automation would be more, since the company needs to keep at the regular manual testing and also start building up the effort needed to do the automation of its test cases and test plans
- In many cases, there is a piecemeal implementation, in terms of doing conversion of cases one by one to become automated, but a comprehensive framework is not employed for this purpose. Without this level of framework implementation, as the amount of automation increases, the maintenance of such test cases becomes more complicated and intensive
- In many cases, the organizations have not really thought through the needs for implementation in terms of creating a new sub-structure within the testing team for automation, and this creates strains since the black box testing and automation teams have different needs
- For doing automation, there is a need to even modify existing testing processes (including the plan for creating test plans and test cases), so that creating automation becomes an intrinsic part of the testing process
- Set expectations of the executives with respect to the time frames and effort needed for automation; I know a case where the senior manager wanted the implementation of an automation project without increases in resources and without existing testing getting impacted. This is an impossible task.

Process for creating an automation test framework and how to go about automation testing

Every automation tool will give you the ability to record a series of actions so that these can then be played back and you can create a script out of these. However, this is a very basic level of test automation, which does not provide you any flexibility. This process has its uses when you are creating a simple testing strategy whereby you have to take a few scenarios where you do not expect variations or changes, and you can record these sequences and then use them again and again. However, when you need to make changes or modify values, then the cost of doing so starts increasing tremendously. This is a time when you should evaluating the use of creating testing frameworks for you automated testing. Just the process of starting to design a test automation framework will ensure that you are starting to work through your requirements methodically, and will prevent your team from ending up in a mess (a typical example of a mess is when you have a huge number of unconnected automation scrips with their own needs for maintenance, with poor documentation leading to a disaster when people change).
How do you go about creating test automation frameworks ? Before we even go down this route, we should consider some of the benefits that you would get if you were to have an automation framework / or in the examples below, more of an automation strategy:
- You start to separate your data from tests, something that makes it easier to manipulate different types of data (very useful when you want to test the same test with a wide range of data)
- You can look towards reusing functions (building reusable functions can eventually help in saving a lot of time and make the building of these functions more efficient)
- You prevent a situation where you end up with a huge amount of test cases with high maintenance requirements; maintaining these scripts become easier (very useful when you have teams with high personnel changes or attrition rates)
- You will also start evaluating as to which stage is it practical to start building an automation test plan - in some cases, you may delay till the major elements of UI changes are done and over
- Refinements to the manual test cases are avoided later, instead the necessary adaptation of test cases to automation needs is done when they are getting generated; this saves a lot of effort later

Automation software - Winrunner - automated functional GUI testing tool

For many years, Winrunner, by Mercury Interactive, was a leading software for automation testing. It was fairly expensive to use, based on our experience. Buying a license to use the software was expensive, but this was not the only cost. It used to cost a fairly large fraction of the initial cost to retain the AMC (Annual Maintenance Contract), since that would allow you to ensure that you were able to obtain the regular updates as well as get support from the engineers of the company. However, this software is not available for sale anymore; it was for sale when the software was with Mercury Interactive; however, when Mercury was taken over by Hewlett Packard (HP) in 2006, the software was retired after 2 years, with users being suggested to move over to another software called ‘Functional testing software’.
For the time when Winrunner was available, our teams had a fairly good experience with it. The level of training required for using the software was not very extensive, and could learn while using the software (although some amount of scripting knowledge was very useful). So, how would the software work ? Well, the software depended on the functionality of letting users record and play back their UI based testing and recording these interactions as test scripts.
The software worked by emulating user actions, and then customize the created scripts to meet their actual requirements (which is what we mentioned earlier that some amount of scripting experience is useful). And then you could do more steps that you normally find in code debuggers such as adding checkpoints, such that testers can compare the actual results versus the expected results from the testing. And you would get the bonus of being able to do additional steps such as be able to check database integrity and also check transaction accuracy.
You also had add-ins that allowed interfaces to various platforms such as C++, C, Visual Basic, Forte, Delphi, Smalltalk, Baan, Browsers such as IE and AOL, etc.
Link to Winrunner User Guide.

Some of the challenges involved in the process of using test automation tools

Description of automated decision making problems

In the past decade or so, test automation has jumped up to being a solution touted by a number of experts for improving quality. Even when you consider the case of institutions that provide training on testing concepts, their courses that cost the most are those related to automated testing tools and how to become an expert on using them. However, it should be considered that taking a decision to go for automated testing needs to be a thoroughly considered decision, with the cost and benefits needing to be calculated (in one small company, I realized that the decision to go in for automated testing was made based on the fact that the company had hired 2 people who had done some amount of automated testing earlier). After all, it is very easy to mockup a Excel file and Powerpoint with the benefits of introducing automated testing, but as I said earlier, this should not be a decision taken in haste.

Here are a few points that you should be aware of before you go in for automated testing tools

- Resources: If you are going in for automated testing, then consider the fact that you will need to have more people for your testing team (this also means that some of the people you need will have to have experience with using automated testing). The typical work for a testing team is normally challenging enough, without having to spend the additional time required to automates the test cases.
- Time required: Automation of test cases can take some time to happen. It takes time to learn how to use a testing tool, it takes time to create a proper automation framework, and then starting to write the proper automation scripts and cover all your test cases (or atleast the desired level of test cases that are considered proper for automation).
- Skill set differences. For a team comprising of black box testers, as well as a manager of such people, having a team of people expert in automation testing means a different skill sets. This can cause various morale issues, as well as a lack of understanding of the various specialized needs of the testing team.
- When the application needs to change frequently. During the course of the product development process, the product can change, including the UI. I have seen cases in the past where the UI changes, and the product automation no longer works. In such cases, the script needs to be re-worked, which needs more effort.
- Maintainability of such scripts. Typically, over a period of time, automation scripts need to be maintained, and as people move on in the company, maintaining the same scripts (and ensuring that new people are able to understand the entire structure of the test automation framework) can take increasing periods of effort and time.
- Interlap between test automation areas and manual testing areas. This can be hard to figure for many products where the product functionality cannot be broken down into clean separable areas. In such cases, duplication of testing efforts can happen.
- Expensive. From my experience, the cost needed for test automation is not a one-time effort. We had bought a fairly expensive tool, but then realized that we still had to pay around 40% of the cost annually for the AMC and for the ability to get regular updates.
- Platform compatibility. This was something new that we learnt. We had a product that was only available on Windows, and after a couple of cycles, we also moved onto the Mac and onto Linux because of the potential of getting more sales. And then we realized that our testing tool did not support Linux, and we either needed to get a new tool for Linux (and which meant that we needed to port all our test scripts), or we needed to go in for manual testing (in which case also we did not get the benefits of automated testing on Linux).

Some cases in which there is low or negligible benefit of going in for automation of test cases

Automation of test cases can lead to major benefits for teams that implement the automation of their test cases in an effective way, and with a proper strategy. However, this is not an absolute, and there are many situations when it may not be worth the effort to do an automation. One needs to do an analysis of the cost / benefits of going in for automation (but be sure to include long term benefits as well - such as for a regular long cycle, it may be worth going in for automation if it is beneficial in the long term). Some of the situations in which it may not be beneficial to go in for a test automation strategy are:
- Calculate the effort required for writing the test cases for automation: Just because you have test cases for manual testing does not mean that you can go in for automated testing. You would need to convert those test cases into automation script, and depending on the size of the product, this can be a sizable effort. The effort required for this (converted into number of people required) is a prime input into the calculation of whether test automation should be done.
- How likely is it that there will be a need for automation in the future. Automation has much more value if there is a constant ongoing effort for testing. If there is not likely to be much more need of testing of the product, then the benefit that you can get by automating the test cases is significantly reduced.
- Diversity of cases. This is related to the overall effort required for automation. If there are less chances of situations of load testing or testing of multiple input parameters, then some of the benefits of going in for automation of testing reduce
- When the test software is expensive. Say you have a small product or project that sells for a few hundred dollars or less, and is also not sold in appreciable numbers, then it does not make sense to go in for the more commercially available automation softwares that can be fairly expensive
- When the test case is likely to change during the cycle. If you consider the case of an automation software that is UI based, and the UI of the target application keeps on getting modified, then the cost of modifying the automation test scripts to take the changes in the UI can increase the costs involved in an automation strategy
- Getting people to use these automation testing tools is expensive. Using these automation testing tools can take some level of expertise, and it can take some amount of experience to master using such tools (even though many of them claim that they are simple to use). People who are skilled in using such tools can be expensive to hire, and retain.