Some of the challenges involved in the process of using test automation tools

Description of automated decision making problems

In the past decade or so, test automation has jumped up to being a solution touted by a number of experts for improving quality. Even when you consider the case of institutions that provide training on testing concepts, their courses that cost the most are those related to automated testing tools and how to become an expert on using them. However, it should be considered that taking a decision to go for automated testing needs to be a thoroughly considered decision, with the cost and benefits needing to be calculated (in one small company, I realized that the decision to go in for automated testing was made based on the fact that the company had hired 2 people who had done some amount of automated testing earlier). After all, it is very easy to mockup a Excel file and Powerpoint with the benefits of introducing automated testing, but as I said earlier, this should not be a decision taken in haste.

Here are a few points that you should be aware of before you go in for automated testing tools

- Resources: If you are going in for automated testing, then consider the fact that you will need to have more people for your testing team (this also means that some of the people you need will have to have experience with using automated testing). The typical work for a testing team is normally challenging enough, without having to spend the additional time required to automates the test cases.
- Time required: Automation of test cases can take some time to happen. It takes time to learn how to use a testing tool, it takes time to create a proper automation framework, and then starting to write the proper automation scripts and cover all your test cases (or atleast the desired level of test cases that are considered proper for automation).
- Skill set differences. For a team comprising of black box testers, as well as a manager of such people, having a team of people expert in automation testing means a different skill sets. This can cause various morale issues, as well as a lack of understanding of the various specialized needs of the testing team.
- When the application needs to change frequently. During the course of the product development process, the product can change, including the UI. I have seen cases in the past where the UI changes, and the product automation no longer works. In such cases, the script needs to be re-worked, which needs more effort.
- Maintainability of such scripts. Typically, over a period of time, automation scripts need to be maintained, and as people move on in the company, maintaining the same scripts (and ensuring that new people are able to understand the entire structure of the test automation framework) can take increasing periods of effort and time.
- Interlap between test automation areas and manual testing areas. This can be hard to figure for many products where the product functionality cannot be broken down into clean separable areas. In such cases, duplication of testing efforts can happen.
- Expensive. From my experience, the cost needed for test automation is not a one-time effort. We had bought a fairly expensive tool, but then realized that we still had to pay around 40% of the cost annually for the AMC and for the ability to get regular updates.
- Platform compatibility. This was something new that we learnt. We had a product that was only available on Windows, and after a couple of cycles, we also moved onto the Mac and onto Linux because of the potential of getting more sales. And then we realized that our testing tool did not support Linux, and we either needed to get a new tool for Linux (and which meant that we needed to port all our test scripts), or we needed to go in for manual testing (in which case also we did not get the benefits of automated testing on Linux).