Data Driven Testing - Automation Frameworks

Data driven testing is a very important aspect of test automation. In data-driven testing, the scripts read data from an external storage site like a file or database, rather than use values hard-coded in the script. A data-driven test includes the operations like retrieving input data from storage, entering data in an application form, verifying the results and continuing with the next set of input data. It significantly increases test coverage and also helps reduce the need to create more tests with different variables.

The data-driven testing approach can be used with unit, functional and load testing. Data-driven testing separates the test data from the test itself. This makes both the test and the data more flexible and reusable and certainly much more easy to maintain.

Data driven scripts are those application-specific scripts captured or manually coded in the automation tool’s proprietary language and then modified to accommodate variable data. Variables will be used for key application input fields and program selections allowing the script to drive the application with external data supplied by the calling routine or the shell that invoked the test script.

Data-driven testing is built around the need to test how an application deals with a range of inputs. An important use of data driven tests is ensuring that applications are tested for boundary conditions and invalid input. A data-driven test alleviates the pains of testing with large sets of data by separating test input from the test itself.

Some reasons why test automation projects fail, and some precautions to take to reduce these chances

One of the most important reasons why test automation projects fail is because not enough planning is give for such projects. In so many organizations, people talk about automation, decide that bringing in software automated testing will solve things for them. One of the prime reasons is that there is a drive towards bringing in software testing automation as a solution, looking at the benefits, but not keeping in mind that an organization needs to also plan carefully for software automation.
So, for example, when a company brings in software automation, it would need to do the following steps, else, it will face problems during the implementation, and could actually reach a situation where the organization considers that the project is a failure.
- Plan the required resources that you need for implementation of software automation, which can be different from the profile needed for testing purposes
- In the initial stages, the amount of effort needed for software testing automation would be more, since the company needs to keep at the regular manual testing and also start building up the effort needed to do the automation of its test cases and test plans
- In many cases, there is a piecemeal implementation, in terms of doing conversion of cases one by one to become automated, but a comprehensive framework is not employed for this purpose. Without this level of framework implementation, as the amount of automation increases, the maintenance of such test cases becomes more complicated and intensive
- In many cases, the organizations have not really thought through the needs for implementation in terms of creating a new sub-structure within the testing team for automation, and this creates strains since the black box testing and automation teams have different needs
- For doing automation, there is a need to even modify existing testing processes (including the plan for creating test plans and test cases), so that creating automation becomes an intrinsic part of the testing process
- Set expectations of the executives with respect to the time frames and effort needed for automation; I know a case where the senior manager wanted the implementation of an automation project without increases in resources and without existing testing getting impacted. This is an impossible task.

Test Automation Framework: What is the Hybrid Test Automation Framework

In previous articles, we learned about the following Automation frameworks:
- Data Driven Testing (link)
- Test Library Architecture Framework (link)
- Keyword-driven/table-driven testing (link)
- Test script modularity (link)
This post covers the 5th such test automation framework, called Hybrid Test Automation Framework. Just like the name suggests, this is what people would ask for, why can't you combine the benefits and strong points of the above test automation frameworks and try to remove their weaknesses, and that is what you get. This is called the Hybrid Test Automation Framework. It is one of the most successful automation frameworks, and is also one of the frameworks that other frameworks eventually mature into.
So, since the keyword driven architecture has some powerful benefits in the form of libraries and utilities, this framework allows the data driven scripts to take advantage of these libraries and utilities; and also make these data driven scripts smaller and reduce the risk of their failure. This framework has many utilities that allow the conversion of currently used scripts into the equivalent of keyword driven (when needed).
Some of the properties of the Hybrid Test Automation Framework are: Core Data Driven Engine, the Component Functions, the Support Libraries, and the Application Map (App Map).

Test Automation Framework: What is Data Driven testing (definition and more) ..

In the previous posts, we have been talking about different testing automation frameworks. In the current post, we talk about another test automation framework, called 'Data Driven Testing', including definition and some details.
In this framework, the variables are used for testing of both the output verification values and input values. These values are read from data files (different kind of data objects such as datapools, ODBC sources, csv files, Excel files, DAO objects, ADO objects, and such), and are then loaded into variables (and these variables could be used in scripts that are either manually written or recorded). The test script in turn is supposed to take care of all the process of moving through the application, opening and reading of the data files, and logging of test results.
This may sound similar to the table driven testing (and it is similar in the sense that the test case is contained not in the test script, but in a data file), with the script just being used for moving through the workflow (navigating through the application). The difference is that in this case the data is stored in data objects, not in tables (and the navigation is not stored in the data).
What are some of the advantages of using this automation framework ? There is a reduction in the number of scripts you need for your overall test cases, and if you have a need to accommodate bugs in your workflow, then this framework is the one to use. It is also very handy in terms of effort required for maintenance.

Test Automation Framework: What is The Test Library Architecture Framework (including definition)

In the previous 2 posts on the subject of Test Automation Frameworks (Keyword Driven / Table Driven, Test Script Modularity), we have covered 2 of the models used for test automation frameworks. In this post, we will cover another test automation framework called 'Test Library Architecture'.
The Test Library Architecture is similar to the Test Script Modularity framework using the same level of abstraction; the difference being that the application being tested is broken down into functions and procedures (or you could break it down into objects and methods rather than scripts). The tester will need to create libraries (in the form of SQABasic libraries, APIs, DLLs, and such) that are supposed to be representing the modules, and functions of the application; once these are created, they are called directly from the test case script.
One of the advantages of this framework is that it provides a high degree of modularization, and increases the ease of maintenance of the automation test case. You can simply modify the script if a control is changed by modifying the library file.

Test Automation Frameworks: What is Keyword-driven/table-driven testing (including definition)

For a few of the last posts, we have been looking at more details about test automation, including the benefits of automation, some scenarios in which we should not use automation, and the scenarios in which we should use automation. In addition, we started discussions around the use of test automation frameworks and how they prove to be more beneficial than just creation test cases as you do recording of your test scenarios. In this post, let us consider one of the test automation frameworks, based on "Keyword-driven/table-driven testing".
'Keyword driven testing' and 'table driven testing' seem like 2 different sets of words, but they actually are used to refer to the same method. They denote an application independent framework, which requires the development of data tables and keywords. These data tables and keywords are independent of the test automation took being used, and also independent of the test scripts used to drive the application that is being tested. These keyword driven tests look very similar to the manual test cases. When keyword driven tests are being used, a table is used to document the functionality being used, and this functionality is also mapped through step by step instructions for each test. The entire testing process is driven based on data.

Benefits of keyword driven testing:
- If the tester needs to be quickly on the job, extensive training on the tool can be done later, but at the instant of testing, the tester needs to know the keywords, and the format of the test plan.
- The Scripting language can be written by somebody who has expertise on the scripting language, and this activity can happen earlier to the test plan. The tester does not have to be bothered about the scripting process.
- A spreadsheet format can be used for writing the detail test plan.

Some problems with this technique:
- If there are a large number of keywords, then the tester needs to learn all these, and this initial effort can take time. Once done, then this is no longer a constraint.
- You need people skilled in using the Scripting language of the tool being used.

Test automation framework: What is Test script modularity (definition and some details)

In the previous post (Test Automation Frameworks), I started out by doing a short definition of Test Automations Frameworks, some benefits, and also listed out the 5 different types of Test Automation Frameworks that are currently there (and based on the number of tools available, the combination can be much more since there are many high end and complex tools available for test automation). In this post, I will talk more about one of these, which is called "Test script modularity".
A simple definition for the Test Script Modularity Framework: This framework is the most basic of the various test automation frameworks, with the ideology being towards the creation of a number of small and independent test scripts that in turn represent the modules, sections and functions of the application that is under testing. Once these scripts are created, they are then added in a hierarchical fashion to create larger tests, with the aim of creating test cases.
What is the basic principle behind this ? It is a basic principle of design to create a layer (or to be more technical, an abstraction layer) for a component. This layer in turn ensures that the component is available to the rest of the application in a way that even when the component is modified, the rest of the application is not affected. This concept is one of the key concepts of the test script modularity framework.
One of the key advantages of this framework is that it results in a high degree of modularization, and ensures that the test suite is easily maintainable. By encapsulating all components, when a component changes, you don't have to change the other components or the test cases that call this component.

What is a test automation framework and why you should go in for one ..

In the previous post (problems with manual automation testing), I had talked about why it makes sense to go in for using a strategy for test automation, and to use test automation frameworks. Working without a strategy results in issues dealing with maintenance, loss of efficiency, and ever expanding list of files.
So what is a test automation framework ? Well, if you were to take a basic description, then it is something as simple as a set of concepts, practices, tools that provide support for automated software testing; with one of the main benefits of adopting such practices resulting in lower costs of maintenance. Using a test automation framework means that the need to update a test case can be done with minimal effort, with just the actual test script needing to be updated, and everything else remaining the same. It also helps in documenting the overall set of automated test scripts, thus ensuring that even with team turnover, the process still keeps on working with minimal disturbance.
Let us take some of the test automation frameworks that are used, and then define each of these.
- Test script modularity:
- Keyword-driven/table-driven testing
- Data-driven testing
- Test library architecture
- Hybrid test automation

We will move with these definitions in the next post, since each of them needs to be covered in some detail ..

Some of the challenges involved in the process of using test automation tools

Description of automated decision making problems

In the past decade or so, test automation has jumped up to being a solution touted by a number of experts for improving quality. Even when you consider the case of institutions that provide training on testing concepts, their courses that cost the most are those related to automated testing tools and how to become an expert on using them. However, it should be considered that taking a decision to go for automated testing needs to be a thoroughly considered decision, with the cost and benefits needing to be calculated (in one small company, I realized that the decision to go in for automated testing was made based on the fact that the company had hired 2 people who had done some amount of automated testing earlier). After all, it is very easy to mockup a Excel file and Powerpoint with the benefits of introducing automated testing, but as I said earlier, this should not be a decision taken in haste.

Here are a few points that you should be aware of before you go in for automated testing tools

- Resources: If you are going in for automated testing, then consider the fact that you will need to have more people for your testing team (this also means that some of the people you need will have to have experience with using automated testing). The typical work for a testing team is normally challenging enough, without having to spend the additional time required to automates the test cases.
- Time required: Automation of test cases can take some time to happen. It takes time to learn how to use a testing tool, it takes time to create a proper automation framework, and then starting to write the proper automation scripts and cover all your test cases (or atleast the desired level of test cases that are considered proper for automation).
- Skill set differences. For a team comprising of black box testers, as well as a manager of such people, having a team of people expert in automation testing means a different skill sets. This can cause various morale issues, as well as a lack of understanding of the various specialized needs of the testing team.
- When the application needs to change frequently. During the course of the product development process, the product can change, including the UI. I have seen cases in the past where the UI changes, and the product automation no longer works. In such cases, the script needs to be re-worked, which needs more effort.
- Maintainability of such scripts. Typically, over a period of time, automation scripts need to be maintained, and as people move on in the company, maintaining the same scripts (and ensuring that new people are able to understand the entire structure of the test automation framework) can take increasing periods of effort and time.
- Interlap between test automation areas and manual testing areas. This can be hard to figure for many products where the product functionality cannot be broken down into clean separable areas. In such cases, duplication of testing efforts can happen.
- Expensive. From my experience, the cost needed for test automation is not a one-time effort. We had bought a fairly expensive tool, but then realized that we still had to pay around 40% of the cost annually for the AMC and for the ability to get regular updates.
- Platform compatibility. This was something new that we learnt. We had a product that was only available on Windows, and after a couple of cycles, we also moved onto the Mac and onto Linux because of the potential of getting more sales. And then we realized that our testing tool did not support Linux, and we either needed to get a new tool for Linux (and which meant that we needed to port all our test scripts), or we needed to go in for manual testing (in which case also we did not get the benefits of automated testing on Linux).