What issues should be considered when deciding whether to automate a test? How to generate an automated test script?est Scripts, Scripts

Execution of the test sets lies at the center of any testing process. As the software system or application encounters changes, the defects are located by running both manual as well as automated tests in the project. Also, the quality of the tests is assessed by running the tests itself. 

Issues considered when deciding to go for automation?

- Deciding which all tests have to be automated is a part of the test planning process. 

- Two options namely manual and automated are available for the execution of the tests. 

- If you go with the manual execution of the tests, you can begin with the execution just after you finish defining the test steps. 

- If you go for automating the tests, the test scripts need to be generated and completed. 

- Below we state the issues that should be considered when deciding to go for automation:

1. Do Automate: Only the tests which are data driven or which make use of multiple data values for the same operation, which run with every new version of the application as a measure to check its functionality i.e., regression testing, which are for stress testing i.e., run many times and tests that facilitate the checking of a server system or multi-user client system (load testing) must always be automated.
2. Do not automate: The tests which are meant for a single execution, need to be executed immediately, check for the usability of the tests and whose result cannot be predicted should not be automated.

Steps for generating Automated Test Scripts

Below mentioned steps can be followed for the generation of the automated test scripts:

1. First, click on the test plan tab in order to enable the display of the test plan module.
2. Locate the manual test that you want to automate by selecting the subject folder available at the root of the test plan tree. There click on the find button and the find folder/ test dialog box will open up.
3. Type the name of the test to be searched for in the ‘value to find’ field of the box. Check the ‘include tests’ check box so that the test director can be instructed to look for folders and tests. Finally click on find option. ‘Search results’ dialog box will pop up thus displaying all the possible matches. Click on go to button and the test will be highlighted in the test plan tree. Close this dialog box.
4. Click on the design steps tab in order to display the design steps tab.
5. For generating a test script click on the generate script button. You can choose either of the following options:

a) QUICKTEST_ TEST: For generating an astra quicktest test or quick test professional test.

b)   WR – AUTOMATED: For generating a winrunner test.

The above two options will be available if the corresponding add ins have been installed. Once the test has been automated, the manual test symbol ‘M’ will be replaced by automated test icon.

6. For viewing the test script click on the test script lab. Click on the launch button for displaying and modifying the test script in the testing tool where it was created.

Whenever an automated test is run, the testing tool selected by the tester is opened by the test director automatically and the test is run on either the remote hosts or local machines. The tests can be run either from the execution flow tab or the execution grid tab.

Some problems inherent in creating a software test plan - Part 1

Creating an effective software test plan & test cases is critical for the success of a software project. Sometimes, when teams are in a big hurry in terms of the time available for the project, there is a tendency to see whether some of this time can be compromised in terms of creating the test plans and test cases. Such compromises sometimes pay off (I will not say that teams get into trouble every time), but taking such short cuts means that you are indeed compromising on the quality of the end project, and dramatically increasing the risks that the project will run into serious problems.
So what are some of the problems that happen in creating a test plan:
- The test plan and test cases are not created by the people who have analysed the design. Typically, test plans and test cases should be created well in advance of the actual testing process; the test engineers should study the requirements and design, ensure that all their doubts are answered, and then start work on the testing documents. They should have access to the people who have created the requirements and the design documents, and be able to have a discussion where required. One big problem that occurs is when the test documents are prepared long after the requirements have been created (with the test engineers not being involved in the process). When such a situation happens, any doubts that the test engineers have in terms of the requirements or the design process are left unanswered and carry over to the actual test documents. If the discrepancy is minor, then such problems can go through to the actual product without much of an impact, but where this is a questions of requirements (consider cases such as weapons engineering, aerospace, financial, etc), such errors can be very significant, and yet not easily caught. End user testing does not really cover all the requirements of the users (especially in the case of complex requirements) and such errors can go all the way through to the actual finished product.

More details in part 2 ..

What are the steps to prepare a test plan?

To prepare a test plan, you should have a well thought-out approach. It is not such a difficult task but in immersed with one’s deep thoughts and experience. You should be well aware of the reason of testing before you continue with the testing plan. Test plan give a general idea of testing strategy, assets and list of the projected testing actions. Test plan is generally written in combination with testing policy. The strategy of the test plan also envisages and highlights various levels of the test, which would be carried out while performing the test.

In the simple words it can be said that a test plan provides you information about what you are going to test and which testing procedure is involved in it. Here, the steps for preparing a test plan are being elaborated in six stages, which might be quite helpful for doing the test.

- First stage: For each stage of the test, proper responsibilities should be assigned to the people that who will prepare the design of the test, who will gather the information of each stage, who will carry out the test, and who will verify that the test plan is being executed in a decent way. When these points would be discussed then only the problems will arise, and those should be identified and tackled out at the initial stage.

- Second stage: You should be very well aware of your budget limitations, which involves lump sum idea of the expenditures that would arise on performing the test, which also includes the procurement of the testing equipments.

- Third stage: The requirement of testing equipments should be observed. Every testing equipment, which is required for testing purpose, should be recognized. Many people neglect this part of the whole process, which is also the most important part. If one testing model is required or two or more testing models are required, then it should be planned very at the initial stage. It is to be decided that whether you are going to test on one testing model or on one pre-production portion of hardware? Also in how many shifts the testing would take place? Also how many persons would get involved in the testing process? It the testing equipments should be identified at the early stage then those equipment can be easily procured or can be taken on rent, which would be also be beneficial not only on the financial front but also it prevents the wastage of one’s precious time, as it would allow stress to occupy one’s mind. Moreover, it also involves the availability of the hardware or software material, required for the test.

- Fourth stage: The number of risks and defects that are involved in the testing procedure should be identified at the early stage, which will help in preventing the financial, as well as the any kind of physical loss to the person conducting the test.

- Fifth stage: The testing resource and the testing environment for executing the test should also be considered at the early stage.

- Sixth stage: After executing the test, the test leader must consider and verify, where exactly the test stands in terms of testing actions. He should also checkout that how many tests have become successful and how many tests have failed or proved useless.It helps in improving the mistakes that are committed in the present, so that the same should not be repeated in the future, to avoid any kind of damage to the person performing the test and also in preventing the wastage of the testing material.

What is the outline of test plan? What are test specification items?

There is an outline that should be followed while writing a test plan. It consists of the following:
- The Background
- The Introduction
- The Assumptions
- The Test Items to be tested.
- The Features to be tested.
- The Features not to be tested.
- The Approach that is to be followed.
- Item Pass/Fail Criteria which is an itemized list of expected output and tolerances.
- The Suspension or Resumption Criteria.
- Test Deliverable which includes beside software, what else would be delivered?
- Testing Tasks which consists of functional and administrative tasks.
- Environmental needs like security clearance, office space and equipment, hardware and software requirements.
- Responsibilities.
- Staffing and Training
- Schedule
- Resources
- Risks and Contingencies
- Approvals

Test specifications are developed from test plan and are a part of second phase of test development life cycle. How to implement the test cases is explained through test specifications. It consists of following:
- Case Number
- Title of Test
- ProgName which consists of program name containing test.
- Author
- Date
- Background which consists of Objectives, Assumptions, References, Success Criteria.
- Expected Errors
- References
- Data that flows between the implementation under test and test engine.
- Script

What comprises Test Ware Development : Test Plan - Acceptance Test Plan (ATP)

The client at their place performs the acceptance testing. It will be very similar to the system test performed by the software development unit. Since the client is the one who decides the format and testing methods as part of acceptance testing, there is no specific clue on the way they will carry out the testing. But it will not differ much from the system testing. Assume that all the rules, which are applicable to the system test, can be implemented to acceptance testing also.

Since this is just one level of testing done by the client for the overall product, it may include test cases including the unit and integration test level details.

Test Plan Outline

- BACKGROUND: This item summarizes the functions of the application system and the tests to be performed.
- INTRODUCTION
- ASSUMPTIONS: Indicates any anticipated assumptions which will be made while testing the application.
- TEST ITEMS: List each of the items(programs) to be tested.
- FEATURES TO BE TESTED: List each of the features(functions or requirements) which will be tested or demonstrated by the test.
- FEATURES NOT TO BE TESTED: explicitly lists each feature, function, or requirement which would not be tested and why not.
- APPROACH: Describe the data flows and test philosophy. This section also mentions all the approaches which will be followed at the various stages of the test execution.
- ITEM PASS/FAIL CRITERIA: Itemized list of expected output and tolerances.
- SUSPENSION/RESUMPTION CRITERIA: Must the test run from start to finish? Under what circumstances it may be resumed in the middle? Establish check-points in long tests.
- TEST DELIVERABLES: What, besides software, will be delivered? It includes test report and test software.
- TESTING TASKS: It includes functional and administrative tasks.
- ENVIRONMENTAL NEEDS: It includes security clearance, office space and equipment and hardware/software requirements.
- RESPONSIBILITIES: It includes what are the tasks in section 10? What does the user do?
- STAFFING AND TRAINING
- SCHEDULE
- RESOURCES
- RISKS AND CONTINGENCIES
- APPROVALS

The schedule details of the various test pass such as unit tests, integration tests, system tests should be clearly mentioned along with estimated efforts.

What comprises Test Ware Development : Test Plan - System test Plan

The system test plan is the overall plan carrying out the system test level activities. In the system test, apart from testing the functional aspects of the system, there are some special testing activities carried out, such as stress testing etc. The following are the sections present in system test plan:

- What is to be tested?
This section defines the scope of system testing, very specific to the project. Normally, the system testing is based on the requirements. All the requirements are to be verified in the scope of the system testing. This covers the functionality of the product. Apart from this, what special testing is performed are also stated here.

- Functional groups and the sequence
The requirements can be grouped in terms of the functionality. Based on this, there may be priorities also among the functional groups. For example, in a banking application, anything related to customer accounts can be grouped into one area, anything related to inter-branch transactions may be grouped into one area etc.

- Special Testing Methods
This covers the different special tests like load/volume testing, stress testing, interoperability testing etc. These testing are to be done based on the nature of the product and it is not mandatory that every one of these special tests must be performed for every product.

Apart from above sections, the following sections are also addressed:
- System Testing Tools
- Priority of functional groups
- Naming Convention for test cases
- Status reporting mechanism
- Regression test approach
- ETVX criteria
- Build/Refresh Criteria

What comprises Test Ware Development : Test Plan - Integration Test Plan

The integration test plan is the overall plan for carrying out the activities in the integration test level, which contains the following sections:

- What is to be tested?
This section clearly specifies the kinds of interfaces fall under the scope of testing internal, external interfaces, with request and response is to be explained. This need not go deep in terms of technical; details but the general approach how the interfaces are triggered.

- Sequence of Integration
When there are multiple modules present in an application, the sequence in which they are to be integrated will be specified in this section. In this, the dependencies between the modules play a vital role. If a unit B has to be executed, it may need the data that is fed by unit A and unit X. In this case, the units A and X have to be integrated and then using the data, the unit B has to be tested. This has to be stated to the whole set of units in the program. Given this correctly, the testing activities will lead to the product, slowly building the product, unit by unit and then integrating them.

- List of modules and interface functions
There may be N number of units in the application but the units that are going to communicate with each other, alone are tested in this phase. If the units are designed in such a way that they are manually independent, then the interfaces do not come into picture.This is almost impossible in any system, as the units have to communicate to other units, in order to get different types of functionalities executed. In this section, we need to list the units and for what purpose it talks to the others needs to be mentioned. This will not go into technical aspects, but at a higher level, this has to be explained in plain English.

Apart from above sections, it also includes:
- Integration Testing Tools
- Priority of Program Interfaces
- Naming Convention for test cases
- Status reporting mechanism
- Regression test approach
- ETVX criteria
- Build/Refresh criteria.

What comprises Test Ware Development : Test Plan - Unit Test Plan

The test strategy identifies multiple test levels, which are going to be performed for the project. Activities at each level must be planned well in advance and it has to be formally documented. Based on the individual plans only, the individual test levels are carried out.
The plans are to be prepared by experienced people only. In all test plans, the (ETVX) Entry-Task-Validation-Exit criteria are to be mentioned. Entry means the entry point to that phase. Task is the activity that is performed. Validation is the way in which the progress and correctness and compliance are verified for that phase. Exit tells the completion criteria of that phase, after the validation is done.

ETVX is a modeling technique for developing worldly and atomic level models. It is a task based model where the details of each task are explicitly defined in a specification table against each phase i.e. Entry, Exit, Task, Feedback In, Feedback Out, and measures.
There are two type of cells, unit cells and implementation cells. The implementation cells are basically unit cells containing the further tasks. A purpose is also stated and the viewer of the model may also be defined e.g. to management or customer.

Types of Test Plan

Unit Test Plan (UTP)
The unit test plan is the overall plan to carry out the unit test activities. The lead tester prepares it and it will be distributed to the individual tester, which contains the following sections:

- What is to be tested?
The unit test plan must clearly specify the scope of unit testing. In this, normally the basic input/output of the units along with their basic functionality will be tested. In this case, mostly the input units will be tested for the format, alignment, accuracy and the totals.

- Sequence of testing
The sequence of test activities that are to be carried out in this phase are to be listed in this section. This includes, whether to execute positive test cases first or negative test cases first, to execute test cases based on the priority, to execute test cases based on test groups etc.

- Basic functionality of units
The independent functionalities of the units are tested which excludes any communication between the unit and other units. The interface part is out of scope of this test level.

Apart from these, the following sections are also addressed:
- Unit testing tools
- Priority of program units
- Naming convention for test cases
- Status reporting mechanism
- Regression test approach
- ETVX criteria

Validation Phase - Beta Testing - Objectives

The beta testing is conducted at one or more customer sites by the end-user of the software. The beta test is a live application of the software in an environment that cannot be controlled by the developer. The software reaches beta stage when most of the functionalities are operating. The software is tested in customer's environment, giving the user an opportunity to exercise the software, find the errors so that they could be fixed before product release. Beta testing is a detailed testing and needs to cover all the functionalities of the product and also the dependent functionality testing. It also involves the user interface testing and documentation testing. Hence, it is essential that this is planned well and the task accomplished. The test plan document has to be prepared before the testing phase is started, which clearly lays down the objectives, scope of test, tasks to be performed and the test matrix which depicts the schedule of testing.

The objectives of beta testing is to:
- evaluate software technical content.
- evaluate software ease of use.
- evaluate user documentation draft.
- identify errors.
- report errors/findings.

The role of a test lead is to provide test instruction sheet that describes items such as testing objectives, steps to follow, data to enter, functions to invoke and to provide feedback forms and comments.
The role of a tester is to understand the software requirements and the testing objectives and carry out the test cases and report defects.

The four phases of testing management process in Test Director - Part II

TestDirector offers an organized framework for testing applications before
they are deployed. While using TestDirector, the testing management process can be defined using the following four steps :

- Specifying the test requirements.
- Planning the tests.
- Run the tests in manual or automatic mode.
- To analyze the defects.

Run the tests in manual or automatic mode

This phase is the most crucial phase of testing process. A test set is a group of tests in a TestDirector project designed to achieve specific testing goals. TestDirector enables you to control the execution of tests in a test set by setting conditions and scheduling the date and time for executing your tests. After you define test sets, you can begin to execute your tests. When you run a test automatically, TestDirector opens the selected testing tool, runs the test, and exports the test results to TestDirector. It includes defining test sets, adding tests to a test set, scheduling test runs, running tests manually, running tests automatically.

To analyze the defects

Locating and repairing defects is an essential phase in application development. When a defect is submitted it is tracked through the new,open,fixed,and closed stages. A defect may also be rejected or reopened after it is fixed. It includes :
- How to Track Defects
- Adding New Defects
- Matching Defects
- Updating Defects
- Mailing Defects
- Associating Defects with Tests
- Creating Favorite Views

The four phases of testing management process in Test Director - Part I

TestDirector offers an organized framework for testing applications before
they are deployed. While using TestDirector, the testing management process can be defined using the following four steps :

- Specifying the test requirements.
- Planning the tests.
- Run the tests in manual or automatic mode.
- To analyze the defects.

Specifying the test requirements

Testing process is started by specifying the testing requirements in TestDirector’s Requirements module. It provides the test team with the foundation on which the entire testing process is based. A requirement tree is created to define the requirements. This is a graphical representation of your requirements specification,
displaying your requirements hierarchically. After you create tests in the Test Plan module, you can link requirements to tests. A track of your testing needs at all stages of the testing process can be kept. It includes defining, viewing, modifying and converting requirements.

Planning the tests

Testing goals can be determined after the requirements are defined. After this a test plan tree is built to divide the application into testing units or subjects. Tests are defined for each subject. Actions are specified for each test step that are performed on application. Links can be added to keep track of the relationship between your tests and your requirements. After you design your tests, you can decide which tests to automate and after that a test script can be generated. It includes developing a test plan tree, designing test steps, copying test steps, calling tests with parameters, creating and viewing requirements coverage and generating automated test scripts.

What are the four phases of TestDirector ?

TestDirector use can be divided into four phases:
- Test Requirements Management
Requirements Manager is used to link the requirements with the tests to be carried out. Each requirement in the SRS has to be tested at least once. In SRS, the functional and performance requirements are specified. Functional requirements are generated from use-case scenarios. Performance requirements are dependent on the application.
- Test Planning
In test planning, the QA manager does a detailed planning and addresses the following issues:
- Hardware and software platforms.
- Various tests to be formed.
- Time schedule to conduct the tests.
- Roles and responsibility of the persons associated with the project.
- Procedure for running the test.
- Various test cases to be generated.
- Procedure for tracking the progress of testing.
- Documents to be generated during testing process.
- Criteria for completion of testing.
Test design is done during test planning phase which involves defining the sequence of steps to execute a test in manual testing. The test plan is communicated to all the test engineers and also the development team.
- Test Execution
The actual testing is carried out based on the test cases generated, either manually or automatically. In case of automated testing, the test scheduling is done as per plan. A history of all test runs is maintained and audit trail, to trace the history of tests and test runs, is also maintained. Test sets are created. A test set is a set of test cases. In addition execution logic is also set. The logic specifies what to do when a test fails in a series of tests.
- Test Results Analysis
In this phase, test results are analyzed i.e. which tests passed and which failed. For the tests that failed, an analysis is carried out as to why they failed. Bug is classified based on severity. A simple way of classification is critical, major, minor. A more detailed way of classification is:
- Cosmetic or GUI related.
- Inconsistent performance of application.
- Loss of functionality.
- System crash.
- Loss of data.
- Security Violation.
The bug report is stored in a database. The privileges to read, write, and update the database need to be decided by the QA manager. Based on the bug tracking and analysis tools, QA manager and the project manager can take the decision whether the software can be released to the customer or still more testing is required.