Introduction to Formalized Exploratory Testing ....

In normal testing style, the test process is planned well in advance before the actual testing begins. In this, the test design is separated from the test execution phase. Many times the test design and test execution is entrusted on different persons. Exploratory testing should not be confused with the dictionary meaning of ad hoc. Ad hoc testing normally refers to a process of improvised, impromptu bug searching.

Formalized Exploratory Testing

A structured and reasoned approach to exploratory testing is termed as formalized exploratory testing. This approach consists of specific tasks, objectives, and deliverables that make it a systematic process. Using the systematic approach, an outline of what to attack first, its scope, the time required to be spent etc is achieved. The approach might be using simple notes to more descriptive charters to some vague scripts. By using the systematic approach, the testing can be more organized focusing at the goal to be reached. Thus, solving the problem where the pure exploratory testing might drift away from the goal.

The formalized approach used for the exploratory testing can vary depending on the various criteria like resource, time, the knowledge of the application available etc. Depending o this criteria, the approach used to attack the system will also vary. It may involve creating the outlines on the notepad to more sophisticated way by using charters etc.

Exploratory Testing - Introduction and Overview

Exploratory testing is an interactive process of concurrent product exploration, test design and test execution. This testing is defined as simultaneous test design, test execution and bug reporting. In this approach, the tester explores the system without having any prior test cases or test scripts. It is also called as ad hoc testing, gorilla testing or intuitive testing. In operational terms, exploratory testing is an interactive process of concurrent product exploration, test design, and test execution. The outcome of an exploratory testing session is a set of notes about the product, failures found, and a concise record of how the product was tested. When practiced by trained testers, it yields consistently valuable and audit-able results. Every tester performs this type of testing at one point or the other. This testing totally depends on the skill and creativity of the tester. Different testers can explore the system in different ways depending on their skills. Thus, the tester has a very vital role to play in exploratory testing.

A systematic approach of exploratory testing can also be used where there is a plan to attack the system under test. This systematic approach of exploring the system is termed as formalized exploratory testing. It is a powerful approach but it has not got recognition and is often misunderstood and not gained the respect it needs. In many situations, it can be more productive than the scripted testing.
This testing believes in concurrent phases of product exploration, test design and test execution. It is categorized under black box testing. It is basically a free-style testing approach where you do not begin with the usual procedures of elaborate test plans and test steps. The test plan and strategy is very well in the tester's mind. The tester asks the right question and judges the outcome. During this phase, the tester is actually learning the product as he tests it. It is interactive and creative. A conscious plan by the tester gives good results.

Validation Phase - Beta Testing - Objectives

The beta testing is conducted at one or more customer sites by the end-user of the software. The beta test is a live application of the software in an environment that cannot be controlled by the developer. The software reaches beta stage when most of the functionalities are operating. The software is tested in customer's environment, giving the user an opportunity to exercise the software, find the errors so that they could be fixed before product release. Beta testing is a detailed testing and needs to cover all the functionalities of the product and also the dependent functionality testing. It also involves the user interface testing and documentation testing. Hence, it is essential that this is planned well and the task accomplished. The test plan document has to be prepared before the testing phase is started, which clearly lays down the objectives, scope of test, tasks to be performed and the test matrix which depicts the schedule of testing.

The objectives of beta testing is to:
- evaluate software technical content.
- evaluate software ease of use.
- evaluate user documentation draft.
- identify errors.
- report errors/findings.

The role of a test lead is to provide test instruction sheet that describes items such as testing objectives, steps to follow, data to enter, functions to invoke and to provide feedback forms and comments.
The role of a tester is to understand the software requirements and the testing objectives and carry out the test cases and report defects.

Validation phase - User Acceptance Testing and Installation Testing

User Acceptance Testing occurs just before the software is released to the customer. The end-users along with the developers perform the User Acceptance Testing with a certain set of test cases and typical scenarios.

Installation testing is often the most under tested area in testing. this type of testing is performed to ensure that all installed features and options function properly. It is also performed to verify that all necessary components of the application are, indeed, installed. Installation testing should take care of the following points:
- To check if while installing product checks for the dependent software/patches.
- The product should check for the version of the same product on the target machine, say the previous should not be over installed on the newer version.
- Installer should give a default installation path.
- Installation should allow user to install at location other than the default installation path.
- Check if the product can be installed "Over the Network".
- Installation should start automatically when the CD is inserted.
- Installer should give the Remove/Repair options.
- When uninstalling, check that all the registry keys, files, DLL, shortcuts, activeX components are removed from the system.
- Try to install the software without administrative privileges.
- Try installing on different operating system.
- Try installing on system having non-compliant configuration such as less memory/RAM/HDD.

Software Testing - Validation Phase - Alpha Testing

A software prototype stage when the software is first available for run. The software has the core functionalities in it but complete functionality is not aimed at. It would be able to accept inputs and give outputs. Usually, the most used functionalities are developed more. This test is conducted at the developer's site only. In a software development life cycle, depending on the functionalities, the number of alpha phases required are laid down in the project plan itself.

During this, the testing is not a through one since only the prototype of the software is available. The basic installation and un-installation tests and the completed core functionalities are tested.

The aim of alpha testing is :
- to identify any serious errors.
- to judge if the intended functionalities are implemented.
- to provide to the customer the feel of the software.

A thorough understanding of the product is done now. During this phase, the test plan and test cases for the beta phase which is the next stage is created. The errors reported are documented internally for the testers and developers reference. No issues are usually reported and recorded in any of the defect management or bug trackers.

The role of the test lead is to understand the system requirements completely and to initiate the preparation of test plan for the beta phase. The role of the tester is to provide input while there is still time to make significant changes as the design evolves and to report errors to developers.

Validation Phase - System Testing - Regression Testing

The process of regression testing is simple i.e the test cases that have been prepared can be used and the expected results are also known. If the process is not automated it can be very time consuming and tedious operation. Some of the tools that are available for regression testing are:
- Record and Playback tools - In this, the previously executed scripts can be re-run to verify whether the same set of results are obtained. e.g. Rational Robot.

The end goals of regression testing are :
- to ensure that the unchanged system segments function properly.
- to ensure that the previously prepared manual procedures remain correct after the changes have been made to the application system.
- to verify that the data dictionary of data elements that have been changed is correct.

Most of the time the testing team is asked to check the last minute changes in the code just before making a release to the client, in this situation the testing team needs to check only the affected areas.
In short, regression testing should get the input from the development team about the nature and amount of change in the fix so that the testing team can first check the fix and then the affected areas.
Regression testing is the testing in which maximum automation can be done. The reason being the same set of test cases will be run on different builds multiple times. But then, the extent of automation depends on whether the test cases will remain applicable over the time. In case, the automated test cases do not remain applicable for some amount of time then test engineers will end up in wasting time to automate and don not get enough out of automation.

Validation Phase - System Testing - Regression Testing

Regression testing is used to test or check the effect of changes that are made in the code. Most of the time the testing team is asked to check last minute changes in the code just before making a release to the client. In this situation, the testing team needs to check only the affected areas. In short, for regression testing, the testing team should get the input from the development team about the nature or amount of change in the fix so that the testing team can first check the fix and then the side effects of the fix.

Regression testing is the testing in which maximum automation can be done. The reason being the same set of test cases will be run on different builds multiple times. But, the extent of automation depends on whether the test cases will remain applicable for some amount of time, test engineers will end up in wasting time to automate with minimal results.

Regression testing is re-testing unchanged segments of application. It involves re-running tests that have been previously executed to ensure that the same results can be achieved currently as were achieved when the segment was last tested. The selective re-testing of a software system that has been modified to ensure that any bugs have been fixed and that no other previously working functions have failed as a result of newly added features have not created problems with previous versions of the software.

Regression testing is initiated after a programmer has attempted to fix a recognized problem or has added source code to a program that may have inadvertently introduced errors. It is a kind of quality control measure to ensure that the newly modified code still complies with its specified requirements and that unmodified code has not been affected by the maintenance activity.

During regression testing, following activities are performed:
- Re-running of previously conducted tests.
- Reviewing previously prepared manual procedures.
- Comparing the current test results with the previously executed test results.