What are different kind of application programming interface testing tools?

There are many testing tools available. Depending on the level of testing required, different tools could be used. Some of the API testing tools available are:

- JVerify: This is from Man Machine Systems. JVerify is a java class/API testing tool that supports a unique invasive testing model. The invasive model allows access to the internals of any Java object from within a test script. The ability to invade class internals facilitates more effective testing at class level, since controllability and observability are enhanced. This can be very valuable when a class has not been designed for testability.

- JavaSpec: JavaSpec is a SunTest's API testing tool. It can be used to test java applications and libraries through their API. JavaSpec guides the users through the entire test creation process and lets them focus on the most critical aspects of testing. Once the user has entered the test data and assertions, JavaSpec automatically generates self-checking tests, HTML, test documentation, and detailed test reports.

To automate API testing, the assumptions are as follows:
- Test engineer is supposed to test some API.
- The APIs are available in form of library(.lib).
- Test engineer has the API document.

There are mainly two things to test in API testing:
Black box testing of the APIs :
In this, we have to test the API for outputs. When we give a known input then we also know the ideal output. So, we have to check for the actual output against the idle output.
A simple program in C can be written as:
- Take the parameters from text file.
- Call the API with these parameters.
- Match the actual and idle output and also check the parameters for good values that are passed with reference.
- Log the result.

Interaction/ integration testing of the APIs.
Suppose there are teo APIs say
Handle h = handle createcontext(void);
When the handle to the device is to be closed then the corresponding function
Bool bishandledeleted = bool deletecontext(handle &h);

Here, we have to call the two APIs and check if they are handled by the created createcontext() and are deleted by the deletecontext().
This will ensure that these two APIs are working fine.For this we can write a simple C program that will do the following:
- Call the two APIs in the same order.
- Pass the output parameter of the first as the input of the second.
- Check for the output parameter of the second API.
- Log the result.

Step 4 To test API : Call Sequencing, Step 5 To Test API : Observe the output

Step 4: Call Sequencing
When combinations of possible arguments to each individual call are unmanageable, the number of possible call sequences is infinite. Parameter selection and combination issues further complicate the problem call-sequencing problem. Faults caused by improper call sequences tend to give rise to some of the most dangerous problems in software. Most security vulnerabilities are caused by the execution of some such seemingly improbable sequences.

Step 5: Observe the output
The outcome of an execution of an API depends upon the behavior of that API, the test condition and the environment. The outcome of an API can be at different ways i.e. some could generally return certain data or status but for some of the APIs. It might not return or shall be just waiting for a period of time, triggering another event, modifying certain resource and so on.

The tester should be aware of the output that needs to be expected for the API under test. The outputs returned for various input values like valid/invalid, boundary values etc needs to be observed and analyzed to validate if they are as per the functionality. All the error codes returned and exceptions returned for all the input combinations should be evaluated.

Step 3 To test API : Identify the combination of parameters

Identify the combination of parameters
Parameter combinations are extremely important for exercising stored data and computation. In API calls, two independently valid values might cause a fault when used together which might not have occurred with the other combinational values. Therefore, a routine called with two parameters requires selection of values for one based on the value chosen for the other. Often the response of a routine to certain data combinations is incorrectly programmed due to the underlying complex logic.

The API needs to be tested taking into consideration the combination of different parameter. The number of possible combinations of parameters for each call is typically large. For a given set of parameters, if only the boundary values have been selected, the number of combinations, while relatively diminished, may still be prohibitively large. For example, consider an API which takes three parameters as input. The various combinations of different values for the input values and their combinations needs to be identified.

Parameter combination is further complicated by the function overloading capabilities of many modern programming languages. It is important to isolate the differences between such functions and take into account that their use is context driven. The APIs can also be tested to check that there are no memory leaks after they are called. this can be verified by continuously calling the API and observing the memory utilization.

Step 1 and Step 2 To test API : Identify the initial condition and Input/Parameter Selection

STEP 1: Identify the initial condition
The testing of an application programming interface (API) would depend largely on the environment in which it is to be tested. Hence, initial condition plays a very vital role in understanding and verifying the behavior of the API under test. the initial conditions for testing APIs can be classified as:

- Mandatory Pre-setters
The execution of an API would require some minimal state, environment. These type of initial conditions are classified under the mandatory initialization for the API. For example, a non-static member function API requires an object to be created before it could be called. This is an essential activity required for invoking the API.

- Behavioral Pre-setters
To test the specific behavior of the API, some additional environmental state is required. These types of initial conditions are called the behavioral pre-setters category of initial condition. These are optional conditions required by the API and need to be set before invoking the API under test thus influencing its behavior. Since these influence the behavior of the API under test, they are considered as additional inputs other than the parameters.

Thus, to test any application programming interface, the environment required should also be clearly understood and set up. Without these criteria, API under test might not function as required and leave the tester's job undone.

STEP 2: Input/Parameter Selection
the list of valid input parameters need to be identified to verify that the interface actually performs the tasks that it was designed for. While there is no method that ensures this behavior will be tested completely, using inputs that return quantifiable and verifiable results is the next big thing. The techniques like boundary value analysis and equivalence partitioning need to be used while trying to consider the input parameter values. The boundary values or limits that would lead to errors or exceptions needs to be identified.

It would also be helpful if the data structures and other components that use these data structures apart from the API are analyzed. The data structure can be loaded by using the other components and the API can be tested while the other component is accessing these data structures.

The availability of the source code to the testers would help in analyzing the various input values that could be possible for testing the API. It would also help in understanding the various paths which could be tested. Therefore, not only are testers required to understand the calls, but also all the constants and data types used by the interface.

What is the strategy that is needed to test an API?

By analyzing the problems faced by the testers, a strategy needs to be formulated for testing the application programming interface.
- The API to be tested would require some environment for it to work. Hence, it is required that all the conditions and prerequisites understood by the tester.
- The next step would be to identify and study its points of entry. The graphical user interfaces would have items like menus, buttons, check boxes, and combo lists that would trigger the event or action to be taken.
- Similarly, for APIs, the input parameters, the events that trigger the API would act as the point of entry. Subsequently, a chief task is to analyse the points of entry as well as significant output items. The input parameters should be tested with the valid and invalid values using strategies like the boundary value analysis and equivalence partitioning.
- The fourth step is to understand the purpose of the routines, the contexts in which they are to be used. Once all this parameter selections and combinations are designed, different call sequences need to be explored.

The steps can be summarized as follows:
- Identify the initial conditions required for testing.
- Identify the parameters i.e. choosing the values of individual parameters.
- Identify the combination of parameters i.e. pick out the possible and applicable parameter combination with multiple parameters.
- Identify the order to make the calls i.e. deciding the order in which to make the calls to force the API to exhibit its functionality.
- Observe the output.

How can the testing of API calls be done?

Testing of API calls can be done in isolation or in sequence to vary the order in which the functionality is exercised and to make the API produce some useful results from these tests. Designing tests is essentially designing sequences of API calls that have a potential of satisfying the test objectives.This in turn boils down to designing each call with specific parameters and to building a mechanism for handling and evaluating return values.
Designing of test cases depends on the following criteria:
- What value should a parameter take?
- What values together make sense?
- What combination of parameters will make APIs work in a desired manner?
- What combination will cause a failure, a bad return value, or an anomaly in the operating environment?
- Which sequences are the best candidates for selection?

Some of the interesting problems for testers are:
- Ensuring that the test harness varies parameters of the API calls in ways that verify functionality and expose failures. This includes assigning common parameter values as well as exploring boundary conditions.
- Generating interesting parameter value combinations for calls with two or more parameters.
- Determining the content under which an API call is made. This might include setting external environment conditions like files, peripheral devices and also the internal stored data that affect the API.
- Sequencing API calls to vary the order in which the functionality is exercised and to make the API produce useful results from successive calls.

Understanding Application Programming Interfaces(APIs)

Application Programmable Interfaces (APIs) are collections of software functions or procedures that can be used by other applications to fulfill their functionality. APIs provide an interface to the software component.These form the critical elements for the developing the applications and are used in varied applications from graph drawing packages, to speech engines, to web-based airline reservations systems, to computer security components.

Each API is supposed to behave the way it is coded i.e. it is functionality specific. These APIs may offer different results for different type of the input provided. The errors or the exceptions returned may also vary. However, once integrated within a product, the common functionality testing/integration testing may cover only those paths. By considering each API as a black box, a generalized approach of testing can be applied. But, there may be some paths which are not tested and lead to bugs in the application. Applications can be viewed and treated as APIs from a testing perspective.

The distinctive attributes that make testing of APIs slightly different from testing other common software interfaces like GUI testing.
- Testing APIs requires a thorough knowledge of its inner workings: Some APIs may interact with the operating system kernel, other APIs, with other software to offer their functionality. Thus, an understanding of the inner workings of the interface would help in analyzing the call sequences and detecting the failures caused.
- Adequate programming skills: API tests are in form of sequences of calls, namely, programs.Each tester must possess expertise in the programming language that are targeted by the API.
- Lack of domain knowledge: Involve the testers from the initial stage of development. This would help the testers to have some understanding on the interface and avoid exploring while testing.
- No documentation: Without the documentation, it is difficult for the test designer to understand the purpose of calls, the parameter types and possible valid/invalid values, their return values, the calls it makes to other functions and usage scenarios. Hence, proper documentation would help test designer design the tests faster.
- Access to source code: The availability of the source code would help tester to understand and analyze the implementation mechanism used; and can identify the loops or vulnerabilities that may cause errors.
- Time constraints: Thorough testing of APIs is time consuming, requires a learning overhead and resources to develop tools and design tests.

How to define a practice for agile testing ?

Practice for agile testing should encompass the following features:
Conversational Test Creation
- Test case writing should be a collaborative activity including majority of the entire team. As the customers will be busy, we should have someone representing the customer.
- Defining tests is a key activity that should include programmers and customer representatives.
- It should not be done alone.

Coaching Tests
- It is a way to think about acceptance tests.
- It turns user stories into tests.
- Tests should provide goals and guidance, instant feedback and progress measurement.
- Tests should be in specified in a format that is clear enough that users or customers can understand and that is specific enough that it can be executed.
- Specification should be done by example.

Providing Test Interfaces
- Developers are responsible for providing the fixtures that automate coaching tests.
- In most cases, extreme programming teams are adding test interfaces to their products, rather than using external test tools.

Exploratory Learning
- Plan to explore, learn and understand the product with each iteration.
- Look for bugs, missing features and opportunities for improvement.
- We do not understand software until we have used it.

What are the basic components of Extreme Programming (XP) ?

The basic components of Extreme Programming (XP) are:

Test-First Programming
- Developers write unit tests before coding. It has been noted that this kind of approach motivates the coding, speeds coding and also improves design results in better designs.
- It supports a practice called re-factoring.
- Agile practitioners prefer tests to text for describing system behavior. Tests are more precise than human language and they are also a lot more likely to be updated when the design changes. How many times have you seen design documents that no longer accurately described the current workings of the software? Out-of-date design documents look pretty much like up-to-date documents. Out-of-date tests fall.
- Many open source tools like xUnit have been developed to support this methodology.

Refactoring
- It is the practice changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure.
- Traditional development tries to understand how all the code will work together in advance. This is the design. With agile methods, this difficult process of imagining what code might look like before it is written is avoided. Instead, the code is restructured as needed to maintain a coherent design.Frequent refactoring allows less up-front planning of design.
- Agile methods replace high level design with frequent re-design. It also requires a way of ensuring whether the behavior was not changed inadvertently. That's where the tests come in.
- Make the simplest design that will work and add completely only when needed and re-factor as necessary.
- Re-factoring requires unit tests to ensure that design changes do not break existing code.

Acceptance Testing
- Make up the user experiences or user stories which are short descriptions of the features to be coded.
- Acceptance tests verify the completion of user stories.
- Ideally, they are written before coding.

How much testing is relevant in an agile scenario?

Testing is as relevant in an agile scenario if not more than a traditional software development scenario. Testing is the headlight of the agile project showing where the project is standing now and the direction it is headed. Testing provides the required and relevant information to the teams to take informed and precise decisions. The testers in agile frameworks get involved in much more than finding software bugs, anything that can bug the potential user is a issue for them but testers do not make the final call, it is the entire team that discusses over it and takes a decision over a potential issues.
A firm belief of agile practitioners is that any testing approach does not assure quality, it's the team that does or does not do it, so there is a heavy emphasis on the skill and attitude of the people involved.
Agile testing is not a game of gotcha, it's about finding ways to set goals rather than focus on mistakes.
Among the agile methodologies, XP i.e. Extreme Programming components are:
- Test First Programming
- Pair Programming
- Short iterations and release.
- Re-factoring
- User Stories
- Acceptance Testing

Test-First Programming

- Developers write unit tests before coding. It has been noted that this kind of approach motivates the coding, speeds coding and also improves design results in better designs.
- It supports a practice called re-factoring.
- Agile practitioners prefer tests to text for describing system behavior. Tests are more precise than human language and they are also a lot more likely to be updated when the design changes. How many times have you seen design documents that no longer accurately described the current workings of the software? Out-of-date design documents look pretty much like up-to-date documents. Out-of-date tests fall.
- Many open source tools like xUnit have been developed to support this methodology.

Understanding Agile Testing and Agile Development Methodologies

First Definition: Agile testers treat the developers as their customer and follow the agile manifesto. The Context driven testing principles act as a set of principles for the agile tester.
Second Definition: It can also be treated as the testing methodology followed by testing team when an entire project follows agile methodologies.

Traditional QA seems to be totally at loggerheads with the agile manifesto in the following regard where process and tools are a key part of QA and testing, QA people seem to love documentation, QA people want to see the written specification and where is testing without a plan.
The question arises is there a role for QA in agile projects or not? The answer is maybe but the roles and tasks are different.

The context driven principles guidelines for the agile tester are:
- The value of any practice depends on its context.
- There are good practices in context, but there are no best practices.
- People, working together, are the most important part of any project's context.
- Projects unfold over time in ways that are often not predictable.
- The product is a solution. If the problem is not solved, the product does not work.
- Good software testing is a challenging intellectual process.
- Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right times to effectively test our products.

In the second definition, we described agile testing as a testing methodology adopted when an entire project follows agile development methodology.
Some agile development methodologies that are being practiced currently are extreme programming (XP), crystal, adaptive software development (ASD), scrum, feature driven development (FDD), dynamic systems development method (DSDM) and Xbreed.

Understanding Scenario Based Testing

Scenario based tests (SBT) are best suited when you need to tests need to concentrate on the functionality of the application than anything else.
Suppose, you are testing an application which is quite old and it is a banking application. This application has been built based on the requirements of the organization for various banking purposes. Now, this application will have continuous upgrades in the working.
Let us assume that the application is undergoing only functional changes and not the user interface changes. The test cases should be updated for every release. Over a period of time, maintaining the test ware becomes a major set back. The Scenario based tests would help you there.
As per the requirements, the base functionality is stable and there are no user interface changes. There are only changes with respect to the business functionality. As per the requirements and the situation, it is clearly understood that only regression tests need to be run continuously as a part of testing phase. Over a period of time, the individual test cases would become difficult to manage. This is the situation where we use scenarios for testing.
To derive scenarios, the following can be used as a basis:
- From the requirements, list out all the functionality of the application.
- Using a graph notation, draw depictions of various transactions which pass through various functionality of the application.
- Convert these depictions into scenarios.
- Run the scenarios when performing the testing.

Scenario based tests are not only for legacy application testing, but for any application which requires you to concentrate more on the functional requirements. I f you can plan out a perfect test strategy, then the scenario based tests can be used for any application testing for any requirements. Scenario based tests will be a good choice with a combination of various test types and techniques when you are testing projects which adopt UML (Unified Modeling Language) based development strategies.

Who is a good exploratory tester and maintain a balance between scripted and exploratory testing.

Exploratory testing approach relies a lot on the tester alone. The tester actively controls the design of tests as they are performed and uses the information gained to design new and better ideas.
A good exploratory tester should
- have the ability to explain his work.
- he should have the ability to design good tests, execute them and find important problems.
- he should document his ideas and use them in later cycles.
- he should be a careful observer.
- he should be a critical thinker.
- he should have diverse ideas so as to make new test cases and improve existing ones.
- he should remain alert for new opportunities.

Exploratory Testing is advantageous when :
- rapid testing is necessary.
- test case development time is not available.
- need to cover high risk areas with more inputs.
- need to test software with little knowledge about the specifications.
- develop new test cases or improve the existing.

The drawbacks in exploratory testing includes that it is difficult to quantize and a skilled tester is required.
Exploratory testing relies on the tester and the approach he proceeds with. Pure scripted testing does not undergo much change with time and hence the power fades away. In test scenarios, where repeatability of tests is required, automated scripts have an edge over exploratory approach. Hence, it is important to achieve a balance between the two approaches and combine the two to get the best of both.

Mission of testing and what are the situations where exploratory testing can be practiced?

The goal of testing needs to be understood first before the work begins. This could be the overall mission of the test project or could be a particular functionality or scenario. The mission is achieved by asking the right questions about the product, designing tests to answer these questions and executing tests to get the answers. Quite often, the tests do not completely answer. In such cases, we need to explore. The test procedure is recorded and the result status too.

The tester also needs to have a general plan in mind, though may not be very constrained. The tester needs to have the ability to design good test strategy, execute good tests, find important problems and report them. Out of box thinking is necessary.

The time available for testing is a critical factor. Time falls short due to following reasons:
- In a project life cycles, the time and resources required in creating the test strategy, test plan and design, execution and reporting is overlooked. Exploratory testing becomes useful since the test plan, design and execution happen together.
- Time falls short when testing is essential on a short period of notice.
- Time falls short when a new feature is implemented.
- Time falls short when change request come in much later stage of the cycle when much of the testing is done.

In such situations, exploratory testing becomes very useful. A basic strategy of exploratory testing is to have a general plan of attack but also allows yourself to deviate from it in a short period of time. In a session of exploratory testing, a set of test ideas, written notes and bug reports are the results. This can be reviewed by the test lead or a test manager.
- It is very important to identify the test strategy and the scope of the test carried. It is dependent on the project approach to testing.
- The tester crafts the test by systematically exploring the product. He defines his approach, analyze the product, and evaluate the risk.
- The written notes and scripts of the tester are reviewed by the test lead or manager.

Where does Exploratory Testing fit with its advantages and disadvantages.

Exploratory testing is called for in any situation where it is not obvious what the next test should be, or when you want to go beyond the obvious tests. More specifically, freestyle exploratory testing fits in any of the following situations:
- when you need to provide rapid feedback on a new product or feature.
- when you need to learn the product quickly.
- when you want to investigate and isolate a particular defect.
- when you want to check the work of another tester by doing a brief independent investigation.
- when you have to find out the single most important bug in the shortest time.
- when you have already tested using scripts, and seek to diversify the testing.
- when you want to investigate and isloate a particular defect.
- when you want to investigate the status of a particular risk in order to evaluate the need for scripted tests in that area.

Advantages of Exploratory Testing

- It does not require extensive documentation.
- It is responsive to changing scenarios.
- Under tight schedules, testing can be more focused depending on the bug rate or risks.
- Improved coverage.

Disadvantages of Exploratory Testing

- Dependent on tester's skills.
- Test tracking not concrete.
- More prone to human error.
- No contingency plan if the tester is unavailable.

What specifics affect Exploratory testing

- The mission of the particular test session.
- The tester skills, talents, preferences.
- Available time and other resources.
- Status of other testing cycles for the product.
- How much the tester knows about the product.

Defect Driven Exploratory Testing - Formalized Approach for Exploratory Testing

Defect driven exploratory testing is another formalized approach used for exploratory testing. It is a goal oriented approach focused on the critical areas identified on the defect analysis study based on procedural testing results.
In procedural testing, the tester executes readily available test cases, which are written based the requirement specifications. Although the test cases are executed completely, defects were found in the software while doing exploratory testing by just wandering through the product blindly. A reliable basis was needed for exploring the software. Thus, defect driven exploratory testing is an idea of exploring that part of the product based on the results obtained during procedural testing. After analyzing the defects found during defect driven exploratory testing process, it was found that these were the most critical bugs, which were camouflaged in the software and which if present could have made the software not fit for use.

There are some pre-requisites for defect driven exploratory testing:
- In-depth knowledge of the product.
- Procedural testing has to be carried out.
- Defect analysis based on scripted tests.

Advantages of defect driven exploratory testing:
- Tester has clear clues on the areas to be explored.
- Goal oriented approach, hence better results.
- No wastage of time.

What are some of the formal approaches used for exploratory testing? Continued...

Charter states the goal and the tactics to be used.

A charter can be simple one to more descriptive giving the strategies and outlines for the testing process.
Charter summary contains
- Architectural the charters i.e. test planning.
- Brief information or guidelines on:
1.)Mission: Why do we test this?
2.)What should be tested?
3.)How to test?
4.)What problems to look for?
5.)tools to use
6.)Specific test techniques or tactics to use
7.)What risks are involved?
8.)Documents to examine
9.)Desired output from testing

Session Based Test Management(SBTM)

Session based test management is a formalized approach that uses the concept of charters and the sessions for performing the exploratory testing. A session is not a test case or bug report. It is the reviewable product produced by chartered and uninterrupted test effort. A session can last from 60 to 90 minutes but there is no hard and fast rule on the time spent for testing. If a session lasts closer to 45 minutes, we call it a short session. If it lasts closer to two hours, we call it a long session. Each session designed depends on the tester and the charter. After the session is completed, each session is de-briefed. The primary objective in the de-briefing is to understand and accept the session report, provide feedback and coaching to the tester. The de-briefings should help the manager to plan the sessions in future and also to estimate the time required for testing the similar functionality.
The de-briefing session is based on agenda called PROOF.
Past: What happened during the session?
Results: What was achieved during the session?
Outlook: What still needs to be done?
Obstacles: What got in the way of good testing?
Feeling: How does the tester feel about all this?

A session can be broadly classified into three tasks:
- Session test up: Time required in setting up the application under test.
- Test Design and execution: time required scanning the product and test.
- Bug investigation and reporting: time required finding the bugs and reporting to the concerned.

The entire session report consists of session charter, tester name, data and time started, task breakdown, data files, test notes, issues, bugs.

What are some of the formal approaches used for exploratory testing? Continued...

Approaches that are used for exploratory testing are:
- Record Failures
In exploratory testing, testing is done without having any documented test cases. If a bug is found, it is very difficult for us to test it after fix. This is because there are no documented steps too navigate to that particular scenario. Hence, we need to keep the track of the flow required to reach where a bug has been found. So while testing, it is important that at least the bugs that have been discovered are documented. By recording failures, we are able to keep track of work that has been done. This would also help even if the tester who was actually doing exploratory testing is not available. Since the document can be referred and list all the bugs that have been reported as well the flows for the same can be identified.

- Document issues and questions
The tester trying to test an application using exploratory testing methodology should feel comfortable to test. Hence, it is advisable that the tester navigates through the application once and notes any ambiguities or queries he might feel. He can even get the clarification on the work-flows he is not comfortable. Hence by documenting all the issues and questions that have been found while scanning or navigating the application can help the tester have testing done without any loss in time.

- Decompose the main task into smaller tasks. The smaller ones to still smaller activities
It is always easier to work with the smaller tasks when compared to large tasks. This is very useful in performing exploratory testing because lack of test cases might lead us to different routes. By having a smaller task, the scope as well as the boundary are confined which will help the tester to focus on his tetsing and plan accordingly.
If a big task is taken up for testing, as we explore the system, we might get deviated from our main goal or task. It might be hard to define boundaries if the application is a new one. With smaller tasks, the goal is known and hence the focus and the effort required can be properly planned.

What are some of the formal approaches used for exploratory testing? Continued...

Some of the formal approaches used for exploratory testing are:

- Identify the break points
Break points are the situations where the system starts behaving abnormally. It does not give the output it is supposed to give. So, by identifying such situations also, testing can be done. Use boundary values or invariance for finding the break points of the application. In most of the cases, it is observed that system would work for normal inputs and outputs. Try to give input that might be the ideal situation or the worse situation. By trying to identify the extreme conditions or the breakpoints would help the tester to uncover the hidden bugs. Such cases might not be covered in the normal scripted testing. hence, this helps in finding the bugs which might not be covered in normal testing.

- Check the UI against Windows Interface etc standards
The exploratory testing can be performed by identifying the user interface standards. There are set standards laid down for the user interfaces that need to be developed. These user standards are nothing but the look and feel aspects of the interfaces, the user interacts with. The user should be comfortable with any of the screens that he or she is working on. These aspects help the end user to accept the system faster. By identifying the user standards, define an approach to test because the application developed should be user friendly for the user's usage.

- Identify expected results
The tester should know what he is testing for and expected output for the given input. Until and unless, the aim of the testing is not known, there is no use of the testing that is done because the tester may not succeed in distinguishing the real error and normal work-flow. The tester needs to analyze what is the expected output for the scenario he is testing.

- Identify the interfaces with other interfaces/external applications
In the age of component development and maximum re-usability, developers try to pick up the already developed components and integrate them. In some cases, it would help the tester explore the areas where the components are coupled. The output of one component should be correctly sent to other component. Hence, such scenarios or work-flows need to be identified and explored more. There may be external interfaces, like the application is integrated with another application for the data. In such cases, focus should be more on the interface between the two applications.

What are some of the formal approaches used for exploratory testing?

Some of the formal approaches used for exploratory testing are:

- Identify the domain
The exploratory testing can be performed by identifying the application domain. If the tester has good knowledge of domain, the it would be easier to test the system without having any test cases. If the tester were well aware of the domain, it would help analyzing the system faster and better. His knowledge would help in identifying the various workflows that usually exist in the domain. He would also be able to decide what are the different scenarios and which are most critical for that system. Hence, he can focus his testing depending on the scenarios required. If a QA lead is trying to assign the tester to a task, it is advisable that the tester identifies the person who has the domain knowledge of that testing for exploratory testing.

- Identify the purpose
Another approach to exploratory testing is by identifying the purpose of the system i.e. What is that system used for. Thus, by identifying the primary and secondary functions for the system, testing can be done where more focus and effort can be given to primary functions as as compared to secondary functions.

- Identify the workflows
Identifying the workflows for testing any system without any scripted test cases can be considered as one of the best approaches used. The workflows are nothing but a visual representation of the scenarios as the system would behave for any given input. The workflows can be simple flow charts or data flow diagrams or something like state diagrams, use cases, models etc. The workflows will also help to identify the scope for that scenario. The workflows would help the tester to keep track of the scenarios for testing. It is suggested that the tester navigates through the application before he starts exploring. It helps the tester in identifying the various possible workflows and issues any found which he is comfortable can be discussed with the concerned team.

Introduction to Formalized Exploratory Testing ....

In normal testing style, the test process is planned well in advance before the actual testing begins. In this, the test design is separated from the test execution phase. Many times the test design and test execution is entrusted on different persons. Exploratory testing should not be confused with the dictionary meaning of ad hoc. Ad hoc testing normally refers to a process of improvised, impromptu bug searching.

Formalized Exploratory Testing

A structured and reasoned approach to exploratory testing is termed as formalized exploratory testing. This approach consists of specific tasks, objectives, and deliverables that make it a systematic process. Using the systematic approach, an outline of what to attack first, its scope, the time required to be spent etc is achieved. The approach might be using simple notes to more descriptive charters to some vague scripts. By using the systematic approach, the testing can be more organized focusing at the goal to be reached. Thus, solving the problem where the pure exploratory testing might drift away from the goal.

The formalized approach used for the exploratory testing can vary depending on the various criteria like resource, time, the knowledge of the application available etc. Depending o this criteria, the approach used to attack the system will also vary. It may involve creating the outlines on the notepad to more sophisticated way by using charters etc.

Exploratory Testing - Introduction and Overview

Exploratory testing is an interactive process of concurrent product exploration, test design and test execution. This testing is defined as simultaneous test design, test execution and bug reporting. In this approach, the tester explores the system without having any prior test cases or test scripts. It is also called as ad hoc testing, gorilla testing or intuitive testing. In operational terms, exploratory testing is an interactive process of concurrent product exploration, test design, and test execution. The outcome of an exploratory testing session is a set of notes about the product, failures found, and a concise record of how the product was tested. When practiced by trained testers, it yields consistently valuable and audit-able results. Every tester performs this type of testing at one point or the other. This testing totally depends on the skill and creativity of the tester. Different testers can explore the system in different ways depending on their skills. Thus, the tester has a very vital role to play in exploratory testing.

A systematic approach of exploratory testing can also be used where there is a plan to attack the system under test. This systematic approach of exploring the system is termed as formalized exploratory testing. It is a powerful approach but it has not got recognition and is often misunderstood and not gained the respect it needs. In many situations, it can be more productive than the scripted testing.
This testing believes in concurrent phases of product exploration, test design and test execution. It is categorized under black box testing. It is basically a free-style testing approach where you do not begin with the usual procedures of elaborate test plans and test steps. The test plan and strategy is very well in the tester's mind. The tester asks the right question and judges the outcome. During this phase, the tester is actually learning the product as he tests it. It is interactive and creative. A conscious plan by the tester gives good results.

Validation Phase - Beta Testing - Objectives

The beta testing is conducted at one or more customer sites by the end-user of the software. The beta test is a live application of the software in an environment that cannot be controlled by the developer. The software reaches beta stage when most of the functionalities are operating. The software is tested in customer's environment, giving the user an opportunity to exercise the software, find the errors so that they could be fixed before product release. Beta testing is a detailed testing and needs to cover all the functionalities of the product and also the dependent functionality testing. It also involves the user interface testing and documentation testing. Hence, it is essential that this is planned well and the task accomplished. The test plan document has to be prepared before the testing phase is started, which clearly lays down the objectives, scope of test, tasks to be performed and the test matrix which depicts the schedule of testing.

The objectives of beta testing is to:
- evaluate software technical content.
- evaluate software ease of use.
- evaluate user documentation draft.
- identify errors.
- report errors/findings.

The role of a test lead is to provide test instruction sheet that describes items such as testing objectives, steps to follow, data to enter, functions to invoke and to provide feedback forms and comments.
The role of a tester is to understand the software requirements and the testing objectives and carry out the test cases and report defects.