What are different security testing techniques?

Security testing techniques are needed to defend the vulnerabilities of a software system or application and to protect it from security threats. Every web site and application has some kind of vulnerabilities which weaken the application’s security and exposes it to the exploitation.

The security should be carried out along all other testing phases so that a uniform security is maintained throughout the application and vulnerabilities and threats are discovered and addressed from time to time. But, unfortunately the security testing is usually conducted at the last phase of the development cycle.

PENETRATION & STATIC ANALYSIS TESTING TOOLS
- Penetration testing tools or web application scanners help a great deal in identifying the vulnerabilities of a web site or application.
- The human brain cannot mentally check out the whole source code and aspects for vulnerabilities and weak points.
- Another class of tools that can be used for this purpose is static code analysis tools.
- Both these penetration tools and static code analysis tools are needed for security testing of web sites and web applications.
- These tools work very effectively in digging out the vulnerabilities.
- On the other hand, static code analysis tools belong to the white box testing tools.
- They are used by the security groups to complement the penetration tools and they focus up on finding the specific root vulnerabilities.
- Penetration testing tools are to be used when the tester is having a limited knowledge of the web application under testing.
- These security tools are employed to check out the following security issues:
(a) SQL injection attacks
(b) Cross site scripting attacks
(c) Directory traversal attacks
(d) Issues related to session management
(e) Validation of the supplied input

The penetration testing tools emphasize upon the following security areas of a web site or web application:
- Network security
- Data base security
- Security sub- system
- Web application security

The penetration testing tools though focus up on both the positive and negative requirements; the more emphasis is on the negative ones. On the other hand normal software testing focuses only up on the positive requirements.

TECHNIQUES USED TO CARRY OUT SECURITY TESTING OF WEB APPLICATIONS

1. Fuzz Test Technique
- This type of testing involves injection of various types of generated data at the interface of the web site or application under testing.
- The data is either randomly generated or systematically.

2. Syntax Test Technique
- This type of testing involves generation of both legal and illegal data.
- This data input values are fed to the application and the behavior of the web application is observed i.e., whether it accepts or rejects the input.

3. Data Analysis
- The data generated by the web application is checked and the context of cryptography is employed here.

4. Exploratory Testing
- This testing is carried out without any test plan.
- There are no specific expectations in this type of security testing and the outcomes are also not expected.

5. Scaffolding
- Testers require some support to carry out their own specially designed testing techniques.
- For that they require supportive tools.
- This is termed as scaffolding.

6. Monitoring the behavior of the program
- Automated tools are used to monitor the outcomes of the different security testing techniques applied and also the behavior of the web application is monitored.
- This technique saves a lot of time.

Security testing of the web sites and applications is crucial to the security of the enterprise since the web sites and applications need to be available to the people all time and the threats and vulnerabilities possess a big danger to the cyber world.

What is the difference between scripted testing and exploratory testing?

Though exploratory testing and scripted are considered being a part of each other, there is a lot of difference between the two.

The scripted testing is performed by a set of test scripts which can be defined as a set of instructions that are to be performed on the software system or application under the test to test a particular functionality as required.

There are following ways of executing test scripts:

1. Automated testing:
- It can be easily repeated and is faster than manual testing.
- It is very useful when it is required to run the same tests over and over again.
- It is performed as a part of regression testing.
- Automated tests are poorly written and they may break during the execution.
- They only test what has been programmed into them.
- They only ensure that old bugs do not reappear.
- Manual testing and automated testing should be mixed together and used for exploratory testing.

2. Manual testing:
- This is the oldest and the most rigorous forms of testing.
- The test cases are reviewed by the whole testing team before using them.
- The test cases are either created at the basic functionality level or at the scenario level.

3. Short programs or test scripts written in a programming language (C#, C++, expect, TCL, java, Perl, PHP, python, power shell) or a special automated functional GUI test tool (like Borland silk test, HP quick professional test, rational robot etc) can be used to test functionality of a software system or application.

4. Short programs that have been extensively parameterized or data driven testing.

5. Table driven testing or keyword driven testing i.e., using reusable tests created in a table.

CHARACTERISTICS OF SCRIPTED TESTING
- Scripted testing is considered to be as a waste of resources by most of the testers.

- According to them the scripted manual testing inhibits the mind of the tester to use their creativity.

- Also, this testing requires heavy documentation as well as considerable amount of resources to create them.

- They often get out dated since the changes in the system are inevitable.

- Though being so criticized manual scripted testing is used by many organizations.

- The test cases are made repeatable and easy enough for a tester to perform testing with minimum supervision.

- Manual scripted testing is used in contracts where contractual agreement states that written requirements and specifications of the software must be met.

- Scripted test cases are useful where the tests are used as benchmark and have to be executed in the same way, every time.

CHARACTERISTICS OF EXPLORATORY TESTING
- In contrast to scripted testing, exploratory testing is a usual approach to software testing and can be defined as a test designing, test execution and simultaneously learning process.

- Exploratory testing is a kind of software testing that is emphasized and focused up on the responsibility and the personal freedom of the software tester.

- Exploratory testing is a kind of software testing that is emphasized and focused up to optimize the quality and effectiveness of his /her work continually by taking in to consideration mutually supportive activities like test designing, test related learning, test result interpretation and test execution.

- The activities run in parallel to each other.

- Exploratory testing basically heeds to exploiting the creativity of the tester or the programmer.

- During the testing of the software, the tester comes across various new things to learn together with experiencing them.

- Exploratory testing can be thought of as a black box testing methodology or technique instead of thinking of it as an approach to testing that can be used for any type of testing methodology and also at any phase of development of the software system or application.

What are different aspects of exploratory testing?

First let us discuss a little about what is exploratory testing actually?
It is a term rarely heard of! Exploratory testing is a usual approach to software testing and can be defined as a test designing, test execution and simultaneously a learning process. It was named by Cem Kaner in 1983.

He stated:

"exploratory testing as a kind of software testing that is emphasized and focused up on the responsibility and the personal freedom of the software tester to optimize the quality and effectiveness of his /her work continually by taking in to consideration mutually supportive activities like test designing, test related learning, test result interpretation and test execution".

- These activities run in parallel to each other.

- Exploratory testing basically heeds to exploiting the creativity of the tester or the programmer.

- During the testing of the software, the tester comes across various new things to learn together with experiencing them.

- The tester or the developer with his /her creativity can generate new and better test cases to run.

- Exploratory can be thought of as a black box testing methodology or technique instead of thinking of it as an approach to testing that can be used for any type of testing methodology and also at any phase of development of the software system or application.

According to some misconceptions, the testing technique or the object is considered to be the key here but, it is not so.

- In actual, the key is the cognitive engagement of the software tester and his /her responsibility of managing time.

- The effectiveness of the exploratory testing depends basically on how it is performed.

- Exploratory testing should always be performed by skilled testers or developers.

- Exploratory testing can also be though of as an intellectual activity.

- Exploratory testing is all about discovering how the software system or application actually works and how it handles both the easy and difficult cases.

- The effectiveness or the quality of the testing is largely dependent on the skills for developing test cases by the tester.

- It also depends on his/ her ability to find defects in the software code.

We can say that the quality of the testing is directly proportional to the knowledge of the tester about the software product and different test methods.

- The more he/ she know about the software system under test and testing methodologies, the better will be the exploratory testing.

- While carrying out the exploratory testing, the expectations should be kept open.

- Only some results can be expected or predicted but not all.

- The tester is responsible for configuring, operating, observing and evaluating the product itself and its behavior, investigating the result and reporting information.

- If we see practically the actual testing is a combination of both scripted testing and exploratory testing but, depending on the context of the testing the tendency is towards either one.

Exploratory testing is more about thinking how the testing should be done more than a testing technique. The documentation part of the exploratory testing includes:

- Documentation of all the test cases used.
- Documentation of the bugs.

Exploratory testers use tools like:

- Screen capture
- Video tools

These tools are used to generate situations of interest or as a record of the exploratory session. Bugs are discovered quickly and the approach tends to be more intellectual. Deductive reasoning based on the results of the previous tests can also be used in the exploratory testing. This leads to increased bug detection when used properly. The main disadvantage is that the tests can’t be reviewed before hand.

What are different aspects of exploratory testing?

Exploratory testing is a usual approach to software testing and can be defined as a test designing, test execution and simultaneously learning process.

It was named by Cem Kaner in 1983. "He stated exploratory testing as a kind of software testing that is emphasized and focused up on the responsibility and the personal freedom of the software tester to optimize the quality and effectiveness of his /her work continually by taking in to consideration mutually supportive activities like test designing, test related learning, test result interpretation and test execution".

These activities run in parallel to each other.

- Exploratory testing basically heeds to exploiting the creativity of the tester or the programmer.
- During the testing of the software, the tester comes across various new things to learn together with experiencing them.
- The tester or the developer with his /her creativity can generate new and better test cases to run.
- Exploratory can be thought of as a black box testing methodology or technique instead of thinking of it as an approach to testing that can be used for any type of testing methodology and also at any phase of development of the software system or application.

According to some misconceptions, the testing technique or the object is considered to be the key here but, it is not so.
- In actual, the key is the cognitive engagement of the software tester and his /her responsibility of managing time.
- The effectiveness of the exploratory testing depends basically on how it is performed.
- Exploratory testing should always be performed by skilled testers or developers.
- Exploratory testing can also be thought of as an intellectual activity.
- Exploratory testing is all about discovering how the software system or application actually works and how it handles both the easy and difficult cases.

The effectiveness or the quality of the testing is largely dependent on the skills for developing test cases by the tester.
- It also depends on his/ her ability to find defects in the software code.
- We can say that the quality of the testing is directly proportional to the knowledge of the tester about the software product and different test methods.
- The more he/ she know about the software system under test and testing methodologies, the better will be the exploratory testing.
- While carrying out the exploratory testing, the expectations should be kept open.
- Only some results can be expected or predicted but not all.
- The tester is responsible for configuring, operating, observing and evaluating the product itself and its behavior, investigating the result and reporting information. - If we see practically the actual testing is a combination of both scripted testing and exploratory testing but, depending on the context of the testing the tendency is towards either one.
- Exploratory testing is more about thinking how the testing should be done more than a testing technique.

The documentation part of the exploratory testing includes:

- Documentation of all the test cases used.
- Documentation of the bugs.

Exploratory testers use tools like:
- Screen capture
- Video tools

These tools are used to generate situations of interest or as a record of the exploratory session. For exploratory testing less before hand testing is needed. Bugs are discovered quickly and the approach tends to be more intellectual. Deductive reasoning based on the results of the previous tests can also be used in the exploratory testing. This leads to increased bug detection when used properly. The main disadvantage is that the tests can’t be reviewed before hand.

Who is a good exploratory tester and maintain a balance between scripted and exploratory testing.

Exploratory testing approach relies a lot on the tester alone. The tester actively controls the design of tests as they are performed and uses the information gained to design new and better ideas.
A good exploratory tester should
- have the ability to explain his work.
- he should have the ability to design good tests, execute them and find important problems.
- he should document his ideas and use them in later cycles.
- he should be a careful observer.
- he should be a critical thinker.
- he should have diverse ideas so as to make new test cases and improve existing ones.
- he should remain alert for new opportunities.

Exploratory Testing is advantageous when :
- rapid testing is necessary.
- test case development time is not available.
- need to cover high risk areas with more inputs.
- need to test software with little knowledge about the specifications.
- develop new test cases or improve the existing.

The drawbacks in exploratory testing includes that it is difficult to quantize and a skilled tester is required.
Exploratory testing relies on the tester and the approach he proceeds with. Pure scripted testing does not undergo much change with time and hence the power fades away. In test scenarios, where repeatability of tests is required, automated scripts have an edge over exploratory approach. Hence, it is important to achieve a balance between the two approaches and combine the two to get the best of both.

Mission of testing and what are the situations where exploratory testing can be practiced?

The goal of testing needs to be understood first before the work begins. This could be the overall mission of the test project or could be a particular functionality or scenario. The mission is achieved by asking the right questions about the product, designing tests to answer these questions and executing tests to get the answers. Quite often, the tests do not completely answer. In such cases, we need to explore. The test procedure is recorded and the result status too.

The tester also needs to have a general plan in mind, though may not be very constrained. The tester needs to have the ability to design good test strategy, execute good tests, find important problems and report them. Out of box thinking is necessary.

The time available for testing is a critical factor. Time falls short due to following reasons:
- In a project life cycles, the time and resources required in creating the test strategy, test plan and design, execution and reporting is overlooked. Exploratory testing becomes useful since the test plan, design and execution happen together.
- Time falls short when testing is essential on a short period of notice.
- Time falls short when a new feature is implemented.
- Time falls short when change request come in much later stage of the cycle when much of the testing is done.

In such situations, exploratory testing becomes very useful. A basic strategy of exploratory testing is to have a general plan of attack but also allows yourself to deviate from it in a short period of time. In a session of exploratory testing, a set of test ideas, written notes and bug reports are the results. This can be reviewed by the test lead or a test manager.
- It is very important to identify the test strategy and the scope of the test carried. It is dependent on the project approach to testing.
- The tester crafts the test by systematically exploring the product. He defines his approach, analyze the product, and evaluate the risk.
- The written notes and scripts of the tester are reviewed by the test lead or manager.

Where does Exploratory Testing fit with its advantages and disadvantages.

Exploratory testing is called for in any situation where it is not obvious what the next test should be, or when you want to go beyond the obvious tests. More specifically, freestyle exploratory testing fits in any of the following situations:
- when you need to provide rapid feedback on a new product or feature.
- when you need to learn the product quickly.
- when you want to investigate and isolate a particular defect.
- when you want to check the work of another tester by doing a brief independent investigation.
- when you have to find out the single most important bug in the shortest time.
- when you have already tested using scripts, and seek to diversify the testing.
- when you want to investigate and isloate a particular defect.
- when you want to investigate the status of a particular risk in order to evaluate the need for scripted tests in that area.

Advantages of Exploratory Testing

- It does not require extensive documentation.
- It is responsive to changing scenarios.
- Under tight schedules, testing can be more focused depending on the bug rate or risks.
- Improved coverage.

Disadvantages of Exploratory Testing

- Dependent on tester's skills.
- Test tracking not concrete.
- More prone to human error.
- No contingency plan if the tester is unavailable.

What specifics affect Exploratory testing

- The mission of the particular test session.
- The tester skills, talents, preferences.
- Available time and other resources.
- Status of other testing cycles for the product.
- How much the tester knows about the product.

Defect Driven Exploratory Testing - Formalized Approach for Exploratory Testing

Defect driven exploratory testing is another formalized approach used for exploratory testing. It is a goal oriented approach focused on the critical areas identified on the defect analysis study based on procedural testing results.
In procedural testing, the tester executes readily available test cases, which are written based the requirement specifications. Although the test cases are executed completely, defects were found in the software while doing exploratory testing by just wandering through the product blindly. A reliable basis was needed for exploring the software. Thus, defect driven exploratory testing is an idea of exploring that part of the product based on the results obtained during procedural testing. After analyzing the defects found during defect driven exploratory testing process, it was found that these were the most critical bugs, which were camouflaged in the software and which if present could have made the software not fit for use.

There are some pre-requisites for defect driven exploratory testing:
- In-depth knowledge of the product.
- Procedural testing has to be carried out.
- Defect analysis based on scripted tests.

Advantages of defect driven exploratory testing:
- Tester has clear clues on the areas to be explored.
- Goal oriented approach, hence better results.
- No wastage of time.

What are some of the formal approaches used for exploratory testing? Continued...

Charter states the goal and the tactics to be used.

A charter can be simple one to more descriptive giving the strategies and outlines for the testing process.
Charter summary contains
- Architectural the charters i.e. test planning.
- Brief information or guidelines on:
1.)Mission: Why do we test this?
2.)What should be tested?
3.)How to test?
4.)What problems to look for?
5.)tools to use
6.)Specific test techniques or tactics to use
7.)What risks are involved?
8.)Documents to examine
9.)Desired output from testing

Session Based Test Management(SBTM)

Session based test management is a formalized approach that uses the concept of charters and the sessions for performing the exploratory testing. A session is not a test case or bug report. It is the reviewable product produced by chartered and uninterrupted test effort. A session can last from 60 to 90 minutes but there is no hard and fast rule on the time spent for testing. If a session lasts closer to 45 minutes, we call it a short session. If it lasts closer to two hours, we call it a long session. Each session designed depends on the tester and the charter. After the session is completed, each session is de-briefed. The primary objective in the de-briefing is to understand and accept the session report, provide feedback and coaching to the tester. The de-briefings should help the manager to plan the sessions in future and also to estimate the time required for testing the similar functionality.
The de-briefing session is based on agenda called PROOF.
Past: What happened during the session?
Results: What was achieved during the session?
Outlook: What still needs to be done?
Obstacles: What got in the way of good testing?
Feeling: How does the tester feel about all this?

A session can be broadly classified into three tasks:
- Session test up: Time required in setting up the application under test.
- Test Design and execution: time required scanning the product and test.
- Bug investigation and reporting: time required finding the bugs and reporting to the concerned.

The entire session report consists of session charter, tester name, data and time started, task breakdown, data files, test notes, issues, bugs.

What are some of the formal approaches used for exploratory testing? Continued...

Approaches that are used for exploratory testing are:
- Record Failures
In exploratory testing, testing is done without having any documented test cases. If a bug is found, it is very difficult for us to test it after fix. This is because there are no documented steps too navigate to that particular scenario. Hence, we need to keep the track of the flow required to reach where a bug has been found. So while testing, it is important that at least the bugs that have been discovered are documented. By recording failures, we are able to keep track of work that has been done. This would also help even if the tester who was actually doing exploratory testing is not available. Since the document can be referred and list all the bugs that have been reported as well the flows for the same can be identified.

- Document issues and questions
The tester trying to test an application using exploratory testing methodology should feel comfortable to test. Hence, it is advisable that the tester navigates through the application once and notes any ambiguities or queries he might feel. He can even get the clarification on the work-flows he is not comfortable. Hence by documenting all the issues and questions that have been found while scanning or navigating the application can help the tester have testing done without any loss in time.

- Decompose the main task into smaller tasks. The smaller ones to still smaller activities
It is always easier to work with the smaller tasks when compared to large tasks. This is very useful in performing exploratory testing because lack of test cases might lead us to different routes. By having a smaller task, the scope as well as the boundary are confined which will help the tester to focus on his tetsing and plan accordingly.
If a big task is taken up for testing, as we explore the system, we might get deviated from our main goal or task. It might be hard to define boundaries if the application is a new one. With smaller tasks, the goal is known and hence the focus and the effort required can be properly planned.

What are some of the formal approaches used for exploratory testing? Continued...

Some of the formal approaches used for exploratory testing are:

- Identify the break points
Break points are the situations where the system starts behaving abnormally. It does not give the output it is supposed to give. So, by identifying such situations also, testing can be done. Use boundary values or invariance for finding the break points of the application. In most of the cases, it is observed that system would work for normal inputs and outputs. Try to give input that might be the ideal situation or the worse situation. By trying to identify the extreme conditions or the breakpoints would help the tester to uncover the hidden bugs. Such cases might not be covered in the normal scripted testing. hence, this helps in finding the bugs which might not be covered in normal testing.

- Check the UI against Windows Interface etc standards
The exploratory testing can be performed by identifying the user interface standards. There are set standards laid down for the user interfaces that need to be developed. These user standards are nothing but the look and feel aspects of the interfaces, the user interacts with. The user should be comfortable with any of the screens that he or she is working on. These aspects help the end user to accept the system faster. By identifying the user standards, define an approach to test because the application developed should be user friendly for the user's usage.

- Identify expected results
The tester should know what he is testing for and expected output for the given input. Until and unless, the aim of the testing is not known, there is no use of the testing that is done because the tester may not succeed in distinguishing the real error and normal work-flow. The tester needs to analyze what is the expected output for the scenario he is testing.

- Identify the interfaces with other interfaces/external applications
In the age of component development and maximum re-usability, developers try to pick up the already developed components and integrate them. In some cases, it would help the tester explore the areas where the components are coupled. The output of one component should be correctly sent to other component. Hence, such scenarios or work-flows need to be identified and explored more. There may be external interfaces, like the application is integrated with another application for the data. In such cases, focus should be more on the interface between the two applications.

What are some of the formal approaches used for exploratory testing?

Some of the formal approaches used for exploratory testing are:

- Identify the domain
The exploratory testing can be performed by identifying the application domain. If the tester has good knowledge of domain, the it would be easier to test the system without having any test cases. If the tester were well aware of the domain, it would help analyzing the system faster and better. His knowledge would help in identifying the various workflows that usually exist in the domain. He would also be able to decide what are the different scenarios and which are most critical for that system. Hence, he can focus his testing depending on the scenarios required. If a QA lead is trying to assign the tester to a task, it is advisable that the tester identifies the person who has the domain knowledge of that testing for exploratory testing.

- Identify the purpose
Another approach to exploratory testing is by identifying the purpose of the system i.e. What is that system used for. Thus, by identifying the primary and secondary functions for the system, testing can be done where more focus and effort can be given to primary functions as as compared to secondary functions.

- Identify the workflows
Identifying the workflows for testing any system without any scripted test cases can be considered as one of the best approaches used. The workflows are nothing but a visual representation of the scenarios as the system would behave for any given input. The workflows can be simple flow charts or data flow diagrams or something like state diagrams, use cases, models etc. The workflows will also help to identify the scope for that scenario. The workflows would help the tester to keep track of the scenarios for testing. It is suggested that the tester navigates through the application before he starts exploring. It helps the tester in identifying the various possible workflows and issues any found which he is comfortable can be discussed with the concerned team.

Introduction to Formalized Exploratory Testing ....

In normal testing style, the test process is planned well in advance before the actual testing begins. In this, the test design is separated from the test execution phase. Many times the test design and test execution is entrusted on different persons. Exploratory testing should not be confused with the dictionary meaning of ad hoc. Ad hoc testing normally refers to a process of improvised, impromptu bug searching.

Formalized Exploratory Testing

A structured and reasoned approach to exploratory testing is termed as formalized exploratory testing. This approach consists of specific tasks, objectives, and deliverables that make it a systematic process. Using the systematic approach, an outline of what to attack first, its scope, the time required to be spent etc is achieved. The approach might be using simple notes to more descriptive charters to some vague scripts. By using the systematic approach, the testing can be more organized focusing at the goal to be reached. Thus, solving the problem where the pure exploratory testing might drift away from the goal.

The formalized approach used for the exploratory testing can vary depending on the various criteria like resource, time, the knowledge of the application available etc. Depending o this criteria, the approach used to attack the system will also vary. It may involve creating the outlines on the notepad to more sophisticated way by using charters etc.

Exploratory Testing - Introduction and Overview

Exploratory testing is an interactive process of concurrent product exploration, test design and test execution. This testing is defined as simultaneous test design, test execution and bug reporting. In this approach, the tester explores the system without having any prior test cases or test scripts. It is also called as ad hoc testing, gorilla testing or intuitive testing. In operational terms, exploratory testing is an interactive process of concurrent product exploration, test design, and test execution. The outcome of an exploratory testing session is a set of notes about the product, failures found, and a concise record of how the product was tested. When practiced by trained testers, it yields consistently valuable and audit-able results. Every tester performs this type of testing at one point or the other. This testing totally depends on the skill and creativity of the tester. Different testers can explore the system in different ways depending on their skills. Thus, the tester has a very vital role to play in exploratory testing.

A systematic approach of exploratory testing can also be used where there is a plan to attack the system under test. This systematic approach of exploring the system is termed as formalized exploratory testing. It is a powerful approach but it has not got recognition and is often misunderstood and not gained the respect it needs. In many situations, it can be more productive than the scripted testing.
This testing believes in concurrent phases of product exploration, test design and test execution. It is categorized under black box testing. It is basically a free-style testing approach where you do not begin with the usual procedures of elaborate test plans and test steps. The test plan and strategy is very well in the tester's mind. The tester asks the right question and judges the outcome. During this phase, the tester is actually learning the product as he tests it. It is interactive and creative. A conscious plan by the tester gives good results.