How is test execution done by Fitnesse Testing Tool?

FitNesse testing tool based on the integration testing frame work is now widely used for acceptance testing more than unit testing since it has the ability to facilitate a description of the function in a detailed manner. 

In this article, we shall discuss about how the execution of the tests is carried out using the Fitnesse testing tool. 

The testing process using the fitnesse testing tool involves 4 major components for every test:

1. Wiki page that is used to express the test in the form of a decision table.
2. A testing engine for interpreting this wiki page.
3. A test fixture that is invoked by the engine and itself invokes the SUT or system under test in turn.
4. SUT(system under test) that is currently under test.

- Out of the above 4 mentioned components two are produced by the software development team namely the fixture and the wiki page. 

- The team produces the system under test but it is not considered from the point of view of the black box test.

- Decision tables which express a test are included in the wiki page.

- The test fixture is the link that is made between the SUT and the generic testing through a piece of code written in java. 

- Mapping that is done between the fixture and the wiki page is an example of simple convert – to – camel case mapping. 

- Such kind of mapping is applicable to almost all the headings and is used for the identification of the name of the class of the fixture and its methods as well. 

- Whenever a heading ending is encountered, its value is to be considered to be read from the fixture while values from the other headers are considered as input for the fixture.

- The column order of the table i.e., left to right order is followed for calling the methods of the fixtures. 

- Testing engine is the component that actually carries out the mapping process. 

- It also invokes majority of the fixture methods. 

What kind of engines are supported by Fitnesse Testing Tool?

Two types of engines are supported by the fitnesse testing tool:

The FIT Engine: 

- This is more like a frame work rather than just being an engine. 

- It carried out the following purposes:

a)   It combines the functionality for invoking the tests,

b)   Interpretation of the wiki pages,

c)   Generation of the output pages.

- This engine was named so because the tool originally developed around this. - This engine works by combining the responsibilities rather by separating them.

The SLIM Engine: 

- SLIM stands for simple list invocation method and is used as an alternative to the FIT engine. 

- This engine implements the slim protocol. 

- Like FIT, SLIM does not combine all of the elements of the wiki – based testing, rather its focus is on invocation of the fixture. 

- This engine works as a separate server that is invoked by the fitnesse wiki engine remotely. 

- Simple POJOs like light–weighted fixtures are allowed by this engine. 

- These fixtures neither use any frame work classes nor extend it. 

- This leads to a simplified design thus allowing the designer to focus up on the call of the SUT properly and in as simple way as possible.

- This also keeps the ways for inheritance open, thus allowing the fixtures to be developed whenever necessary. 

- Inputs and expected output are coupled together to form the tests described in Fitnesse. 

- These couplings are considered to be a kind of variation of the tables. 

- A number of such variations are supported by the Fitnesse testing tool.

Explain FitNesse testing tool? What are principles of Fitnesse?

- FitNesse is an automated testing tool that has been developed to serve as a wiki and a web server for the development of software systems and applications. 

- This testing tool is entirely based up on the frame work for integrated testing that has been developed by Ward Cunningham. 

- It has been designed to support acceptance testing more than unit testing. 

- It comes with the feature of facilitating the description of the system functions in detail.

- With FitNesse testing tool, the users using a developed system can enter the input when it has been specially formatted i.e., non – programmers are able to access this format. 

- FitNesse tool interprets this input and automatically creates all the tests. 

- The system then execute these tests and returns the output to the users. 

- The main advantage of following this approach is that a very fast feedback can be obtained from the users. 

- Support in the form of classes called ‘fixtures’ is provided by the developer of the SUT i.e., system under testing. 

- The credit of writing the code for the fitNesse tool goes to Robert C. Martin and his colleagues in java language. 

- Since the program was developed in java, therefore initially it supported only java but now over the time it has got versions in a number of languages such as Python, C++, Delphi, ruby, C# and so on. 

Principles of FitNesse Testing Tool

This software works on certain principles which we shall discuss now:

FitNesse as a testing method: 

- Originally, it was designed as an interface using the fit frame work and it proved to be highly usable. 

- As such it is known to support the regression tests and black box acceptance tests in an agile style.

- This style of testing involves all the functional testers working in collaboration with the software developers in a software development project in an effort of developing a testing suite. 

- FitNesse testing revolves around the notion of black box testing.

- This involves considering the system to be a black box and testing it in the terms of the output that is automatically generated by the tool in accordance with the given inputs. 

- The responsibility of the functional tester is to design tests in a sense of functionality and expressing the same in the fitNesse tool. 

- On the other hand, the responsibility of the software developer is to connect the tool to the SUT so that tests can be executed and the actual output can be compared to the expected one. 

- The idea that drives this tool is forcing the functional testers and software developers to come up with a common language for an improved collaboration which will eventually lead to an improved mutual understanding of the SUT.

Fitnesse as a testing tool:

- Fitnesse defines the tests as inputs and outputs coupled together. 

- These inputs and outputs coupled together are expressed as variations of a decision table. 

- It supports a number of variations that range from tables that execute queries to tables for literal decision and to tables that express the testing scripts. 

- A free form table is the most generic variation that the designers can interpret in any way they like. 

- However, some sort of table is always used to express the tests. 

- The primary focus of FitNesse is on the easy creation of the tests, thus allowing the testers to maintain a high quality for the tests rather focusing on how the tests are to be executed. 

- Three factors are involved in the creation of the tests through fitNesse:

a)   Easy creation of the tables.

b)   Easy translation of the tables in to calls to SUT.

c)   Maintaining flexibility in the documentation of the tests. 

What is meant by Statistical Usage Testing?

Statistical usage testing is the testing process that is aimed at the fitness of the software system or application.

The test cases chosen for carrying out statistical usage testing mostly consist of the usage scenarios and so the testing has been named as statistical usage testing. Software quality is ensured by the extensive testing but that has to be quite efficient. Testing expenditures covers about 20 – 25 percent of the overall cost of the software project. In order to reduce the testing efforts, deploy the available testing tools since they can create automated tests. But usually what happens is that the important tests require manual intervention with the tester requiring thinking about the usage as well behavior of the software. This is just the repetition of the tasks that were done during the requirements analysis phase.

About Statistical Usage Testing

- A usage model forms the basis for the creation of tests in statistical usage testing.

- Usage model is actually a directed usage graph more like a state machine and it consists of various states and transitions. 

- Every transition state has a probability associated with it regarding the traversal of the transition when the system would be in a state that marks the beginning of the transition arc. 

- Therefore, the sum of the probabilities of outgoing transitions sum up to unity for every state.

- Every transition can be associated with an event and more with parameters that are known to trigger the particular transition. 

- Such event associated transitions can be further related to certain conditions called the guard conditions. 

- These conditions imply that the transition occurs only if the value of the event parameter satisfies the condition.

- For assigning probabilities to the transitions, 3 approaches have been defined as follows:

1. Uninformed approach: In this approach, same probability is assigned to the exit arcs of a state.
2. Informed approach: In this approach, a sample of user event sequences for calculating suitable properties. The sample is captured from either an earlier version of the software or its prototype.
3. Intended approach: This approach is used for shifting the focus of the test to certain state transitions and for modeling the hypothetical users.

- According to a property termed as the marcov property, the actual state is what on which the transition probabilities are dependent. 

- However, they are independent of the history again by the property. 

- This implies that the probabilities must be fixed numbers. 

- A system based up on this property is termed as a marcov chain and it requires conclusion of some analytical descriptions. 

- Usage distribution is one among such descriptions. 

- It gives for every state its steady–state probability i.e., appearance rate that is expected.

- All the states are associated to one or the other part of the software system or application and the part of the software that attracts more attention from the tests is shown by the usage distribution. 

- Some other important descriptions are:

1. Expected test case length
2. Number of test cases required for the verification of the desired reliability of the software system or application.

- The idea of the usage model generation can be extended by handling guard conditions and enabling the non–deterministic behavior of the system depending on the state of the system’s data. 

- All this helps towards the application of the statistical usage testing to systems over a wide range. 

- The use cases are defined by the top–level structure of the unified modeling language (UML). 

Explain Cucumber Testing Tool?

A need was felt for a tool that could create and run automated acceptance tests that were developed in so called BDD or behavior driven development style. Thus, cucumber testing tool came in to existence. 

About Cucumber Testing Tool

- The software for this tool has been developed in ruby programming language. 

- However, the cucumber testing tool does work for the projects beyond the ruby level. 

- For some ruby, cucumber testing tool is like a bridge to go to the target language. 

- For other, gherkin parser still continues to be relevant while they use the target language for the implementation purpose.

- There are certain feature documentations that are created in business – facing text and these can also be executed with the help of cucumber testing tool. 

- Features might be defined using one or more than one scenarios. 

- These are executed using the ruby definition of the test implicit comprising of few steps. 

- For the ruby tool box, the cucumber testing tool proved to be the most popular testing frame work in the month of January 2012. 

- The leading frame work was RSpec. 

- In the same year in the month of April, the cucumber testing tool was moved to a category of the acceptance test frameworks. 

- A survey was carried out which revealed that cucumber testing tool stands 5^th in the ranking of the popular testing frameworks being used by the startup companies.

- Cucumber is just an implementation of the behavior driven development based up on ruby.

- Using it, the user can describe how the system behaves with some natural language comprising of some specific keywords. 

- Firstly, a feature file is created explaining a feature of the software system or application as well as certain scenarios pertaining to a number of different test situations. 

- Secondly, step definitions are written in ruby programming language that explains the purpose of that particular step. 

- Writing those definitions in ruby language provides flexibility in the execution of the test cases. 

- The testing tool comes with definitions for an acceptance testing tool called webrat. 

- This tool can be used for simulation of the browser not having javascript support. 

- It also provides option for making a combination with water that is a web automation frame work and is used for implementing the web automation tests based up on web browser.

- Using cucumber, with this combination counts as a traditional way of using it but anyways it works quite well. 

- In testing using cucumber, a feature is tested against a number of test scenarios that might describe several operations. 

- The step definitions part needs to be explained so that the tool knows what it has to do next. 

- In some of the cases, water is used with water – web driver gem and Rspec gem. 

- Constants can be defined as a reference to the browser and for web.

- Hash PAGES is not required in every case but proves to be beneficial when there are a number of pages involved. 

- The tests thus developed can be executed either when the full web site is ready or during the implementation phase itself. 

- Even if you decide to go with the second option, you will have to run the test one more time when the next release goes out. 

- You have a third option i.e., for using the test as a stand – alone set for the web site. 

- Another plus point of the cucumber testing tool is that it displays in quite a formatted way that displays steps that were both successful and failure. 

- Green color is used for displaying successful steps and red is used for the failed ones. 

- All that has been described for the cucumber implies for the water web testing frame work also. 

What is meant by Behavior Driven Development (BDD)?

BDD or Behavior Driven Development is one of the most important development approaches in the field of software engineering and is just a modified implementation of the TDD or test driven development. It actually combines the guiding principles and general techniques of the test driven development with the combined ideas from two sources namely:

1. Domain – driven design
2. Object – oriented analysis and design

- The behavior driven development thus provides a shared process and tools to business analysts and software developers for their collaboration on the software development process.

- Ideally, behavior driven development represents an idea regarding the management of the software development process by both technical sight and business interests.

- It is assumed in this approach that the specialized software tools are being used to provide support to the development process. 

- The specialized tools thus involved have been developed especially to be used in projects following the BDD approach. 

- They can be considered as special tools that supports test driven development. 

- The primary purpose of these tools is to automate the ubiquitous language statements around which the BDD process is centralized. 

- Agile software development gets successful only when it is considered from the beginning. 

- On the contrast, in some projects this is the last thing to be considered which means that there is no sustainable operation in the websites that serve purposes other than blogging.

-  A blogging web site is just a software that is just modified by the users as it is ready for use. 

- However, BDD and TDD provides the only ways for achieving this goal. 

- The word ‘driven’ in TDD signifies that the test cases are written first and then the code for passing the test. 

- TDD is actually a low level methodology of accomplishing tasks and is sort of developer oriented. 

- On the other hand in BDD, the description of the tests is written in a natural language, thus making the tests more accessible to people outside the development team. 

- Such descriptions can describe the functionality in the same way as done by the specifications. 

Dan North was the person who actually brought the concept of behavior driven development in response to the issues experienced in implementing the test driven development such as:

1. From where to start testing?
2. What is to be tested?
3. How much should be tested in one turn?
4. How the reason for the failure of the test is to be understood?

- At the heart of the behavior driven development approach lies the approach to acceptance testing and unit testing which were again identified by North. 

- He emphasized that the acceptance tests should be written using the user story frame work. 

- Thus starting from the scratch, the development of the BDD approach continued over a couple of years and finally we have it as a communication and collaboration framework. 

- It has been designed especially for the QA people and business participants involved in a project.

- The agile specifications, BDD and testing eXchange meet highlighted the following characteristics of BDD:

1. Second generation methodology
2. Outside – in approach
3. Pull based
4. Multiple stake holder support
5. Multiple scale generation
6. High automation process
7. Agile methodology

- Further, it was said that it involves a cycle of interactions with outputs that are well defined and results in delivery of working software that makes sense. - First, BDD frame work that came in to existence was Jbehave which was followed by Rbehave–a story level BDD ruby frame work.

- All these frameworks were later replaced by cucumber testing tool. It was developed by Aslak Hellesoy. 

What kinds of functions are used by Cleanroom Software Engineering approach?

Harlan Mills and his colleagues namely Linger, Poore, Dyer in the year of 1980 developed a software process that could promise building zero error software at IBM. This process is now popularly known as the Cleanroom software engineering. The process was named in accordance with an analogy with the manufacturing process of the semiconductors. 

The Clean room software engineering process makes use of the statistical process and its control features. The software systems and applications thus produced have certified software reliability. The productivity is also increased as the software has no defects at delivery. 

Below mentioned are some key features of the Cleanroom software engineering process:

1. Usage scenarios
2. Incremental development
3. Incremental release
4. Statistical modeling
5. Separate development
6. Acceptance testing
7. No unit testing
8. No debugging
9. Formal reviews with verification conditions

Basic technologies used by the CSE approach are:

1. Incremental development
2. Box structured specifications
3. Statistical usage testing
4. Function theoretic verification

- The incremental development phase of the CSE involves overlapping of the incremental development and from beginning of specification to the end of the test execution it takes around 12 – 18 weeks.

- Partitioning of the increments is critical as well as difficult. 

- Formal specification of the CSE process involves the following:

1. Box structured Designing: Three types of boxes are identified namely black box, state box and clear box.
2. Verification properties of the structures and
3. Program functions: These are one kind of functions that are used by the clean room approach.

- State boxes are the description of the state of the system in terms of data structures such as sequences, sets, lists, records, relations and maps. 

- Further, they include specification of operations and state in-variants.

- Each and every operation that is carried out needs to take care of the invariant. 

- The syntax errors present in a constructed program in clean-room are checked by a parser but is not run by the developer.

- A team review is responsible for performing verification which is driven by a number of verification conditions. 

- Productivity is increased by 3–5 times in the verification process as compared to the debugging process. 

- Proving the program is always an option with the developers but it calls for a lot of math intensive work.

- As an alternate to this, clean room software engineering approach prefers to use a team code inspection in terms of two things namely:

1. Program functions and
2. Verification conditions

- After this, an informal review is carried out which confirms whether all conditions have been satisfied or not. 

- Program functions are nothing but functions describing the prime program’s function.

- Functional verification steps are:

1.    Specifying the program by post and pre-conditions.

2.    Parsing the program in to prime numbers.

3.    Determining the program functions for SESE’s.

4.    Defining verification conditions.

5.    Inspection of all the verification conditions.

- Program functions also define the conditions under which a program can be executed legally. Such program functions are called pre-conditions.

- Program functions can even express the effect the program execution is having up on the state of the system. Such program functions are called the post conditions.

- Programs are mostly expressed on terms of the input arguments, instance variables and return values of the program. 

- However, they cannot be expressed by local program variables. 

- The concept of nested blocks is supported by a number of modern programming languages and structured programs always require well nesting. 

- The process determining SESE’s also involves parsing rather than just program functions.

What is a Cleanroom approach?

In this article we discuss the cleanroom approach in detail. The size of the team is usually small and is divided in to following three sub – teams:

1. Specification team: This team is responsible for the development and maintenance of the specifications.
2. Development team: This team is responsible for the development and verification of the software.
3. Certification team: This team is responsible for the development of statistical tests and reliability growth models. 

The incremental development is always carried out under statistical quality control so that the performance can be assessed at the end of every iteration using the following measures:

1. Errors per KLOC
2. Rate of growth in MTTF
3. Number of sequential error free tests.

The software development in cleanroom approach is purely based up on the mathematical principles whereas the testing is based up on the statistical principles. 

- Firstly, the system to be developed is formally specified and an operational profile is created. This profile and the formal specifications are then used to define the software increments which are then used for the two purposes namely:

1. Construction of a structured program
2. Designing of statistical tests: These tests also contribute to the first purpose.

- The constructed program is then formally verified and integrated with the increment.

Below mentioned is the flow of cleanroom approach:

1. Software requirements specification
2. Software design and development
3. Incremental software delivery
4. Incremental statistical testing
5. Regression testing
6. Software reliability measurement
7. Process error diagnosis and correction

- The incremental development planning is divided in to two parts namely:

1. Functional specification: It involves formal design correctness verification.
2. Usage specification: It involves statistical test case generation.

- Both these processes then merge down to statistical testing which then follows quality certification model and MTTF estimates.

- The whole cleanroom project develops around the incremental strategy. 

- Requirements are gathered from the customers and elicited and refined via the traditional methods.

- The definition of the data, its behavior and procedures are isolated and separated by the box structures at every level of refinement. 

- Specifications or the black boxes when iteratively refined become state boxes i.e., architectural designs and clear boxes i.e., the component–level designs.

- Formal inspections are carried out to make sure that the code confirms to standards, it is syntactically correct and its correctness has been verified. 

- Statistical usage planning involves creation of tests cases that match with the probability distribution of the usage pattern

- In place of the exhaustive testing, a sample of all the test cases is employed. 

- Once the programmers are done with all 3 activities (i.e., verification, inspection, usage testing, and defect removal) the increment is considered to be certified and ready to be integrated. 

- For developing a right system, customer feedback and involvement are 2 necessary elements throughout the process. 

- Increment planning is required so that the customer’s system requirements can be clarified. 

- There is a requirement of management of resources and control of complexity which is also achieved through incremental planning.

- In order to develop a quality product a control over the software development cycle and process measurement is very much required.

- Following are the benefits of concurrent planning:

1. Concurrent engineering
2. Step wise integration
3. Continuous quality feedback
4. Continuous customer feedback
5. Risk management
6. Change management

- All of the above benefits are achieved respectively by:

1. Certification and scheduling parallel development
2. Testing cumulative increments
3. Statistical process control
4. Through actual use
5. Treatment of the high risk elements in early phases
6. Systematic accommodation of the changes

Design verification advantage allows the cleanroom teams to verify each and every line of code.