Orthogonal Array Testing Strategy (OATS)

The Orthogonal Array Testing Strategy (OATS) is a systematic, statistical way of testing pair-wise interactions. It provides representative (uniformly distributed) coverage of of all variable pair combinations. This makes the technique particularly useful for integration testing of software components (especially in OO systems where multiple subclasses can be substituted as the server for a client). It is also quite useful for testing combinations of configurable options (such as a web page that
lets the user choose the font style, background color, and page layout).
Orthogonal Array Testing is a statistical testing technique implemented by Taguchi. This method is extremely valuable for testing complex applications and e-comm products. The e-comm world presents interesting challenges for test case design and testing coverage. The black box testing technique will not adequately provide sufficient testing coverage. The underlining infrastructure connections between servers and legacy systems will not be understood by the black box testing team. A gray box testing team will have the necessary knowledge and combined with the power of statistical testing, an elaborate testing net can be set-up and implemented.

The theory:
Orthogonal Array Testing (OAT) can be used to reduce the number of combinations and provide maximum coverage with a minimum number of test cases. OAT is an array of values in which each column represents a variable - factor that can take a certain set of values called levels. Each row represents a test case. In OAT, the factors are combined pair-wise rather than representing all possible combinations of factors and levels.
Orthogonal arrays are two dimensional arrays of numbers which possess the interesting quality that by choosing any two columns in the array you receive an even distribution of all the pair-wise combinations of values in the array.

Why use this technique?
Test case selection poses an interesting dilemma for the software professional. Almost everyone has heard that you can't test quality into a product, that testing can only show the existence of defects and never their absence, and that exhaustive testing quickly becomes impossible -- even in small systems.
However, testing is necessary. Being intelligent about which test cases you choose can make all the difference between (a) endlessly executing tests that just aren't likely to find bugs and don't increase your confidence in the system and (b) executing a concise, well-defined set of tests that are likely to uncover most (not all) of the bugs and that give you a great deal more comfort in the quality of your
software.

The basic fault model that lies beneath this technique is:
1. Interactions and integrations are a major source of defects.
2. Most of these defects are not a result of complex interactions such as "When the background is blue and the font is Arial and the layout has menus on the right and the images are large and it's a Thursday then the tables don't line up properly."
3. Most of these defects arise from simple pair-wise interactions such as "When the font is Arial and the menus are on the right the tables don't line up properly."
4. With so many possible combinations of components or settings, it is easy to miss one.
5. Randomly selecting values to create all of the pair-wise combinations is bound to create inefficient test sets and test sets with random, senseless distribution of values.

OATS provides a means to select a test set that:
1. Guarantees testing the pair-wise combinations of all the selected variables.
2. Creates an efficient and concise test set with many fewer test cases than testing all combinations of all variables.
3. Creates a test set that has an even distribution of all pair-wise combinations.
4. Exercises some of the complex combinations of all the variables.
5. Is simpler to generate and less error prone than test sets created by hand.