Cleanroom Testing - Statistical Use Testing and Certfication

Clean room testing is different from conventional testing approaches. The goal of clean room testing is to validate software requirements by demonstrating that a statistical sample of use-cases have been executed successfully.

Statistical use testing tests the software in a way the users intend to use it. the clean room testing teams determine the usage probability distribution or the software. Each increment's specification is analyzed to define a set of inputs or events that cause the software to change its behavior. A probability of use is assigned to each input or event based on the interviews with users, the creation of usage scenarios and a general understanding o application domain. The testing team executes these use cases and verifies software behavior against the specification for the system. Using the recorded interval times, the certification team can compute mean time to failure.

Within the clean room software engineering approach, certification implies that the reliability can be specified for each component. Reusable software components can be stored along with their usage scenarios, program stimuli, and probability distributions. The certification approach involves :
- usage scenarios must be created.
- usage profile is specified.
- test cases are generated from profile.
- tests are executed and failure data are recorded and analyzed.
- reliability is computed and certified.

Certification for clean room software engineering requires creation of three models:
- Sampling Model : Certification is given if no failure or a specified number of failures occur after executing m random test cases. The value of m is derived mathematically.
- Component Model : This model enables the analyst to determine the probability that component i will fail prior to completion.
- Certification Model : The overall reliability of the system is projected and certified.

Cleanroom Software Engineering - Design Refinement and Cleanroom Testing

Design Refinement & Verification
- If a function f is expanded into a sequence g and h, the correctness condition for all input to f is:
• Does g followed by h do f?
When a function f is refined into a conditional (if-then-else), the correctness condition for all input to f is:
• Whenever condition is true does g do f and whenever is false, does h do f?
- When function f is refined as a loop, the correctness conditions for all inputs to f are:
• Is termination guaranteed?
• Whenever is true does g followed by f do f, and whenever is false, does skipping the loop still do f?

Advantages of Design Verification :
- It reduces verification to a finite process.
- It lets cleanroom teams verify every line of design and code.
- It results in a near zero defect level.
- It scales up.
- It produces better code than unit testing.

CLEANROOM TESTING :
The strategy and tactics of cleanroom testing are fundamentally different from conventional testing approaches.
Statistical Testing
- Generation of test cases
* each test case begins in a start state and represents a random walk through the usage model ending at a designated end state.
- Control of statistical testing
* a well-defined procedure is performed under specified conditions.
* each performance is a trial and can be used as part of an empirical probability computation.
- Stopping criteria for testing
* when testing goals or quality standards are achieved.
* when the difference between the predicted usage chain and the actual testing chain becomes very small.