What is meant by peer review in software testing?

In the process of software development at every stage the software system or application under development has to be reviewed be it before its completion or after its completion. Many techniques have been designed to carry out an effective review of the software systems or applications.

What is Peer Review in Software Testing?

- Peer review is one such technique employed in software testing.

- Peer review is a type of software review technique which involves the examination of the software product or code or documentation by its author or developer as well as by his/ her colleagues for the evaluation of the quality of the product and its technical content.

- The number of peers for carrying out a peer review may vary from one to any number the author wishes.

- The process of peer review is aimed at providing an engineering practice which is well disciplined for the detection as well as the correction of the bugs and errors in the software product.

- It is also aimed at the prevention of the defects from leaking in to the operational field.

- These aims of the peer review have been defined on the basis of the CMM or capability maturity model.

- Peer reviews form an essential part of the software development cycle and proves to be very helpful in the detection of the bugs in the early stages of the software development.

- A requirements problem as identified by the peer review during the requirements testing is a lot easier and cheaper to rectify than what it would have been in the development stages of the software testing and architecture.

Now the question comes “how the peer review is different from the other kinds of reviews?”

- Peer reviews are somewhat different from their counterpart of management reviews.

- The management reviews are performed by the management representatives rather than by colleagues as in peer review.

- The management reviews are focussed up on the control and management purposes and not on the technical evaluation like the peer review.

- Peer reviews are also a way apart from the software audit reviews in the way that the software audit reviews are carried out by the personnel external for the evaluation of the level of compliance with the standards, specifications and agreements etc.

- Several formal and informal approaches have been designed for carrying out the peer reviews.

- Buddy checking is one informal approach.

- Some formal approaches are:

1. Technical peer reviews
2. Walk throughs and
3. Software inspections.

- The roles, processes and structures for the above mentioned formal processes are governed by the IEEE standards.

- Management representatives are chosen to carry out a peer review only very rarely when specific technical expertise is required or when the documentation to be reviewed is of management level.

- The third formal approach i.e., software inspection involves the assignment of specific roles to the participants, quantification of the stages by defining the entry and exit criteria, and capturing of the software metrics.

- The peer review carried out for the open sources is commonly known as open source review.

Advantages of Peer Reviews

- Peer reviews have always been observed as a powerful methodology for a great improvement in the software quality.

- A peer review can be performed in many forms like an inspection, team review, pair programming, peer desk check or pass around etc.

Which approach is to be followed is decided on the basis of the rigor, cost, spectrum of formality and of course effectiveness!

The cheapest review method should be selected that is more effective in reducing the project associated risks. But, the inspections are always the best approach and are fit for both high level and low level risk projects.

What are the different tips for writing test cases?

What is the foremost thing that should be taken care of to carry out effective testing? Its test cases of course! Writing test cases without thinking is not going to help because any amount of efforts and time you waste on them, you are not going to get effective results. Writing test cases forms a major component of any software testing methodology.

SOME IMPORTANT FACTS

1. Any change in the requirements tends to affect the concerned test case as well. Since the test cases are based upon the requirements, any change in requirements is reflected in the test cases. If you make any changes to the requirements, make sure you have altered the related test case also accordingly.

2. After you finish writing your test cases, revise them twice so that any error that you didn’t notice can be found out.

3. It rarely happens that all the written test cases are executed. So keep the number of test cases limited i.e., as many as you can execute. If you have less number of test cases, you will be able to focus on them properly.

4. Some test cases demand other test cases as their prerequisites. If that is the case, then you should go through the list of test cases and decide a sequence for executing the test cases.

5. Some test cases are dependent on each other and so it becomes difficult which one to execute first. So try to keep your test cases as much independent as possible.

Now keeping above points in mind, we define some tips for writing effective test cases that if followed will certainly reap you the expected results:

1. Keep your test cases as simple as possible. i.e., keep them to the point and precise. You should be able to integrate your test cases with the test plan.

2. Document your test cases.

3. Review all your test cases at least twice.

4. Include input test data along with the test cases because if the tester is to choose test data by himself/ herself, he/ she may omit the most important test data making the whole testing go in vain.

5. While writing test cases always keep in mind that the end user is the one who will use the AUT.

6. Pay due attention while writing the test cases because this will affect the whole testing process.

7. For every condition or assertion in the program, write a different test case so that you don’t mix up.

8. Make sure that your each test case tests only one aspect of the program. Otherwise you will have problems like over lapping and over complicated tests. If one test case focuses upon one aspect, then the code will get good coverage and tracking any error will be easy.

9. Prepare documentation for every test case. Your documentation should have assertions and description of 4 stages (set up, execution, verification, and output) of the test case.

10. Select a basis on which you want to organize your test cases like functionality or class, so that it becomes easy for you to add new test cases later. The order of your test cases should be consistent.

11. The test cases should be able to execute with a good speed, should be independent as discussed above, should be small and precise so that they can be repeated without much hassle is needed to and should be transparent. By transparency I mean that the test case should clearly state its purpose.

12. Your test cases should be reusable.

Project Scheduling - Different ways for tracking the schedule.

A proper project schedule is very important for a successful project delivery. The project schedule is a road map for a software project manager. A project schedule needs to be properly developed. A proper project schedule defines the tasks and milestones that are tracked and controlled as project proceeds. This tracking needs to be accomplished nicely and there are different methods and ways to do so...

- Timely project meetings regarding the status of the project are held and it should be ensured that each team member should report the progress achieved and the problems that are faced.
- There are different reviews that are conducted throughout the software engineering process. It becomes necessary to evaluate the results of all these reviews in order to accomplish proper tracking.
- To accomplish proper tracking, for each task of the project, the actual start date and the planned start date should be compared.
- To accomplish tracking, one should determine whether the formal project milestones are achieved by the scheduled date.
- To accomplish tracking, progress should be assessed quantitatively.

The best indication of progress is the completion and successful review of a defined software work product.
The control administers the project resources, cope with problems and direct the project staff. Control is light if the project is on schedule and within budget. The control is exercised to reconcile the problems if they occur.

In case the problem is identified and diagnosed:
- staff is redeployed
- additional resources should be focused on problem area.

A severe deadline can be managed by time-boxing technique. It chooses an incremental software paradigm and a schedule is derived for each incremental delivery. This technique is done when the product may not be deliverable by a pre-defined deadline.

Some best practices that contribute to improved software testing

There is always a serach for best practices going on. Some are well known and some hidden. Testing does not stand alone. It is intimately dependant on the development practices. These practices have come from many sources. These practices can be divided in three parts:
- Basic Practices
- Foundation Practices
- Incremental Practices

The basic practices are like training wheels which you need to get started and when you take them off, you know ow to ride. The basic practices include:

- Functional Specifications
It is basically a development activity but it is also necessary for software functional test. It defines the external view of an object or procedure. It helps the test generation activity to move in parallel with code development. It helps in the clarity from designer's and an architect perspective.
- Reviews and Inspections
Software reviews and inspection provides a ten times gain in debugging process.
- Formal entry and exit criteria
This practice offers a careful management of software development process. Every process step has a precise entry and exit criteria defined by development process and management keeps a track of the movement from one stage to another.
- Functional test varaitions
This practice involves understanding how to write variations which refers to combination of input condition to yield a output and gain coverage to thoroughly test the function.
- Multi platform testing
Today products run on different platforms so the necessity arises of designing and testing the product for different platforms. This practice addresses aspects of multi platform development and testing.
- Internal betas
Beta means the product is released to few customers and their feedback is recorded. This practice deals with beta programs to best levarage it and reduce cost of external beta.
- Automated test execution
Using automated test execution, the amount of manual work is minimized and higher coverage is gained. This is the best practice which is well understood in some areas. This practice needs to levarage and then develop methods for areas where automation is not fully done.
- Beta programs
- Nightly builds
Nightly build captures frequent builds from changes that being promoted. Advantages include firstly, errors are captured quickly if major regression occurs. Secondly, regression tests can run in background. Thirdly, newer releases of software are available to developers and testers sooner.

Formal Technical Review - Fagan's Inspection Method

Fagan's Inspection Method is introduced by Fagon. Apart from checking codes of programs,it is used to check other work products such as technical documents, model elements, data and code design etc. It follows certain procedural rules that each member should follow:
- The time limit for an inspection meeting is for two hours.
- Inspections are led by a trained moderator.
- Inspections are carried out at a number of points in the process of project planning and systems development.
- All classes of defects in documentation and work product are inspected.
- Inspection is carried out by colleagues at all levels of seniority except the big boss.
- Inspectors are assigned specific roles to increase effectiveness.
- Statistics on types of errors are key, and used for reports which are analyzed in a manner similar to financial analysis.

Different activities that are involved in conducting inspections are:
- Planning is very important and in this case the moderator is asked to build up a plan.
- Presentation should be given which gives an overall overview.
- Each inspector is given 1 to 2 hours alone to inspect the workproduct.
- Meeting should be held in which participants of the meeting are the inspectors, moderator and the developer of the work product.
- The defect list is given for repair.
- Follow up with the repair work.
- Casual analysis meeting is held where inspectors are given a chance to express their personal view on errors and improvements.

What are Formal Technical Reviews (FTR)? What is the aim and guidelines for formal technical reviews?

When tasks are performed in software process, the result is a work product. These results contribute to the development of quality software.
A formal technical review (FTR) is a software quality assurance activity performed by software engineers with the following objectives:
- uncover errors in function, logic or implementation of the software.
- verify that the software meets its requirements.
- ensure that the software has been developed according to the standards.
- achieve uniform software.
- make projects manageable.

Each formal technical review is conducted as a meeting and is considered successful only if it is properly planned, controlled and attended.
The purpose of formal technical review serves as a training ground for junior engineers and to promote backup and continuity.
Constraints of formal technical review meeting’s include 3-5 people involvement, advanced preparation not more than 2 hours for each person, the duration of the review meeting should be less than 2 hours and focus on a specific part of a software product.

There are few guidelines while conducting formal technical reviews. They are:
- Work product should be reviewed and not the developer.
- Make a practice to write down notes while conducting reviews.
- Agenda should be planned.
- Minimize the debate and discussions.
- Keep the number of participants to a minimum and insist on preparing for the review.
- The defect areas should be pointed but no solution should be provided.
- A checklist that is to be reviewed is provided.
- Schedule the reviews as part of the software process and ensure that resources are provided for each reviewer
- Check the effectiveness of review.

Performing User Interface Design - Design Evaluation

Design should be evaluated to check whether it meets the user needs or not once an operational user interface prototype has been created. Evaluation spans a formality spectrum that ranges from an informal test drive in which a user provides an impromptu feedback to a formally designed study.

- After design model has been completed, a first level prototype is created.
- Prototype is evaluated by user.
- User provides designer with direct comments about the efficacy of interface.
- Evaluation cycle continues until no further modifications to interface design are necessary.
- Once first prototype is built, designer collects a variety of qualitative and quantitative data that will assist in evaluating the interface.
- Qualitative data includes questionnaires like simple yes/no, numeric response, scaled response, likert scales, percentage response or open ended.
- Quantitative data can be got by a form of time study analysis can be conducted.

The evaluation criterion applied during early design reviews are:
- length and complexity of written specification of the system.
- number of user tasks specified and average number of actions per task.
- number of actions, tasks, and system states indicated by the design model imply memory load.
- interface styles, help facilities, and error handling protocol provides an indication of interface complexity.

What are different types of general metrics? Continued...

- Design To Requirements Traceability
this metric provides the analysis on the number of design elements matching requirements to the number of design elements not matching requirements. It is calculated at stage completion and calculated from software requirements specification and detail design.

Formula:
Number of design elements.
Number of design elements matching requirements.
Number of design elements not matching requirements.

- Requirements to Test case Traceability
This metric provides the analysis on the number of requirements tested vs the number of requirements not tested. It is calculated at stage completion. It is calculated from software requirements specification, detail design and test case specification.

Formula:
Number of requirements.
Number of requirements tested.
Number of requirements not tested.

- Test cases to Requirements Traceability
This metric provides the analysis on the number of test cases matching requirements vs the number of test cases not matching requirements. It is calculated at stage completion. It is calculated from software requirements specification and test case specification.

Formula:
Number of requirements.
Number of test cases with matching requirements.
Number of test cases not matching requirements.

- Number of defects in coding found during testing by severity
This metric provides the analysis on the number of defects by the severity. It is calculated at stage completion. It is calculated from bug report.

Formula:
Number of defects.
Number of defects of low priority.
Number of defects of medium priority.
Number of defects of high priority.

- Defects - state of origin, detection, removal
This metric provides the analysis on the number of defects by the stage of origin, detection and removal. It is calculated at stage completion. It is calculated from bug report.

Formula:
Number of defects.
Stage of origin.
Stage of detection.
Stage of removal.

- Defect Density
This metric provides the analysis on the number of defects to the size of the work product. It is calculated at stage completion. It is calculated from defects list and bug report.
Formula:
Defect Density = [total number of defects/Size(FP/KLOC)] *100

What are different types of general metrics? Continued...

- Review Effectiveness
This metric will indicate the effectiveness of the review process.It is calculated at the completion of review or completion of testing stage. It is calculated from peer review report, peer review defect list and bugs reported by testing.
Formula:
Review Effectiveness = [(Number of defects found by reviews)/((Total number of defects found y reviews)+Testing)] * 100

- Total number of defects found by reviews
This metric will indicate the total number of defects identified by the review process. the defects are further categorized as high, medium or low. It is calculated at completion of reviews. It is calculated from peer review report and peer review defect list.

Formula: Total number of defects identified in the project.

- Defect vs Review Effort - Review Yield
This metric will indicate the effort expended in each stage for reviews to the defects found. It is calculated at completion of reviews. It is calculated from peer review report and peer review defect list.

Formula : Defects/Review Effort

- Requirements Stability Index (RSI)
This metric gives the stability factor of the requirements over a period of time, after the requirements have been mutually agreed and base lined between company and the client. It is calculated at stage completion and project completion. It is calculated from change request and software requirements specification.

Formula:
RSI = 100 * [(Number of base-lined requirements)-(Number of changes in requirements after the requirements are base-lined)]/(Number of base-lined requirements)

- Change Requests by State
This metric provides the analysis on state of the requirements. It is calculated at stage completion. It is calculated from change request and software requirements specification.
Formula:
Number of accepted requirements, Number of rejected requirements, Number of postponed requirements.

- Requirements to Design Traceability
This metric provides the analysis on the number of requirements designed to the number of requirements that were not designed. It is calculated at stage completion. It is calculated from software requirement specification and detail design.
Formula:
Total number of requirements, Number of requirements designed, Number of requirements not designed.

What are different types of general metrics? Continued...

- Overall Review Effectiveness
This metric will indicate the effectiveness of the testing process in identifying the defects for a given project during the testing stage. It is calculated at monthly basis and after build completion or project completion. It is calculated from test reports and customer identified defects.

Overall Test Effectiveness OTE = [(Number of defects found during testing)/(Total number of defects found during testing + Number of defects found during post delivery)] *100

- Effort Variance (EV)
This metric gives the variation of actual efforts vs. the estimated effort. This is calculated for each project stage. It is calculated at stage completion as identified in SPP. It is calculated from estimation sheets for estimated values in person hours, for each activity within a given stage and actual worked hours values in person hours.

EV = [(Actual person hours - Estimated person hours)/Estimated person hours] * 100

- Cost Variance (CV)
This metric gives the variation of actual cost vs the estimated cost. This is calculated for each project stage. It is calculated at stage completion. It is calculated from estimation sheets for estimated values in dollars or rupees for each activity within that stage and the actual cost incurred.

CV = [(Actual Cost-Estimated Cost)/Estimated Cost] * 100

- Size Variance
This metric gives the variation of actual size vs the estimated size. This is calculated for each project stage. It is calculated at stage and project completion. It is calculated from estimation sheets for estimated values in function points or KLOC and from actual size.

Size Variance = [(Actual Size-Estimated Size)/Estimated Size] * 100

- Productivity on Review reparation - Technical
This metric will indicate the effort spent on preparation for review. This is to use this to calculate for languages used in the project.It is calculated at monthly or after build completion. It is calculated from peer review report.

For every language used, calculate
(KLOC or FP)/hour(*Language) where Language - C,C++, Java,XML etc...

- Number of defects found per review meeting
This metric will indicate the number of defects found during the review meeting across various stages of the project. It is calculated at monthly or after the completion of review. it is calculated from peer review report and peer review defect list.

Formula : Number of defects/Review Meeting

- Review Team Efficiency(Review Team Size Vs Defects Trend)
This metric will indicate the review team size and the defects trend. This will help to determine the efficiency of the review team. It is calculated at monthly and completion of review. It is calculated from peer review report and peer review defect list.

Formula : Review team size to the defects trend.

Verification Strategies - Reviews - Design Review and Code Review

Design Review: A process or meeting during which a system, hardware, or software design is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Types include critical design review, preliminary design review and system design review.Quality assurance team member leads design review. members from development team and QA team participate in the review.

Input Criteria: Design document is an essential document for the review. A checklist can be used for the review.
Exit Criteria: It includes the filled and completed checklist with the reviewers comments and suggestions and the re-verification whether they are incorporated in the documents.

Code Review: A meeting at which software code is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. QA team member(in case the QA team is only involved in black box testing then the development team lead chairs the review team) leads code review. Members from development team and QA team participate in the review.

Input Criteria: The Coding Standards Document and the Source file are the essential documents for the review. A checklist can be used for the review.
Exit Criteria: It includes the filled and completed checklist with the reviewer's comments and suggestions and the re-verification whether they are incorporated in the documents.

Verification Strategies - Reviews - Technical Reviews and Requirement Review

Technical reviews confirm that product conforms to specifications, adheres to regulations, standards, guidelines, plans, changes are properly implemented, changes affect only those system areas identified by the change specification.
The main objectives of technical reviews are as follows:
- Ensure that the software confirms to the organization standards.
- Ensure that any changes in the development procedures are implemented as per the organization pre-defined standards.

In technical reviews, the following software products are reviewed:
- Software requirements specification.
- Software design description.
- Software test documentation.
- Software user documentation.
- Installation procedure.
- Release notes.
The participants of the review play the roles of decision-maker, review leader, recorder, technical staff.

Requirement Review : A process or meeting during which the requirements for a system, hardware item or software item are presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Types include system requirements review, software requirements review. Product management leads the requirement review. Members from every affected department participates in the review.

Input Criteria: Software requirement specification is the essential document for the review. A checklist can be used for the review.
Exit Criteria: It includes the filled and completed checklist with the reviewers comments and suggestions and the re-verification whether they are incorporated in the documents.

What is Verification and different strategies of Verification - Reviews and Management Reviews

Verification is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase.
Verification process helps in detecting defects early and preventing their leakage downstream. Thus, the higher cost of later detection and re-work is eliminated.

The different strategies in Verification are:

Review

A process or meeting during which a work product or set of work products is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. The main goal of reviews is to find defects. reviews are a good compliment for testing to help assure quality. Few purposes of SQA reviews are assuring the quality of deliverable's before the project moves to the next stage. Once a deliverable has been reviewed, revised as required, and approved, it can be used as a basis for the next stage in the life cycle.
Types of reviews include management reviews, technical reviews, inspections, walk-through and audit.

- Management Reviews :
These reviews are performed by those directly responsible for the system in order to monitor progress, determine status of plans and schedules, confirm requirements and their system allocation. The main objectives of management reviews are:
* Validate from a management perspective that the project is making progress according to the project plan.
* Ensure that the deliverables are ready for management attention.
* Resolve issues that require management attention.
* Identify any project bottlenecks.
* Keeping the project in control.
Support decisions are made during such reviews include corrective actions i.e. changes in the allocation of resources or changes to the scope of the project.
In management reviews, audit reports, contingency plans, installation plans, risk management plans, software q/a are reviewed.
The participants of the review play the roles of decision maker, review laeder, recorder, management staff and technical staff.

Introduction To Software Reviews

What are software reviews?
- serves as a “filter” for the software engineering process.

Purpose: Software reviews serve to uncover errors in analysis, design, coding, and testing and are hence invaluable.

Why do software reviews? The general principle behind doing reviews are:
- To err is human.
- It may not be easy to catch the errors in engineers’ work, the kind of errors that are not found in bugs.
- Finding errors early in the process is always more efficient than detecting errors much later in the cycle.

A review is a way to :
- identify needed improvements of the various parts and modules of a product.
- confirm the improvement areas of a product (if the improvements had been suggested by others).
- achieve technical work that can be more uniform, predicable, and manageable, making for a more better end result and a smoother cycle.

What are the different types of reviews:
- Informal reviews : informal meeting and informal desk checking.
- Formal reviews : Walkthrough, inspection, and round-robin reviews.

Formal Technical Reviews are used to improve work products. As a part of this process, any deliverable that is produced during the development life-cycle is eligible to be reviewed and should be picked up for review starting at an early stage, beginning in the specification stage and recurring at each identifiable milestone.

- A Review meeting is an important part of the of FTR (Formal Technical Review). There are are some essential parameters for the meeting, such as
1. There should be a reasonable number of persons conducting the meeting, not being a free-for-all
2. Each one of them should have done his/her homework and other preparation
3. The meeting should not be carried out for very long and should have a well defined agenda to avoid wastage of time. The duration should be just enough to churn out some constructive results.
4. The Formal Technical Review is effective when a small and specific part of the overall software is under scrutiny.
5. A review is more productive when the review artefacts have been broken down into small modules that can be reviewed independently. The target of the FTR (Formal Technical Review) is on a component of the project, a single module.

- Record keeping and reporting are additional quality assurance activities. They are applied to different stages of software development. During the FTR, it is the responsibility of reviewer (or recorder) to record all issues that have been raised.
At the end of review meeting, a review issue list that summarizes all issues, is produced. A simple review summary report is also compiled. A review summary report answers the following questions.
1. What was reviewed?
2. Who reviewed it ?
3. What were the findings and conclusions?

- Review Guidelines : A minimum set of guidelines for FTR:
* Review the product, not the producer.
* Set an agenda and maintain it.
* Limit debate and rebuttal.
* Enunciate problem areas, but don’t attempt to solve every problem noted.
* Take written notes.
* Limit the number of participants and insist upon advance preparation.
* Develop a checklist for each work product that is likely to be reviewed.
* Allocate resources and time schedule for FTRs.
* Conduct meaningful training for all reviewers.
* Review your early reviews.