Usability testing methods

Usability testing can take the form of several methods and techniques (depending on the situation and need):

Cognitive Walkthrough: Performed continually during the development cycle, cognitive workflows try to evaluate the system from a user's thought process that help in decision making (these include the ability to reason as well as other factors such as memory load). This will help in terms of understanding how the system will seem like to a infrequent user. The input for such users could be in the form of paper workflows or through a working prototype. More details can be found at this link.

Focus Groups: Focus groups are actually a method used by marketing companies whereby a group of people from the target segment are brought together in a discussion type of format. A person from the product team acts as the moderator, and prepares a list of issues to be discussed. More details from this site (link).

GOMS: Goals, Operators, Methods, and Selection Rules; a slightly difficult mechanism to evaluate and model human task performance.

Prototyping: Prototyping is a very famous method for displaying a system for internal users and for external usability testers. Prototypes can range from paper models to actual almost working types of prototypes.

Walk-Around Review: This type of study is meant to harness the observations of other people in the company who are not part of the engineering team. The idea is to place paper mockups on the walls of the company along with space for comments so that people can write their comments; in addition, early builds can be provided to company people for them to use and provide comments on. Such a method can provide a lot of useful tips (and have an incidental benefit of providing greater visibility to your project).

Field Observation: Representatives of the product team go to the actual user workplaces and observe how they use the software in their daily lives (example, if a digital imaging application for professional photographers is being developed, then a good way is to have multiple days where people tag along with photographers and observe their usage of the application).

Task Analysis: # An analyst determines the user goals and tasks, then makes recommendations aimed at increasing efficiency and user-friendliness. The objective is to determine how the user actually feels comfortable in using software / sites.

Interviews/Observations: These are one-to-one discussions. Do a thorough discussion with them about what they do, and observe their usage at the same time. This will help in doing a realistic determination about their pattern of usage.

Usability Inspection: reviews a system based on a set of usability guidelines. Experts familiar with issues of usability in design perform the usability inspection.

User Testing: Users (sampled from the final target customer segment) use the final software / site. These can be final users from the client side (if the software is developed for a specific client), or can be a sample set of users. Usability experts study their interactions and figure out what are the paint points and improvements needed.

About usability testing and timing

Suppose you are in a tight development cycle. You have to deliver either a new product, or the next version of an existing product. Getting the features of a product right is always a touch task, given that there are a number of competing features that seem important, and prioritizing the features is something that is very important. This decides the priorities that the engineering team (the feature development team) will follow during the development cycle.
How is this priority actually decided ? If the company is in in the business of defining an absolutely new product that has not been conceptualized as yet, then getting some feedback from prospective customers is difficult; however, if there are already customers using an existing product (from the same company or a rival company), then it is absolutely essential that these users be polled for the features so that there is a good idea about the features that are most critical (it would also help to identify features that customers would be willing to pay a premium for).
Now consider that we are in the development phase of the project lifecycle, where the UI team works along with the engineering team to define the workflow for the feature. There is a lot of discussion around what the feature should be like (with a possibility of the discussion getting heated as a regular part of feature discussion), and eventually most people agree to what the feature should be like. The UI specs of the feature are drawn up and the feature implementation is based on the spec. At this point, everything may seem settled, but it is critically important that this final implementation be evaluated for usability issues. At this point, the team needs to find a set of people who would adequately represent the final set of users, and get them to see the feature working in the actual product. Such usability testing will help determine whether the determined final feature is actually something that the users can accept, or whether there are problems that need to be modified.
The timing of such user testing is most critical. Typically, such workflows reach a final form close to the end of the cycle, and this is the form in which users can actually exercise the workflows. However, in a contra effect, this time is also very late in the cycle, and the team will be hesitant to accept changes that are significant, since the amount of time required to make these changes may not be easily available.
What is the solution ? The solution that seems to work is to have a much more active involvement with users, starting with showing them mockups as the workflow gets more concrete, active question and answer sessions about what they may be looking for, till the time that they can review the actual product implementation. Further, if a workflow is very new and contentious, then it would make sense to try and complete it earlier. And finally, there needs to be time built into the schedule to take such changes.

TYPES OF SOFTWARE SYSTEMS

BATCH SYSTEMS : They are a set of programs that perform certain activities which do not require any kind of input from the user. Batch systems contain one or more API which perform various tasks.

Ex: When something is typed on a word document by pressing a key, same is displayed on the monitor. The process of conversion of user input of the key to machine language and then displaying what you have typed on monitor is performed by batch system.

EVENT CONTROL SYSTEMS : These systems process real time data to provide the user with results for the command that has been given.

Ex: When you press Ctrl + S, it tells the computer to save the document. This communication is provided by the event controls that are pre-defined in the system.

PROCESS CONTROL SYSTEMS : When two systems communicate, the co-ordination or data transfer becomes vital. Process control systems are the one’s that receive data from a system and instruct the sending system to perform specific tasks based on the reply sent by the receiving system.

PROCEDURE CONTROL SYSTEMS : These systems are one’s which control the functions of another system.

MESSAGE PROCESSING SYSTEMS : A simple example to explain these kind of systems is the SMS management software used by the mobile operator which handles the incoming and outgoing messages.

DIAGNOSTIC SOFTWARE SYSTEMS : These systems help in diagnosing the computer hardware components.
Ex: The “NEW HARDWARE FOUND” dialogue seen when you plug in a new device to the computer is an example of these kind of systems.

SENSOR & SIGNAL PROCESSING SYSTEMS : In these systems the computer receives input in form of signals and then transforms the signals to a user understandable output.

Deadlock - algorithms

BANKER’S ALGORITHM

Consider an example:

There are four customers: A, B, C and D, which are analogous to four processes.
The credit unit is like the resource
The banker himself is the OS
Assume each credit unit = Rs. 1000.

Not all customers need their maximum credit immediately. Hence only 10 credit units are reserved.
Process Current Max. Free = 10
A 0 6 B 0 5 C 0 4 D 0 7

How does the Algorithm Work?

When a new process (customer) enters the system, it (he) must declare the maximum number of instances of each resource type (credit units) that it (he) may need. This number may not exceed the total number of resources (credit units) in the system. When a user (customer) requests a set of resources (credit unit), the system must determine whether the allocation of these resources will leave the system in a safe state. If it will,
the resources are allocated; otherwise, the process must wait until some other process releases enough resources. Consider current allocation to various processes is as shown below.
Process Current Max. Free = 2
A 1 6 B 1 5 C 2 4 D 4 7
Would the System be in a Safe State?
‘C’ requests 2 additional units and gets them. It then runs to completion and frees all the resources it has.
Process Current Max. Free = 4
A 1 6 B 1 5 C 0 - D 4 7
Now either ‘B’ or ‘D’ can request and run to completion. Assume ‘B’ requests 4 additional units and gets them. It then runs to completion and frees all its resources. Process Current Max. Free = 5
A 1 6 B 0 - C 0 - D 4 7
Now ‘D’ runs and requests 3 additional resources and gets them. It then runs to completion and releases all its resources.Process Current Max. Free = 9
A 1 6 B 0 - C 0 - D 0 -
Finally ‘A’ runs and requests 5 additional resources and gets them. It then runs to completion and releases all its resources. Process Current Max. Free = 10
A 0 - B 0 - C 0 - D 0 -
Here is the complete banker’s algorithm:

Several data structures must be maintained to implement banker’s algorithm. Let n be the number of processes and m be the number of resource types. The data structures needed are:
• Available : A vector of length m indicates number of available resources of each type. If Available[j] = k, there are k instances of resource type Rj available.
• Max : A n*m matrix defines the maximum demand of each process. If Max[i,j]=k, then Pi may request at most k instances of resource type Rj.
• Allocation : An n*m matrix defines the number of resources of each type currently allocated to each process. If Allocation[i,j]=k, then process Pi is currently allocated k instances of resource type rj.
• Need : An n*m matrix indicates the remaining resource need of each process. If Need[i,j]=k, then Pi may need k more instances of resource type Rj to complete its task. Need[i,j]=Max[I,j] – Allocation[I,j].

After defining the data structures, algorithm moves into two phases :
• Safety Algorithm
• Resource Request Algorithm


SAFETY ALGORITHM

The safety algorithm is for finding out whether or not a system is in a safe state. It is described below:

1. Let work and finish be vectors of length m and n
respectively. Initialize work = Available and Finish[I] = false for all I = 1, 2, …, n.
2. Find an I such that both
• Finish[I] = false
• Needi <= work
If no such I exists, go to step 4.
3. Work = work + allocationi
finish[I] = true
go to step 2

4. If finish[I] = true for all I, then the system is in a safe state. This algorithm may require an order of m * n2 operations to decide whether a state is safe.

RESOURCE – REQUEST ALGORITHM

Having determined that the system is safe, this algorithm grants the requested resources to the process. Let Request i be the request vector for process Pi. If Request[j] = k, then process Pi wants k instances of resource type Rj. When this request is made, the following actions are taken:

1. If request I <= need I, then go to step 2. Otherwise raise an error condition because the process has exceeded its maximum claim.
2. If request I <= available, go to step 3. Otherwise, Pi must wait since the resources are not available.
3. Have the system pretend to have allocated the requested resources to process Pi by modifying the state as follows:
a. Available = available – request I
b. Allocation = allocation + request I
c. Need I = Need I – request I
4. Call the Safety algorithm. If the state is safe, then transaction is completed and process Pi is allocated the resources. If the new state is unsafe, then Pi must wait and the old resource allocation state is restored.

Deadlock situation

A deadlock is a situation wherein two or more competing actions are waiting for the other to finish, and thus neither ever does. In the computing world deadlock refers to a specific condition when two or more processes are each waiting for another to release a resource, or more than two processes are waiting for resources in a circular chain.

Under the normal mode of operation, a process may utilize a resource in the following sequence:

Request – when a process needs some resource, it requests for it. Process has to wait if resource is busy.
Use – The process should operate on the resource.
Release – After using, process should release the resource.

Conditions In Which Deadlock Can Arise

• Hold and Wait : As the name suggests, a process is holding a resource and also waiting for some other resources that are held by other process.
• No Preemption : A resource can be released only voluntarily by the process holding it, after that process has completed the task.
• Mutual Exclusion : It means only one process can use the resource at a time. If some other process requests that resource, the requesting process has to wait until the resource is released.
• Circular Condition : There must exist a state of waiting processes in which process P0 is waiting for a resource that is held by P1, P1 waiting for resource held by P2, Pn-1 waiting for resource held by Pn an Pn is waiting for resource held by P0.

Defect Management

What is a defect?

For a test engineer, a defect is:
• Any deviation from a given specification.
• Anything that causes user dissatisfaction
• Incorrect output
• Software is not behaving the way it should behave.

Difference between bug, defect and error:-
• Software is said to have a bug if the features deviate from the specifications.
• Software is said to have a defect if it is having some unwanted side effects.
• Software is said to have an error if it is giving an incorrect output.

DEFECT TAXONOMIES

All software defects can be broadly categorized into the below mentioned types:
• Errors of Commission : when something wrong is done.
• Errors of Omission : when something is left out by accident.
• Errors of Clarity & Ambiguity : when there are different interpretations.
• Errors of speed & capacity.
• Conceptual or Design bugs.
• Coding bugs
• Integration bugs.
• User interface errors.
• Functional errors.
• Communication errors.
• Missing commands
• Performance
• Output
• Error handling errors.
• Boundary related errors
• Calculation errors
• Control flow errors.
• Race condition errors.
• Load condition errors.
• Hardware errors.
• Source & Version control errors.
• Documentation errors.
• Testing errors.

Summary of testing needs during the SDLC

There is an often asked question about when testing should occur in the overall product development cycle?

Testing is sometimes incorrectly thought as an after-the-fact activity. Instead, testing should be performed at every development stage of the product. Test data sets must be derived and their correctness and consistency should be monitored throughout the development process. Testing should accompany each phase of “Software Development Life Cycle”. If testing is isolated as a single phase late in the cycle, errors in the problem statement or design may incur exorbitant costs. If this happens, not only must the original error be corrected, but the entire structure built upon it has to be changed. Therefore, testing should be involved throughout SDLC in order to bring out a quality product.

Testing Activities In Each Phase:

1. REQUIREMENTS ANALYSIS
• Determine correctness
• Generate functional test data.

2. Design
• Determine correctness and consistency
• Generate structural and functional test data.

3. PROGRAMMING
• Determine correctness and consistency.
• Generate structural and functional test data.
• Apply test data.
• Refine test data.

4. OPERATION & MAINTENANCE
• Retest

Of course, if you want to add examples of your own to substantiate this, please do so through the comments section.