Resources for usability testing

Previous articles covered some details about usability testing. I hope by now I have been able to cover some details of what usability testing is, why it is important, and some tips / issues that one needs to cover. However, there is always more that you can explore, read about, and hence you should have an idea of some of the secrets to having a good usability testing program.
Infodesign.com.au (link)
Accessibility forum (link)
Userfocus.co.uk (link)
Web Site Usability (link)
Web usability (link)
Usability.gov (link)
Doug Ward's website (link)
Usability testing (link)
deyalexander.com (link)
Column by Jakob Nielsen (link)

An example of unrealistic expectations in the service industry

This is a true example from around 6 years back when I was working for an IT software solutions provider (the firm did software projects for different customers). This was a decent sized company that had something like 12000 people on the rolls, doing everything from development to testing to requirements analysis, and so on. I was more into the area of a business analyst, translating the requirements document into a form that the developers working on the project would understand.
This was a new project with a new medium sized bank in the Midwest, and the hope was that we would be able to do this project well enough and give them a system that would work so well for them that they would continue with the company and be the start of a long and serious (and profitable) relationship. Sounds good, right ? Well, read on.
Around this time, our company, that was a publicly listed company was getting on just like the other service companies of that time, doing okay, but not generating great figures. Management was getting hit by analysts, and passed on a directive that every project needs to meet the company defined margin. Exceptions only when pleaded before the executive committee, and not otherwise. Implicit was the expectation that anybody who does a project that does not promise enough margin would need to explain the project.
Now, since our project was with a new customer with whom we had high hopes for the future, we could not charge our expected rates; after all, why would the customer then select us ? So, our account manager along with the Vice-President of the unit went ahead and quoted a rate that was atleast 20% lower (getting fewer people assigned to the project than necessary). Guess what ? Pretty soon, the strains and missing people started to show.
Ego also plays a part. For a Vice-President to go before the committee and plead for more money (a reduction in margin) would reflect adversely. The members of the committee, who might be expected to provide an experience of being able to handle these kind of situations and offer some latitude did not do so since they were never offered this project for review. Pretty soon, somebody senior in the team had the bright idea that weekends could be converted into work hours (maybe 1 weekend in 3 could be off), and this idea was implemented with gusto.
You can guess the rest. People from outside the project did not want to join, quality reviews of the project were hesitant because of the many exceptions, and eventually the customer could make out that the quality was not as desired. Project over, account over, and pretty soon the project manager and other senior team members quit and went to other companies.
This was a disaster caused by the management reacting adversely to poor numbers, and unwilling to exercise the due diligence in doing a project (after all, the first criteria for a project should be to make it successful).

How many of you have similar experiences ?

Hiring a consultant for generating your requirements: Part 2 of an external article

The previous article presented the DragonPoint.com article outlining 5 essential steps that you need to take when you have people hired to do requirements analysis for a project. Well, here is the link to the part 2 of this article that outlines tips 6 - 10 for the requirements gathering process (link). These next 5 tips are:
6. Make resources available: You need to budget for making sure that the requirements capturing team can access the current system and employees.
7. Include the employees who actually use the system: It is critical that the requirements gathering team get access to actual users who use the system, not only to the management people who drive the project.
8. Let your employees know their input is important: Make sure that the employees working with the requirements team understand that they need to help the team. At the same time, employees who are working with the team can provide an impression of the competence of the requirements team (are they asking the right questions, are they focused on the current processes, etc)
9. Remember why you hired the consultant: Do not take anything for granted. Explain your process in full detail.
10. Take ownership of the project: Hopefully not necessary to explain this. The project is important, and you need to make sure that it gets full importance.

Requirements gathering: DragonPoint Article (#1)

Requirements gathering for a software project (that you really need to expand or maintain your business) is a tough job; the toughness is because it is not an exact science. It depends on the type of project, depends on the nature of the people who will enumerate the requirements, and it depends on the capability of the team that does the actual requirements gathering. In such a scenario, it makes sense to read as much as possible about the requirements process so that one is aware of the best practices, about different use cases in which the process has worked, and so on.
A lot of companies that want to get hire consultants for this critical stage, and here is a great first part of an article at Dragon Point that talks about 5 steps (out of a total of 10) to become better at requirements gathering:

Any needs not identified during the requirements capture stage will result in scope creep. According to Suzie DeBusk, President of DragonPoint, the most effective way to minimize scope creep is to allocate 30% of the time spent on a project to requirements capture, design, and review. How does requirements capture reduce scope creep?
The requirements capture stage is similar to the planning phase of a construction project. If the client and architect do not communicate effectively, the blueprints will not meet the client's needs. And, depending on the size and scope of the discrepancies, this error in communication can result in costly rework as the project evolves.

As per the article, the following are the main points:
1. Be prepared for the initial consultation.
2. Remember your current system
3. Communicate.
4. Look for listening.
5. Listen for insightful questions that demonstrate you and your consultant share common goals.

Guidelines for usability testing

When we were going in for usability testing, I was part of the team that would evaluate the results from the usability testing, and it was important for me to understand more about the usability testing process. Besides understanding the need for usability testing and the process, it was also important for the team to understand what the guidelines were for usability testing. They would also help understand how usability testing works. Some of the guidelines (both from reading the literature on this subject, and also from overall study of the process in practise) that I learnt were:

1. Deciding the target audience for your usability testing: Given that the outcome of the usability testing will help in determining changes to your design strategy, it is absolutely essential that the usability testing be done among people who are good representatives of the target subject; this means that the selection should be done carefully. Avoid the temptation to cut corners, or to select neighbors who seem to represent the target segment. Devise a set of queries that will help determine if the selected people actually are good representatives; similarly, use experts to help you get good target subjects. And, other than a few cases, don't let them know the name of the company - it may cause bias to appear in their results.

1. a) Different users may specialize is separate workflows, for example, if you want to get testing done of a new shopping site catering to working professionals, you should design an appropriate query form. Review the answers so that you can get an idea of which area a user would move more towards; this may help in deciding the exact set of users.

2. Before actually starting the testing: Remember, your tests will only be good when the people taking part are comfortable with the whole process. They need to feel that the environment in which the test is taking place is similar to that of their home or work environment. And of course, stay easy on the legalese - you may want to make sure that your in-process development is safe and want them to sign NDA's, but if you may it full of complicated legal terms, they are likely to get confused. If your office is not exactly very accessible, consider doing the testing in a more convenient location.

3. Starting the usability testing: Don't plunge the users directly into the testing process. Get talking with them, explain what the website is about and what the URL is; get some initial feedbacks about what they expect from such a site. If they mention some phrases or some other such term, then it is good to understand these statements or terms. Getting some words of polite conversation in makes them more comfortable.

4. Decide what you want to get tested: When deciding to select tasks for review by users, it is absolutely essential that you dump the notion about favorite sections of the site. You may have added a great new section that was very difficult to develop, but if it is not critical to the success of the site, then it is not a high priority to get an evaluation done. You should select tasks that are critical for the success of the site.

5. Scenarios are always better for such evaluations: Again, you need to talk in the language of the customer. You may have had great internal debate on the naming of features, but if you asking customers to evaluate some flows, then you should:
- Ask them to try out workflows / use cases (for example, you need to find some white shoes for your kid, and pay for them through the card that you normally use)
- Use simple language (avoid more technical words such as payment processing, and instead use phrases such as payment options)
- Set tasks that have a logical conclusion

6. The actual task execution: It is almost a core logic of usability testing that people should not be tasked very heavily during the process of usability testing. Get them to do one task at a time, and focus on their responses during the execution of the task. In a lot of cases, users may have to be given inputs during the course of the task (say, you want them to test out the convenience of posting videos to YouTube); in such cases, make sure that they have the equipment needed for the task (in this case, they either have a sample of home videos, or they have a camera readily available)

7. The participant is not at fault: Sometimes, you get users struggling during the usability test, or they get stuck at places where you would think that things are very simple. Remember, all their feedback is important, and if they are not able to do some tasks or part of tasks, then it is most likely a reflection of the task rather than any inherent inabilities on the part of the contestants. Further, if they are confused by something, or they have to make a choice, don't guide them down some path (you would most likely introduce a bias if doing so). If you are asked a direct question, then reply, but not venture an opinion.

8. Don't get distractions into place: Once the user has started, minimise distractions. Prevent people from coming or going in the location, and this includes a lot of traffic in front of the room where the test is happening. If people need to see, then they should see this on a video conference or some other facility. Test subjects can get conscious if there are too many there.

9. The user has completed their testing: If the user is done, then you need to gather as much information as possible. Ask the user about their impressions, about what worked well and what did not. Ask about what they feel could be done to improve things, and whether what they saw was how they would have done things. This could involve also asking them about what they recall about the software or the site - this helps in highlighting the parts that remain in the minds of the tester.

10. Go through the recording of the tester interaction; this helps in determining where the tester was able to move fast, where there was hesitation, and most importantly, where the tester expected to find something and did not.

11. If you are still not clear after going through with a number of testers, use more ! Your product success depends on getting the flow right, and if this means that you need more usability testing done, then so be it.

Usability testing tools

Usability testing is a part of the development life cycle that is pretty critical. It is part of the series of steps (along with user testing and beta testing) that validate whether the product (and the features) are actually usable by the actual end users; feedback from this stage can make a difference between success and failure of the software / website. But such a process can only be useful it is done effectively; if done wrongly, it can prove to be either useless or provide wrong results.
Here is a smattering of tools that can be of help if you are in the business of being involved in usability testing:

1. Usability Test Data Logger tool v5.0 (link to site)
Some features:
# Cross-platform: Datalogger is a PC- or Macintosh-compatible Microsoft Excel file (requires Microsoft Excel to run).
# Customisable: You can enter participant details, task names, task order, pre- and post-test interview questions and include your own satisfaction questionnaire.
# Captures quantitative data: The spreadsheet includes preset task completion scores and includes a built-in timer to record time-on-task.
# Captures qualitative data: Allows data entry of qualitative observations for each participant and each task.
# Provides real-time data analysis: Automatically generates charts illustrating task completion, time-on-task and user satisfaction with the product.

2. Morae Usability Testing for Software and Web Sites (link to site)
From the website:
Morae gives you the tools to:
* Instantly calculate and graph standard usability measurements, so you can focus on understanding results
* Visualize important results in ways that make them more understandable and meaningful
* Present results persuasively and professionally
Morae bundle can be bought for $1495 (link)

3. A website that explains how to use Macromedia Director as a Usability testing tool (link to article)
From the website:
While Director will not eliminate standard development environments or programming languages, it will enhance the prototyping and usability testing experience by allowing developers to gather feedback from prospective clients and users early in product development. Early prototyping will allow developers to identify and fix defects early in development.

4. QUIS: The Questionnaire for User Interaction Satisfaction (link to site)
From the website:
The purpose of the questionnaire is to:
1. guide in the design or redesign of systems,
2. give managers a tool for assessing potential areas of system improvement,
3. provide researchers with a validated instrument for conducting comparative evaluations, and
4. serve as a test instrument in usability labs. Validation studies continue to be run. It was recently shown that mean ratings are virtually the same for paper versus computer versions of the QUIS, but the computer version elicits more and longer open-ended comments.

5. Rational Policy Tester Accessibility Edition (link to site)
From the website:
The Accessibility Edition helps ensure website user accessibility by monitoring for over 170 accessibility checks. It helps determine the site's level of compliance with government standards and displays results in user-friendly dashboards and reports.
* Improves visitor experience by exposing usability issues that may drive visitors away
* Facilitates compliance with federally-regulated guidelines and accessibility best practices
* Enlarges your market opportunity: over 10 percent of the online population has a disability (750 million people worldwide, 55 million Americans)
* Operating systems supported: Windows

6. Serco service (link to site)
They have a service that covers the following stages:
Planning and Strategy
User needs
Defining concepts
Usability evaluation

7. Web accessibility toolbar (link to site)
From website:
The Web Accessibility Toolbar has been developed to aid manual examination of web pages for a variety of aspects of accessibility. It consists of a range of functions that:
* identify components of a web page
* facilitate the use of 3rd party online applications
* simulate user experiences
* provide links to references and additional resources

8. WAVE 4.0 Beta (link to site)
From website:
WAVE is a free web accessibility evaluation tool provided by WebAIM. It is used to aid humans in the web accessibility evaluation process. Rather than providing a complex technical report, WAVE shows the original web page with embedded icons and indicators that reveal the accessibility information within your page.

9. Readability Test (link to site)
The website provides a service that helps in determining how readable a site is.
From the website:
Gunning Fog, Flesch Reading Ease, and Flesch-Kincaid are reading level algorithms that can be helpful in determining how readable your content is. Reading level algorithms only provide a rough guide, as they tend to reward short sentences made up of short words. Whilst they're rough guides, they can give a useful indication as to whether you've pitched your content at the right level for your intended audience.

If you have feedback on the above, or other tools that have been useful for you, please comment.

Usability testing methods

Usability testing can take the form of several methods and techniques (depending on the situation and need):

Cognitive Walkthrough: Performed continually during the development cycle, cognitive workflows try to evaluate the system from a user's thought process that help in decision making (these include the ability to reason as well as other factors such as memory load). This will help in terms of understanding how the system will seem like to a infrequent user. The input for such users could be in the form of paper workflows or through a working prototype. More details can be found at this link.

Focus Groups: Focus groups are actually a method used by marketing companies whereby a group of people from the target segment are brought together in a discussion type of format. A person from the product team acts as the moderator, and prepares a list of issues to be discussed. More details from this site (link).

GOMS: Goals, Operators, Methods, and Selection Rules; a slightly difficult mechanism to evaluate and model human task performance.

Prototyping: Prototyping is a very famous method for displaying a system for internal users and for external usability testers. Prototypes can range from paper models to actual almost working types of prototypes.

Walk-Around Review: This type of study is meant to harness the observations of other people in the company who are not part of the engineering team. The idea is to place paper mockups on the walls of the company along with space for comments so that people can write their comments; in addition, early builds can be provided to company people for them to use and provide comments on. Such a method can provide a lot of useful tips (and have an incidental benefit of providing greater visibility to your project).

Field Observation: Representatives of the product team go to the actual user workplaces and observe how they use the software in their daily lives (example, if a digital imaging application for professional photographers is being developed, then a good way is to have multiple days where people tag along with photographers and observe their usage of the application).

Task Analysis: # An analyst determines the user goals and tasks, then makes recommendations aimed at increasing efficiency and user-friendliness. The objective is to determine how the user actually feels comfortable in using software / sites.

Interviews/Observations: These are one-to-one discussions. Do a thorough discussion with them about what they do, and observe their usage at the same time. This will help in doing a realistic determination about their pattern of usage.

Usability Inspection: reviews a system based on a set of usability guidelines. Experts familiar with issues of usability in design perform the usability inspection.

User Testing: Users (sampled from the final target customer segment) use the final software / site. These can be final users from the client side (if the software is developed for a specific client), or can be a sample set of users. Usability experts study their interactions and figure out what are the paint points and improvements needed.