OD FAQs | How many people should do a usability test with?

Many new clients ask me, “How many people should we do a usability test with?” I usually reply, “It depends on the type of testing you are doing.”

Types of usability tests

For Objective Digital there are three different types of usability test. The first two can be undertaken with or without eye tracking. However, the third really only works with eye tracking.

1. Improving usability and design

The most common test we perform is to improve usability and design. Participants are asked to complete a set of 6 (approx) tasks on a given software or website.  The objective is to identify:

  • Where people are having issues
  • Why they are having these issues, and
  • What to do to fix them.

How many people to test?

People who have similar levels of knowledge tend to make similar mistakes when using an application. When trying to complete a task, someone’s behaviour is constrained by the interface they are using.

In usability this means that after only a few participants tested, we have seen most of the errors that people are going to make. In this type of qualitative test, a rule of thumb for us is, 6 people per project.  We generally find we can get through 6 participants in a day of testing.

We often perform this type of testing iteratively for a new website development. This involves testing clickable JPEGS and HTML prototypes with limited functionality to uncover issues before the full development of the site.  Over the development phase we may test two or three times, with 6 (different) participants.

We  would do 12 people if the application has varied functionality that requires a different set of knowledge and skills to be able to complete tasks. For example, with Internet Banking a business customer should be able to easily complete the tasks once they log into business banking.  During the test we could also get them to log into the Personal Banking section and have them do tasks there too.

In comparison, a person who doesn’t work in business would have all sorts of trouble completing the Business Banking tasks and none at all with the personal tasks. 

In a case like this I would recommend 12 people be tested. Six business banking and 6 personal.

Sometimes Government departments and large corporates have many different users who use significantly different functionality. This requires testing with more people.

2. Measuring and benchmarking usability

Larger corporates with many and varied content managers and site owners like to track their site to monitor its performance on a frequent basis. Say twice a year. 

How many people to test?

In this type of test, where metrics such as time to task completion, number of clicks, error rates and other experiential measures are captured we recommend testing with at least 12 – 18 people. 

With this number, you uncover between 98 and 99 per cent of common errors, which are defined as those errors likely to be made by over two thirds of users. For details of these figures,  see this Human Factors blog post.

For measures such as task completion, or time to complete  task, there is a smaller margin of error with the increased number of people tested.  These measures make it easy to compare performance over time.

3. Call to action testing or design comparison

We are finding that many media companies, or organisations with lots of advertising, are keen to test the effectiveness of advertising, or other calls to action, on their websites. Other companies want to know how to make the pages on their sites work harder, to get that extra percentage of conversions. 

In this type of testing we are looking at how people digest the page before they have even started interaction.  What immediately stands out for them, where is their attention drawn? Clients like to capture measurements like:

  • How many people looked at an ad or call to action?
  • How long before they looked at an ad?
  • How long did they look at the ad?

This can only be done accurately using eye tracking, which is described on our website.

 How many people to test?

For this type of test you need at least 30 people to identify the common behaviours between people. This slideshow tells it all:

http://www.slideshare.net/jamesbreeze/why-you-need-50-people-in-eye-tracking-…

 

You can find a list of other FAQs on usability on our website. 

Advertisements
This entry was posted in Uncategorized and tagged , , by Objective Experience. Bookmark the permalink.

About Objective Experience

Objective Experience is your partner for customer experience strategy, research, design and usability testing. Our team of passionate Customer Experience Consultants uncover insights from your customers to optimize customer journeys across all digital and physical channels – mobile apps, websites, systems, shopper and retail. We research on how customers interact with products, services and environments through interviews, surveys, focus groups, desk research, eye tracking and usability testing. Actionable recommendations are developed from our in-depth observations and testing. Our partners and clients come from a wide range of businesses and industries - from large financial institutions and government organisations to technical consultancies, Universities and not-for-profit businesses. Objective Digital Holdings Pty Ltd has offices in Sydney (Objective Experience Sydney) and Singapore (Objective Experience Singapore). Eye Tracking, CX, UX, Usability Testing, Shopper Research & Design Thinking across Australasia and South East Asia.

One thought on “OD FAQs | How many people should do a usability test with?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s