Having a deep understanding of our users’ needs is crucial for building successful digital services. Here Vanessa, one of our User Researchers, rounds up some of the most common user research myths she's come across. If there are any you think we've missed, let us know in the comments.
1. User testing is all you need to do
It’s true that user testing is an incredibly important part of what we do, and I can’t really think of a project that shouldn’t include it in some form, but it’s certainly not all we should be doing.
What user testing does:
- It allows us to test specific elements of products in a structured, scientific way
- It highlights usability or cognitive issues with a product and gives us qualitative insight into why there is a problem.
What user testing does not do:
- It doesn’t tell us how big a problem is and how many people are experiencing it, which can help us to decide how important it is to fix (we can get this from website statistics, surveys or feedback).
- It doesn’t show how users behave across the entire system outside of the task we are testing (we can get this from statistics).
- It doesn’t give us detailed insight into our users opinions and attitudes (we can get this from interviews and focus groups).
- It limits us to watching how users behave in an unnatural environment with a facilitator sitting with them, not in their own homes (we can unpick this over time through contextual testing, statistical analysis and feedback review).
To get a full and complete picture of our users’ experience, we need to be monitoring all of these things, all of the time.
For the SFA’s apprenticeships exemplar, the user researcher Jess Gough is measuring statistic of all kinds, monitoring feedback and survey results, carrying out focus groups and interviews as well as doing weekly user testing.
2. You are one of your users
It’s great when you’re part of designing a system that you want to use yourself, but it’s all too easy to make the mistake of thinking that this means you can design it from your own feature wish-list. For example, if you’re designing a product aimed at parents and you’ve got kids of your own, then surely you know what other parents will want; it’s the same as you’d want for yourself.
It’s true that an affinity with your audience will help you to empathise with them, and this is a valuable emotion in user centred design, but it doesn’t mean you’re one of them.
If you are involved in the design team, part of the policy team or even part of the organisation that runs the service, then you are not a typical user. The knowledge that you’ve gained through being close to the service or product design will affect your usage of it and your cognitive map of how it works. Even your affiliation to an organisation and its culture means that you are part of a social group that the rest of your users aren’t part of. This means that you can’t make design decisions on your users’ behalf.
We tested this out for ourselves in 2011 when we were asked to test the newly developed National Careers Service on 25 internal employees. Even though the employees we used were so far removed from the service that some hadn’t even heard of it, their knowledge of the education sector and their shared public sector career history meant that even though we got a consistent set of results from them, it was completely different to the results we got from external users.
It’s great to be passionate about your users, and important to try and understand them as much as you can, but never assume that this means you can speak for them, or that you’re one of them. Testing out your assumptions with real users will flush out any biases and make sure your designs work for your audience.
3. What works for Amazon will work for us (or Google, or Apple)
Ah, the heady days of Discovery! How often have we started a project imagining that we can be pioneers of innovation, that we’ll have Amazon style personalisation, Trip Adviser’s social interaction, Google’s simplicity and Apple’s killer interface?
It’s good to aim high, and design should never be too tightly constrained before it’s even begun, but there are two reasons why we should be a little wary about trying to mimic the big boys too much.
a) Our environment is different from theirs
We are the public sector. We don’t have billions of pounds to spend on technology, design and research. We don’t have the singular drive of a commercial organisation behind us. It’s just not practical to expect that we can produce something as complex and expensive as Amazon. It doesn’t mean that we can’t be brilliant at what we do, but it’s better to do a great job of one thing than to try and be all things to all people.
b) There’s a danger that we’ll use statements like this to excuse us from doing user research
If Amazon have researched it and it works, why bother doing it all over again when we can just copy their idea? Again, we have to look at our goals and our audience. Are our goals the same as Amazon’s? Does our audience fit the same description? Are their user needs the same? The answer will almost certainly be ‘not entirely’. By all means, we should use their best practice as a starting point – that makes sense. But we must always test it with real users. The results may well be surprising.
4. User research is expensive
Proper consideration should always be given to user research when planning a project, and that includes time and an appropriate budget, but that doesn’t mean user research has to be expensive. There are plenty of ways to get in touch with your users that cost very little, or even nothing at all. For a small agile project on the National Careers Service website last year, we only had a very small amount of money for researching, just enough to hire a lab for a few days. We wanted to make sure we got the most out of the testing days, so we did as much as we could beforehand to make sure that by the time we went into the lab, we were testing a product that we were confident about. Here’s some of the things we did:
Early concept testing
We located a spot in the nearest city where we hoped to bump into a lot of our target audience. We stood (in the rain) for half a day and offered passers by a chocolate in return for answering three simple questions about our prototype. That helped us to determine whether our early thoughts were worth pursuing.
Pop up user testing
We used Silverback on a laptop to test our prototype in a local library. Armed with a plate of cakes and a homemade poster, we set up camp for a day and ran through a twenty minute test with library goers. It was hard going to find users who matched our audience, but the information helped us to iron out usability problems.
Online card sorting
Part of our audience group were aged 13-16. Schools are often willing to help with testing, but to get a properly segmented, diverse group of children with all the appropriate security clearances in place, you need a recruitment agency. We did do some formal testing with this age group later on in the project, but before we got to that stage we needed to do some quick and dirty testing on our architecture. We used a free online card sort and sent our own kids and others we knew out to recruit their friends. We gave each of our kids their own unique link to the card sort so we could reward them with 10p for each child they got to participate. This had the added bonus that since each child only recruited within their own peer group, we had a way of breaking the respondents down by school year.
Guerrilla testing
Part way through the project, we made some changes to the images on the page that we needed to test urgently. I only needed an A/B type first impression, so I printed out different versions of the page with each set of images and took these with me on a long train journey. During the journey, I wandered the train and got around 20 people to contribute, as well as another half a dozen in a local café while I waited for my meeting to start. It takes a bit of nerve to approach people like this, but if you’re friendly, most people don’t mind.
There’s lots of other low cost ways to interact with your audience, and they can be a great way to supplement more robust research. Whilst it’s important to budget for research appropriately, lack of budget should never be a reason not to test; any research is better than no research at all. If you’ve had success with low cost research, add it to the comments below.
5. User testing only happens in a lab
As we’ve already seen in Myth No. 4, there are lots of things you can do to quickly and cheaply to get basic answers from your users. Lab time is really useful, and should be factored into more or less every project, but it can be expensive, and depending on where you are in the country, it can be hard to find a lab locally.
Guerrilla testing simply means getting out of the office with a prototype, drawing or concept, grabbing fairly random people off the street and testing with them. It should never be a complete substitute for lab-based testing; it’s less scientific, less controlled and less detailed, but it’s a great way of maximising your lab time. If you use guerrilla testing to iron out basic usability problems before you get to the lab, you can use your lab time for the really important tasks.
Testing can also happen in a more controlled way outside the lab by taking a laptop out to pre-recruited participants, or remotely. On the Employer Routed Funding project, we’re testing with employers. It’s difficult to get them to take time out of their busy schedules to go to a lab, but they’re much more willing to participate if we take the testing to them. The added benefit of this is that they’re more relaxed in their own environment, and we get more realistic results because of it.
Any we've missed? Add them to the comments.
Among other projects, Vanessa works as a User Researcher on our Find an Apprenticeship exemplar. Read more blog posts by the exemplar team: