Posts tagged " usability testing "

User research biases to be aware of: Demand characteristics

March 14th, 2018 Posted by Analysis & Research No Comment yet

Demand characteristics are one of the so-called experimenter effects in behavioral research. They describe the tendency of interview/usability test participants to give you (the experimenter) what you want based on what the participants think what you might expect from them.

This doesn’t require that you tell them what you want explicitly and give them obvious cues. Participants only need to guess/hypothesize what you want from them based on some subtle cues you send out which are often based on your implicit opinions. So the assumption your participants make regarding what you might want from them is sufficient enough. For example, a general assumption which occurs during usability testing sessions could be that you want them to like the product you’ll be testing.

It’s important because demand characteristics may put the complete validity of your usability test/interview at risk.

Here’s how to weaken/temper the effect:
1) Make clear at the beginning and throughout your test or interview that you want to hear honest feedback. Try to take the fears of non-desireable answers away.

2) Be aware of subtle cues and nonverbal language your participant sends out. Seems the answer forced? Is he/she struggling? Take notes during the test, ask afterward in the debrief interview, eventually play the scenario through again.

3) In addition to an in-person usability test/interview, provide a post-test questionnaire like for example the SUS (System Usability Scale) which is a quantitative tool to measure the perceived usability

4) Last but not least: Demand characteristics may be one of the reasons the designer, who actually designed the system is not always the right person to conduct a usability test or doing debriefing interviews, because he or she may send subtle positive cues about the system he/she designed and so participants might react more positive than they actually feel about the task or question you gave them.
For this reason – even though we also claim research should be not outsourced but should be conducted by the interaction design team itself – it may be worth considering another person to conduct the test or interview – in best case people that are not directly involved in the design or development team. If this is not possible, it’s important to be aware of this bias throughout your test.

 

Comic image by http://www.markstivers.com/wordpress/?p=67

 

Differences in formative and summative evaluations (and why they matter for UX designers)

May 12th, 2017 Posted by Analysis & Research, User Experience No Comment yet

A few weeks ago Steffi spoke at the Berlin Ladies that UX meetup “An evening about user research” about the different types of usability tests. Here is a brief summary of the talk

UX thoughts: Language matters!

April 20th, 2017 Posted by Analysis & Research, User Experience No Comment yet

Have you ever wondered if simply avoiding the term “test” in usability test situations may get you better results? We have and we think replacing the word “test” with the word “evaluation” is an improved description of the situation and will even help us to get less biased and therefore better and more accurate results!

As language determines perception, we think it is better to use the term „evaluation“ when it comes to usability testing situations instead the term „test“ – especially when interacting with our participants, because – simply put – a usability test is nothing else than an evaluation of a product/system with somebody’s help – namely our participants – who are or might be the users of the product/system.

So unlike a heuristic evaluation or cognitive walkthroughs which you will do by analyzing the product by yourself – a usability test is an empirical evaluation of a product/system with the help of people, who use or might use a specific system or product. “Empirical” here meaning: the knowledge or source of the knowledge is acquired by your senses, mostly by observation. In a usability test you will observe (and interview) representative users trying to accomplish certain important tasks as they are using the product.

We prefer the term „usability evaluation (with users)“ over “usability test” or even “user test”, simply because the term “test” is avoided and this little fact may help to reduce stress levels and therefore errors because people will feel less „observed“ and „tested“ during the test situation. “Evaluation” is also a more precise word in this situation because you are not actually testing peoples’ skills while they interact with the product, you are testing/evaluating a product with their help. That is a huge difference, also in the test participants’ perception.  Using the term “test” can result  in so called experimenter effects – like demand characteristics  which describe the tendency of participants to give you (as the experimenter) what you want based on what the participants think/guess what you might expect from them –  here: a test “result”. This may lead to  the participants focusing on how to  solve tasks error-free and to please. Such effects may put the complete validity of your usability test at risk.

In addition, we believe the term „evaluation“ will help people feel much more included because it prevents a „we“ vs „them“ situation, which can help to reduce further stress during the evaluation. We should always keep in mind that our participants help us and cooperate with us in evaluating a product/system and just using slightly different terms may help doing this better.

Image credit: https://flic.kr/p/LJwVMm (CC BY 2.0)

Are representative users important for usability testing?

March 24th, 2017 Posted by Analysis & Research, User Experience No Comment yet

YES, yes and yes – of course: Usability testing is a method in user research. You are evaluating a system or product you design by observing people who think aloud while they use that system/product and solve pre-defined tasks and so help us to spot problems with the system or product.

Recruiting the right participants is a critical, very important and often neglected task when planning usability tests. In best case scenarios, your test participants should also slightly differ within a specified user group. This doesn’t necessarily mean that this has to be totally expensive and is not feasible for smaller companies or organizations.

If you’re going to run usability tests with people who don’t represent your actual users, the problem is, that you might not see incidents or problems that your actual users will have – because of the differences between who you run your test with and the people who will actually be using your system: Think only of expertise – are your actual users technically more sophisticated or do they tend to be relatively new and inexperienced with technology? Or think of cultural backgrounds. All those criteria (plus of course more) will have influence on how people will use the system you’re designing.

Or as userfocus says: „Screen for behaviours, not demographics“

Of course you CAN test with people who doesn’t fit in your criteria and who aren’t representative for the usage of the system you want to test – but then you have to be absolutely aware of that fact, and write it in your report – plus: think about the fact that your findings might or might not apply to the actual people/population who will use your system. You might spot problems or discover ways of thinking and cognitive schemas and mental models that don’t match your actual users.

Even if recruiting the right people may sometimes be a challenging task, it’s absolutely worth the effort, because it gives you a lot more information about how well your system is going to be designed for people who will be actually using it.

Links:
Recruiting Better Research Participants
Writing the perfect participant screener
Recruiting Usability Test Participants

Image source: https://flic.kr/p/QHpkCL

We love to write an speak about UX topics.

We have had many nation wide speaker appearances on many different topics. We have spoken at TEDx, UIG spring conference, Usability Congress, World Usability Day, webmondays and the up.front web design community in Berlin. We speak mostly about topics such as user experience and workflows. Have a date? Need speakers? Get in touch and let us know We might be interested in researching a topic