Inhaltsverzeichnis:
The problem with surveys as a design research tool.
Yes, we know: surveys allow the collection of large amounts of data in a relatively short time. This is why there is conventional wisdom that surveys are easy and cheap, which is not entirely true. A good questionnaire design is hard work and often means investing in pre-tests to ask questions that will lead to meaningful answers. However, these beliefs about surveys are the reason why there is a high tendency to use surveys in design research. But this is a fallacy.
Surveys work well when you want to know about the demographic structure of a population or general opinions (a.k.a. trends) on already-known topics. They are an excellent tool if you want to evaluate hypotheses. But if that is not the goal of your UX Research we suggest not using them in UX Research. Here is why:
There is something called the social-desirability bias, which is the tendency to respond in a way that makes people look better than they are. For example, a survey respondent might report that they engage in more healthy or „better“ behaviours than they actually do. Interviews or — even better — user observations or diary studies allow us to navigate the response bias more efficiently, as you observe the interviewees‘ reactions and/or by asking better follow-up questions.
Additionally, surveys are designed to be standardised. This means everybody gets the same questions and — this is the point — the same options to choose an answer from. Standardization is important for surveys so results can be generalized to a larger population due to comparable answers. To achieve this surveys most often use close-ended questions, providing people with a range of possible answers. Some examples:
Are you satisfied with solution A?
A) Yes
B) No
Which colour do you like most?
A) Red
B) Blue
C) Green
These questions make the results quantifiable: „35% said that they like blue, but 75 % prefer green.“
But these types of questions do not tell us why people do not like blue as much as red. Sure we can implement those questions in our survey and have fun mixing qualitative data with quantitative data when evaluating large amounts of accumulated data. However, we cannot individually address the answers of individual people, ask targeted follow-up questions to that specific answer and dig deeper.
Instead of relying mindlessly on quantitative data generated with surveys, we would recommend sitting down and doing 10 in-person interviews, especially if your research includes the exploration of a new topic or domain. Yes, you will receive fewer answers, but you are guaranteed to receive more insights into topics, which will also lead to a deeper understanding of the why.
Open-ended questions in interviews vs closed-ended questions in surveys
In contrast to closed-ended questions used in surveys, open-ended questions in interviews are intended to give the interviewee space to provide us with more detailed answers. For example:
What do you think of XYZ?
Can you tell me a bit more about XYZ (…)?
Using open-ended questions is especially important if you are unfamiliar with a topic, or you need some background information on that topic and want to dig deeper. Additionally, this will allow us to see *how* people react: non-verbal cues are valuable information – especially during the early stages of topic research when we begin building hypotheses. All this provides valuable information we will not get with a survey.
Unleash your inner journalist
When you listen to someone and let them speak using their own words, you will get new insights which will lead to new possibilities and opportunities that you may not have considered by only asking close-ended questions.
You also get insights into their thinking — their mental models and schemas, and the vocabulary they use to describe things. Journalists often start their conversations with people with simple open-ended questions:
Tell me a little bit about yourself!
How often do you use XY?
Why don’t you use XY?
Tell me more about that experience.
Using open-ended questions can also be a good follow-up to closed-ended questions:
Do you like XYZ?
A) Yes
B) No
C) Don’t know
What exactly don’t you like?
Can you describe it a little bit more?
Why do you think this might not work for you?
By asking open-ended questions you allow your interviewee to expand upon why they think this specific layout/thing/whatever might not work. Depending on those expanded answers you can react, i.e. ask further follow-up questions, which often lead to new insight — a vital benefit you would be missing out on by just presenting a survey to a bazillion people and waiting for their response.
In short, quantity is not always better or more truthful than quality. You will also be surprised by all the new insights you get through in-person interviews and/or observations which is especially important when we conduct design research, especially if this research is the basis for design decisions.
Photo
