Ethical aspects and principles in usability testing

As part of the psychology degree, students must complete a certain amount of what is called subject hours for admission to the thesis. This is about 30 hours in total and this has the function of learning about research also from the perspective of the participants before planning and conducting psychological research yourself. This is time-consuming, but really valuable as you become more aware of some pitfalls in the communication and design of the investigation. In addition, you learn how you feel as a “test subject”.

Knowing the feelings and concerns of the test subjects is not insignificant, because investigations such as usability tests are also accompanied by a certain psychological pressure on the participants, which in turn can also have a negative impact on the test result.

Possible fears of test persons

• “Stage fright” or performance anxiety. This pressure to perform can in turn affect self-confidence as well as self-efficacy and result in a kind of self-fulfilling prophecy.
• Subjects may feel “stupid” if they cannot solve tasks or find things in the system. They blame it on themselves and not the system, it feels like an IQ test to them.
• They compare themselves inwardly with other users according to the motto: “Others are certainly not so stupid…”.

As moderators of a test, we should therefore always try to put ourselves in the participants’ shoes: As test subjects, they sit in the “spotlight”, i.e. they are the focus of observation and are asked to perform tasks with a product that is unknown to them and possibly not optimally usable in front of strangers. It is quite natural to be a bit nervous in the situation if necessary. Thoughts then arise among participants such as, “Am I doing this right?” or, “Do these people think I’m stupid for not understanding?”

This pressure is real. Jared Spool, an American usability consultant, told a story about seeing a test subject cry during a usability test. This situation came about through an accumulation of careless behavior on the part of the moderators and test administrators: the original participant did not show up on the day of the test and then an employee who was just completing her first day of work was simply brought in as a quick replacement. The team thought it was a good idea to take her because, unlike other employees, she knew little about the product. There were also a number of observers in the observation room who were not informed about how to behave during a test, including the subject’s boss. And last but not least, no pilot test was conducted, so the team did not know that the first task participants were asked to complete was in fact completely outdated, incorrectly formulated, and thus impossible to perform. While the woman was desperately trying to perform this first task, everyone but her quickly realized that the task was outdated and unfeasible, and they started laughing at their own stupidity. Unfortunately, the user thought that they were laughing at her, and she then started to cry.

This is exactly a situation we want to avoid.

How to respond to these fears

We can counter such fears by emphasizing again and again that we are not testing the test person, but the system. This “I’m too stupid to use the computer” is unfortunately based on internalized thinking. Self-blaming is a problem here. Many users of computer systems still think that if something doesn’t work, it’s their fault, that they can’t handle the software “properly” and blame themselves rather than the system.

Here is a sentence that we like to repeat like a prayer mill during tests in this or a modified form:
“We’re not testing you, we’re testing the system.” and: “Any problems you may have are the fault of the system. And it is precisely to detect these problems that we need your help to make the system better and easy to use. They help assess/evaluate the system as we are too operationally blind to do so.”

Wording/framing – that is, how you call the examination itself – also plays a role here. For example, when we talk about “user testing”, it implies that we want to test the user. But this is not the case. We test the usability of the system, so we should really rather talk about “usability testing / verification” – especially in front of the users*.

In addition, an appreciative and respectful approach plays a major role. Sounds self-evident, but it is important to remember it again and again, especially when things are hectic.

Therefore, below are some basic and simple ways to reduce psychological stress in test subjects and conduct a usability test according to ethical guidelines.

11 ethical principles for a usability test

1. Language. We advocate avoiding the term “user test”. We do not test the user, we test a system with the help of the users.
2. Creating a friendly atmosphere. The atmosphere should be relaxed and kept free of distractions, interruptions such as interposed questions from others. Drinks and snacks are always welcome during in-person tests.
3. Let the test person arrive. Greetings and some small talk are important to “arrive” and relax the participants. Therefore, you should plan for this in terms of time.
4. Informed Consent. Informed consent must be given to subjects several days prior to testing and signed and given to the testing team no later than the day of testing. This consent should inform about the purpose of the examination and the rough procedure of the test. In addition, there should be an understandable and detailed description of how the results will be used and how it will be ensured that the participant’s data will be kept confidential. Participants should be given comprehensive and comprehensible information about the use of the data. This means that there should be information about who can access the data, where and how long the data is stored (DGSVO) and how long the participants can request the deletion of their data or its non-use – because once data has been anonymized, this is usually difficult. The voluntary nature of participation and the fact that subjects can stop the test at any time should also be mentioned. Furthermore, contact options should be offered for any questions.
5. Repeat the reconnaissance immediately before the test. Before the test begins, it is best to briefly go through all the points that can also be found in the informed consent with the test subjects and make sure that everything has been understood and that there are no more open questions in this regard. It is advisable to go into the anonymization of the data again and to explain to the test persons that e.g. quotations that appear in reports or presentations cannot be traced back to the respective persons. It should also be pointed out once again that the test subjects perform the test absolutely voluntarily and have the right to cancel the test at any time. Thus, the subjects are in control if they start to feel uncomfortable for whatever reason, for example.
6. Warm-up Questions/Pre-Test Interview. Before the actual test, where we start working on the tasks, it is good if we first start with thematically appropriate general warm-up questions. These questions are usually of interest anyway and of a rather open nature, such as questions about previous use of the product/service in question, questions about frequency of use, questions about specific experiences from the last time it was used, etc. This conversation (pre-test interview) helps the test persons to “wind down” a bit, to relax a bit and to familiarize themselves with the environment and the moderators.
7. Control non-verbal cues and facial expressions. While we are sitting with the subjects, we should never be impatient and also have implicit, unconscious behavior patterns on the screen – such as our facial expressions. That’s easier said than done, because a lot of it just happens unconsciously. Therefore, it is a good idea to include this in the moderation script or have the topic on the screen. A raised eyebrow at the wrong moment, breathing too loudly or nervous finger tapping can be completely misinterpreted and referred to oneself or the performance on the part of the test person – with consequences for the test.
8. Respect the time of the participants. It is a sign of courtesy if we respect the time of our participants: For us, this means being well prepared and not overstaying our welcome. This is also where the special role of the so-called pilot test comes into play: the pilot test serves to test our test :)- the test of the test, so to speak. The pilot is very helpful to find out if our tasks cause confusion or ambiguity or contain errors and if the test conditions run “smooth” overall. The pilot test runs exactly like a real test, including subject, consent, script etc…and takes place a few days before the first real test. Thus, we still have enough time to make corrections or to formulate tasks in a more understandable way.
9. After the test: Have data use confirmed again. At the end of the test session, the facilitators should again ensure/ask whether the respondent agrees to the use of the data. It may be that the test persons change their minds during participation. A later withdrawal from this is, as already mentioned, mostly difficult or impossible due to the anonymization of the data sets. This should be clearly communicated.
10. For remote tests: offer technique setups the day before. For moderated remote test sessions it makes sense to offer technique setups e.g. one day before. Here you have the opportunity to familiarize yourself with the technology, such as screensharing, with the help of a test page (not the test product itself!). A positive side effect is that the test persons get to know the test team, which reduces the excitement on the day of the test and any fear of contact.
11. Transparent communication about additional observers. Sometimes managers or developers also want to and should (!) be able to watch the tests. In this way, members of the team experience first-hand the problems that users experience when operating the system in question, and this can be important in establishing an understanding of UX in companies and, in turn, in developing the UX maturity level of a company. On the other hand, it can become problematic when too many The test is not observed by all persons at the same time, because this adds another artificial dimension to the test by making users feel even more “observed” and “tested”. Thus, performance pressure and “performance anxiety” may increase, which in turn could skew test results. So rather less observers and rather rotate the team per test or test series instead of putting the complete team in the observation room.

If there are other people observing the test in addition to the moderators….

• … should be pointed out transparently in any case and it makes sense that these persons are briefly present during the greeting. It should always be communicated openly and honestly who is watching and why.
• ….it is essential that we brief them well on how to behave before, during and after the test. The point is that there should be no interruptions by intermediate questions or remarks or, as mentioned above, unconscious gestures, facial expressions or noises are audible (snorting, yawning, laughing, finger tapping). We like to tell the above-mentioned story of Jared Spool, because it impressively demonstrates how quickly things can unintentionally go wrong.
• …it is helpful to hand out some sticky notes to the observers to avoid interruptions in the form of questions from the observers. This is a way to invalidate the feeling of the “pressing” question that needs to be asked right now. Here, observers have the opportunity to write down their questions or thoughts directly without interrupting the test. They can then hand them over to us moderators at the end of the test. We briefly review these and then pass these questions on to the respondent if necessary. This should also be scheduled. Through this approach, the facilitators remain the main contact person for the test person and there is no unrest and confusion.

Conclusion: Do “subject hours”!

There is certainly a lot more to say, but these basic points should suffice for now. We can only recommend to all designers, product managers, and developers that, like psychology students, they also complete “test subject hours” and more often slip into the role of the test subjects themselves and participate in a usability test from this perspective. This way you get a feel for how it all feels and can bring that experience into designing your own test.

Illustration: Question illustrations by Storyset