UX evaluation and testing: detecting problems before implementation.

“taz, die Tageszeitung” (one of Germany’s most influential alternative-left newspapers) continuously improves its native smartphone apps. We came on board to conduct an expert evaluation during the ongoing design process of the iOS and Android apps to prepare these for subsequent user tests. The results of these tests should steer the further product development.

The goal: Learning before failing

The overarching goal of the taz app is to provide a comprehensive user interface and functionality, supporting a positive digital reading experience and ensuring easy content accessibility. Which newspaper wouldn’t dream of achieving this goal?

As the paper had set a medium-term goal to replace the daily printed newspaper with a digital version, it faced the challenge of ensuring maximum user-friendliness in its apps. The focus was on quickly obtaining concrete results based on the designs to advance functionality and to understand how users cope with the navigation within the app. Ultimately the paper wanted to optimise the interaction and visual design before it went into development.

To achieve this, we conducted an expert review using a heuristic evaluation of the designs. We received insights on which we based a usability test using a click-dummy generated from the visual designs.

Project Goals

1. Identify problems before being coded
2. Raise awareness among internal stakeholders for the needs of their users.
3. Introduce user research methodologies within the company in a way that enables them to apply them to future design processes.
   

Our UX solutions for the taz app team

#1 Issue identification based on established criteria using heuristic evaluation (expert review)

Heuristic evaluation is a method in which one or more experts evaluate a system – in our case, the designs of an app – based on a pre-established checklist of criteria. We used DIN EN ISO 9241-10, Principles of Dialogue Design and Nielsen’s 10 Usability Heuristics as the basis of our evaluation. All well-established tools for evaluating the usability of interactive systems. They include fundamental guidelines of usability such as:

• Task appropriateness (appropriate functionality, minimisation of unnecessary interactions)
• Consistency and standards (adaptation to mental user models to the extent known)
• Error prevention (system reacts tolerantly to errors or enables easy error correction by the user)

Performing a heuristic evaluation before usability testing can help identify and fix critical problems before testing with users. We also included problems causing internal disagreements as hypotheses in this first usability test.

Pop-up usability tests at the taz Cooperative Assembly

For an initial usability test of the taz app prototype, taz aimed to test the iOS prototype with existing readers at the Cooperative Assembly 2019 (annual general meeting), as the thematic orientation of the assembly also included presenting the app prototype.

We recruited five participants who were spontaneously approached and asked if they wanted to test the iOS prototype at the assembly. We recruited digital readers as well as paper newspaper readers. We aimed to determine if existing readers could navigate the app successfully from the start page and find the most important information and features.
We asked each participant to perform tasks while observing their interactions. We requested participants use the “thinking aloud” methodology to gain insights into their thought processes. After task completion, we followed up with a series of questions (post-test interview) to capture the participants’ perceptions and opinions. We also used hypothetical questions such as “If you had magical powers, what would you want from a digital newspaper?”

#2 Usability testing with uninitiated participants, but make it continuous to help reduce operational blindness

Next, we recommended a process which allowed continuous usability testing. We suggested that these tests should involve “external” potential readers/users who may not be familiar with the newspaper to gather insights that regular readers may not notice due to their expert status and familiarity with the paper and its concepts, potentially obscuring uncertainties or difficulties.

In the summer of 2020, we conducted numerous remote usability tests with potential readers further expanding our knowledge. We found that some concepts self-evident to loyal fans of the paper and internal staff were difficult to understand for new potential customers and therefore did not appear in the initial tests. The finding demonstrated the importance of including people in tests who belong to the targeted user group but are unfamiliar with the product or service (at least in the B2C scenario).

Testing digital reading in real-life conditions

We also asked where and in which scenarios people used the taz app. We suspected readers use the app in classic “waiting situations” (such as waiting at a bus or train station) and in-far-from-ideal reading conditions (i.e., busy or noisy surroundings with inadequate light, and an unstable surface). We, therefore, recommended “usability stress tests,” which could include testing under unfavourable lighting conditions (i.e., whether functions are visible or if the text is still readable in bright light) or under “shaky conditions,” such as on a subway train. These types of tests evaluate if touch targets can withstand such conditions. Discussing these scenarios can provide ample opportunities for reflection and improvement.

When recruiting participants, don’t forget about inclusivity

We knew from demographic surveys that taz readers and members were often in the middle to later stages of life. As the future of the paper envisioned replacing the daily printed newspaper with a purely digital version, we emphasised the importance of including older users in the tests who usually read their news on paper. We also recommended including readers with visual impairments or tremors.

#3 Enabling the team to run in-house tests with usability testing training

We then provided training on conducting in-house usability tests at taz. The goal of this training was to reduce the perceived barriers associated with user research (“It’s so time-consuming!”), sharpen the awareness for their readers and enable staff to conduct simple tests with (potential) users regularly.

Most challenging

Most fun

Case Study Image Lena Kaiser, taz, die tageszeitung