Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Two – step Experiment Design for Participative Design Thinking

1 MdynamiX AG, 87734 Benningen, Germany ↗
2 Institute for Driving Assistance Systems and Connected Mobility, 87734 Benningen, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1006670

Abstract

User-centric development of innovative products, functions, or services can be challenging. Usually, users understand the purpose or functionality of the system they use and are therefore in a competent decision-making position when asked to rate the usability of a product or voice suggestions for improvement. However, some products are unfamiliar to the users for whom these products are developed. To test these types of products efficiently, we developed an experiment design composed of well-studied methods like Usability Testing and Design Thinking workshops. The goal is to generate improvements and development tasks collaboratively with the end users. Furthermore, we present experiences of applying this approach in a study with 22 participants on an innovative safety-relevant ADAS function in collaboration with an Original Equipment Manufacturer (OEM).

INTRODUCTION

Developers of rapidly evolving technologies are facing a challenge: While technology is constantly evolving, users’ knowledge and understanding of technology are not necessarily evolving accordingly. When building technology for their prospective users, not only do developers need to develop systems that can communicate with users about concepts familiar to them, but particularly concepts of modern technology that are unfamiliar to users. This challenge affects HMI development, where poor communication might impair UX, usability, and thereby acceptance and usage. Therefore, the aim of this work is exploring a way to understand how an innovative system should communicate with users to accommodate their needs and expectations.

Design Thinking is a broadly used method of User Centered product development. It is attributed to reflections on design raised in the mid- 20th century (Johansson-Sköldberg et al., 2013). During these developments IDEO became known for coining the term Design Thinking to describe a methodology for human centered development (Camacho, 2016). Design Thinking is typically conducted in groups of experts of different fields (Schallmo et al., 2018). In this study, a participatory approach to Design

© 2025. Published by AHFE Open Access. All rights reserved. 84

Two-Step Experiment Design for Participative Design Thinking 85

Thinking is chosen, where users are involved in creating early-stage designs for the product (Ehn, 2008).

Before finding a solution to our problem, another technique is employed traditionally in fields like automotive development: The subjective evaluation. Subjective evaluation is performed both by experts and by selected end-customers with evaluation times ranging from short tests to long-term observational studies. In some cases, this technique is employed in combination with benchmarking approaches, where a competitor’s product, relevant for a selected reason, is chosen to be compared (Sharipov and Zaynutdinova, 2023). In a software-development context this can be compared to Usability testing, albeit lacking the formalizations provided by Human Factor’s community.

However, both approaches (Design thinking, Usability Testing) on their own fail to help find solutions for unfamiliar products or features: Usability Testing lacks an environment in which new, useful ideas can be brought forward while a classic Design Thinking session is difficult to conduct when the participants lack both familiarity with the product or understanding of its purpose.

To mitigate the shortcomings of these approaches, we propose an experiment design with two main phases: A Usability Testing phase in which participants can voice their opinion and criticism without bias from any further explanation. While the designer profits from the feedback given, the customer is being familiarized with the product’s use. The experience is then contextualized in a second phase to then allow the participant to come up with new ideas on how to intuitively improve the product’s usability or achieve a more desirable User Experience. Results of these two phases can then be analyzed to suggest improvements for the functionality of the product und its respective HMI.

METHOD

Overview The experiment is composed of two stages: The Usability Testing and the Design Thinking stage. In the usability stage, the user can start becoming familiar with the product. It starts with a briefing, in which the purpose of the function is explained (not the HMI). After allowing the user to experiment with the product independently, we then ask the user to perform certain tasks. The block ends with questionnaires to rate the functions tested.

After a short break, users can then apply the experience gained to make active suggestions. For this, using storytelling, the product’s use is re-contextualized to a real-world, practical use-case that goes beyond the potentially sterile lab conditions and helps users imagine an actual use. With this, employing journey mapping, users reflect on their experiences and start forming opinions on how an ideal product should function. Then, they develop their own concepts while being guided by the workshop facilitator and pre-prepared content.

Figure 1: Overview on experiment steps.
Original PDF Figure 1:

Planning This study type is complex, because it has several stages that should be carefully planned and tested before execution. Firstly, product developers and the respective experiment conductors should convene to ensure two key points:

  • Those conducting the experiment fully understand the product and its purpose. After all, an innovative product might also be unfamiliar or surprising to the evaluators.
  • Developers who are unfamiliar, in turn, with methods like Design Thinking understand the method sufficiently enough to define goals of what they want to learn about the user and their interaction with the product.

In the example featured, this was executed in the form of a kick-off workshop. Mutual goals were first collected by allowing every participant to voice their individual wishes for the outcome of the study. This pre-study study ensures designers and evaluators develop a mutual strategy for the experiment. Next to goals, other important decisions on the experiment can be jointly addressed, like experiment duration or participant number. In the case of ADAS this also entails defining certain maneuvers or instructions or selecting a relevant benchmarking vehicle. Writing down goals and procedures ensures that outcome is comprehensive to all stakeholders.

Experiment designers can then proceed to plan the experiment, starting with defining roles. We recommend at least two active experiment leads: One for Usability Testing, one for design thinking. This recommendation arises from two ideas: Evaluating with participants can be challenging and separating tasks where possible can reduce probabilities of error and give everyone a clear goal. The second reason is that for Design Thinking, the facilitator should be as uninfluenced as possible. Not observing the participant interacting with the product reduces the risk of unwanted influence on the participants.

Preparation Given a clearly set goal, and a well-designed plan, the experiment can then be prepared. This entails foundational preparation like recruiting a

Two-Step Experiment Design for Participative Design Thinking 87

participant pool fulfilling the requirements set by stakeholders (age, gender distribution etc.), preparing two distinct rooms or environments catering to the requirements set by both testing stages, and the availability of human factors experts for experiment conduction.

Preparation for Usability Testing is strongly dependent on the product to be evaluated. In the example of our ADAS system, preparation included organizing a test-lap to be driven on, maneuver-relevant test props (e.g. obstacles) and the mounting of measurement technology on the vehicles to be tested. Furthermore, based on the relevant context of the use of the actual function, one or more tasks or scenarios must be defined. All preparation for Usability Testing should end in a pre-test in which the study itself is evaluated with respect to understandability of tasks, practicality of actions planned and usefulness of responses.

Design Thinking, on the other hand, involves first preparing an environment suitable for the task. For this, a meeting room was transformed into a “Creative Lab” (see Fig. 2) using guidelines for collaborative workshops developed in storytelling (Litwinow, 2019). This involves dimming outdoor lights from windows with curtains and using sources of warm yellow light, relaxing music, availability of food and drinks, and comfortable seating. The goal is creating a surrounding that doesn’t intimidate participants from voicing their opinion or feeling observed.

Figure 2: Design thinking environment.
Original PDF Figure 2:

A further element to prepare for the room is a mood board. When faced with the task of creating new solutions, participants perform better at ideation when provided with examples (Kulkarni et al., 2013). These examples can be combined with other visual imagery to inspire participants in their thoughts in respect to finding their ideal solution.

Storytelling should follow the context of use and ideally embed the usability tasks fluidly into the story to help participants in better assessing the experience. To ensure consistency, we then let a voice actor perform a recording of the story.

Ultimately, material needs to be prepared to facilitate the generation of content. For a collaborative workshop concept, Litwinow (2019) proposes a set of material to be used for facilitating ideation: A canvas with a transparent layer that can be drawn on, a set of symbols and signs that can be used

and the presence of a mood board. For this design, a similar approach is pursued in digital, using a virtual collaboration program in combination with a Smartboard. Using this setup gives the benefits of digitalized workshop recording and flexibility of analogue approaches like drawing an idea.

The board (see Fig. 3) is set up like a worksheet. It separates the entire Design Thinking process into slides that contain pre-made content to facilitate generation of opinions, improvements, and new ideas. Each slide represents an idea to be explored and affords for generous space for the participants to voice concepts. The sequence of slides prepared in turn follow the Design Thinking stages.

It begins with a Journey Map (Chasanidou et al., 2015) that separates the tasks performed in the Usability evaluation into meaningful episodes. The table prepared then affords to place digital sticky notes or drawn explanations for each stage across a set of criteria (Think, Perceive, Improve; Inspired by Gibbons, 2018). In the case of our innovative driving function, this is composed of the following stages:

Figure 3: Virtual board for journey mapping (redacted for intellectual property protection).
Original PDF Figure 3:
  • Free driving: This first field’s purpose is capturing the baseline of the participant’s experience. What did participants observe before the function is active? Was the condition of being in an experiment stressful to the participant? Was something non-function related influencing the participant’s decision on the function (e.g. seating comfort, perceived aesthetic of interior design).
  • Warning sequence: Before activating the safety-relevant feature, a sequence of warnings is emitted to capture the driver’s attention. For each of these steps, an own episode is defined in the journey map. The goal of each episode in this case is to reconstruct to which extend warnings are

Two-Step Experiment Design for Participative Design Thinking 89

perceivable while performing the driving task and assess how intuitive and actionable they are.

  • Function execution: When the safety-relevant function is active, users are in a particularly stressful situation. While the usability was tested in the previous experiment stage, observations and mental models can be reconstructed to understand how users perceived the execution.
  • Perception upon completion: Like asking about prior experience, this stage is meaningful to understand how the safety function impacts driver behavior. Does the function increase stress and alertness because of perceived unpredictability, or does it increase perceived safety?

The second part of material is used for the creative ideation stage. In this stage, participants can come up with new solutions by building on the acquired understanding of product context and use. As the safety- critical innovative ADAS function requires a warning cascade, ideation not only involves designing isolated messages for each stage, but a composed sequence.

For this, a sequence ideation board (see Fig. 4) is created in which blanks are left for all phases of warning and function execution. Presenting the sequence in this way avoids that participants come up with short-term communication approaches. Instead, this enables participants to design a proper strategy with warning salience increasing accordingly to the gravity of the situation. The sheet includes both an image and text description for each stage (to clarify acuteness), a blank space for a description of the desired functional or communicational behavior, and a second blank space to provide for a reason. Separating these two ensures that even when the concrete approach chosen by the participant, potentially unrealizable technologically, can still be built on later by experts to come up with a solution to cater to the intended purpose.

Figure 4: Strategy ideation worksheet (redacted for intellectual property protection).
Original PDF Figure 4:

Based on the sequential communication strategy developed, participants can then detail out the behavior for each of the stages. To facilitate this, an image of the entire situation from the driver’s perspective is displayed for each stage. Ideas can then be detailed either by drawing or by providing sticky notes that further describe the intended functional behavior or communicational content.

RESULTS

The applied study was performed with 22 participants, made up of 11 sessions with paired participants. The choice of pairing participants was made to allow for a randomized cross-over study procedure in the usability section of the experiment. Each paired time slot was scheduled with an experiment time of two hours.

In total, five experiment leads were employed: An engineer for parallel- running objective measurements, a usability tester, an experiment lead for Design Thinking, a Design Thinking Facilitator, and an organizer to lead participants through the stages of experiment. It proved helpful to the execution using color-coded paper wristbands with participant IDs and randomization group.

The study’s outcomes are twofold. The usability test reveals concrete feedback on the usability of the concrete system employed, while Design Thinking provides a direction for further improvement.

The Journey Map reveals multiple sets of information to improve the product. The PERCEIVE field, where participants describe their emotional state, reveals how participants’ self-reported experience changed on average while interacting with the product. It can highlight sections of the experience where users faced most issues while using the product and provide clustered adjectives to specify them quantitatively beyond a linear “good and bad”. The THINK and IMPROVE answers can both be clustered by category to gain a better understanding of recurring statements. The results of this experiment are summarized in the Journey Map (see Fig. 5).

Coming to the section of ideation, results of desired concepts are summarized in the sequence ideation. For each stage, a set of desired concepts of communication (or of behavior) is clustered and sorted by frequency. The intended goals are also clustered then to be used to sort the desired system behavior. For each stage then, one dashboard (egocentric perspective) is presented to summarize the desired, detailed behaviors that participants would like the system to exhibit.

The study about the ADAS system revealed a set of conclusions. In one case, a clearly preferred system emerged from Design Thinking. The intervention was derived from a system existing in a benchmark vehicle. Furthermore, sequence ideation revealed that participants prefer acoustic and haptic signals to optic signals when in an emergency. This could be supported by Journey Mapping results, where participants reported frequently (13 out of 22 mentions) not perceiving graphic warnings when focusing on safety- critical surroundings in the environment. Participants noted being satisfied with participating, mentioning “having fun” and acknowledging positively that their opinion was considered.

Two-Step Experiment Design for Participative Design Thinking 91

Figure 5: Summarized journey map (results translated and redacted for intellectual property protection).
Original PDF Figure 5:

DISCUSSION AND OUTLOOK To what extent can developers use these results to achieve the developer’s goals: Improving the User Experience by finding better ways the system interacts with the user. In this context, the results from Usability Testing can be seen as a general indicator for necessity to change: Bad usability results are compelling evidence for developers to change the way the system interacts with the user.

However, improvement is difficult to achieve when it is not known what to change and how to change it. The Journey Map results can further show in detail which part of the experience was bad for the user, what they perceived, and suggestions on how to improve. Developers can use these results to correct the User Experience by adjusting system behavior in key moments.

Moreover, the Ideation results create an artifact that can be a valuable guideline to improve the entire system Usability. Together with the other results it draws a clear picture of how users would like the system to communicate and furthermore stating the underlying motives. The results can therefore be used, depending on the Usability results, to either optimize only small parts of the experience when overall feedback is good, up to redesigning the entire experience when poor results make it necessary.

The rather complex and time-intensive preparation required to design, conduct, and interpret a study of this kind makes clear that this experiment design is not practical for fast iterations in a regular fashion. Rather, it can be seen as a tool to come up with a good understanding of desired user experience in early stages of development, especially when the product is a technological novelty and therefore interaction with the user needs to be understood from ground up. It can also be used to initiate a major rehearsal of User Experience, when incremental improvements don’t prove successful in achieving a good UX.

REFERENCES

Chasanidou D., Gasparini A., and Lee E. (2015). Design Thinking Methods and Tools for Innovation. In: Design, User Experience, and Usability: Design Discourse. 4th International Conference, DUXU 2015, Held as Part of HCI International 2015, Los Angeles, CA, USA, August 2–7, 2015, Proceedings, Part I. Springer International Publishing. pp. 12–23. Camacho, M. (2016) ‘David Kelley: From Design to Design Thinking at Stanford

and IDEO’, She Ji: The Journal of Design, Economics, and Innovation, 2(1), pp. 88–101. Ehn, P. (2008) ‘Participation in design things’, in PDC ‘08: Proceedings of the Tenth Anniversary Conference on Participatory Design 2008, pp. 92–101. Gibbons, S. (2018) ‘Journey Mapping 101’, Nielsen Norman Group. Available

at: https://www.nngroup.com/articles/journey-mapping-101/ (Accessed: 13 February 2025). Johansson-Sköldberg, U., Woodilla, J. and Çetinkaya, M. (2013) ‘Design Thinking: Past, Present and Possible Futures’, Creativity and Innovation Management, 22, pp. 121–146. Kulkarni, C., Dow, S. and Klemmer, S. (2013) ‘Early and Repeated Exposure to

Examples Improves Creative Work’, in Design Thinking Research – Building Innovative Eco-Systems. Springer International Publishing, pp. 49–62. Litwinow, V. (2019) Entwicklung eines kundenorientierten Produktentste- hungsprozesses basierend auf der Design Thinking Methode am Beispiel Head-Up-Display. Bachelor’s Thesis. Hochschule für angewandte Wissenschaften Kempten. Schallmo, D., Williams, C. and Lang, K. (2018) ‘An Integrated Design Thinking Approach Literature Review, Basic Principles and Roadmap for Design Thinking’, in Proceedings of the XXIX ISPIM Innovation Conference “Innovation, The Name of The Game”. Sharipov, K. and Zaynutdinova, U. (2023) ‘Prospects for Improving the Benchmarking Activity of Automotive Enterprises in Uzbekistan’, in Ilin, I., Petrova, M. M. and Kudryavtseva, T. (eds) Digital Transformation on Manufacturing, Infrastructure & Service. DTMIS 2022. Lecture Notes in Networks and Systems, vol. 684. Springer, Cham.

Cover of the proceedings volume Usability and User Experience

Explore the complete digital volume

Usability and User Experience

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-70-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →