Lesley-Ann Mathis, Kathrin Werner, Harald Widlroither (2023). Exploring use cases and user perception of a proactive voice assistant in automated vehicles. In: Human Factors in Transportation. AHFE Open Access, vol. 95, pp. 172–181. https://doi.org/10.54941/ahfe1003803
Future development of in-vehicle voice assistants (VA) aims at more adaptive and interactive interaction with users. One focus area is the development of proactive behaviour of VA. With the changing role from driver to passenger in increasingly automated vehicles, new opportunities for interactions with assistants in the car arise. However, potential use cases for proactive VA in automated vehicles have not been investigated so far. We present our approach to brainstorm and prioritize use cases for a proactive VA based on a specific persona and user journey, involving experts and users. Secondly, we present user’s assessment and reflections on the prioritized use cases after experiencing them in a driving simulator study. The findings show that especially office-work related use cases that enable an efficient use of the ride time or offers contributing to user’s well-being receive high acceptance from users. The analysis of the qualitative feedback highlights that for a proactive VA adaptivity to the user’s behaviour and non-intrusive formulation of suggestions and questions is important. The study provides a starting point to investigate proactive behaviour of in-vehicle VA in more detail. The proposed approach for use case derivation can also be applied to other personas or different application domains of VA.
Voice assistants (VA) are an established mode of interaction in many cars and typically respond to the user upon pressing a push-to-talk button or after ini- tiating the conversation by calling a trigger phrase. With the progress in the development of Artificial Intelligence-based systems, future voice assistants are imagined as being adaptive to the user and acting on their own behalf, i.e. proactively. For instance, Hyundai presented for its concept car “Concept-i” the digital assistant “Yui”, which supports the user with proactive offers to enhance his well-being and ride experience (Lugano, 2017). As pointed out by Lugano (2017), future in-car virtual assistants are considered by manu- facturers as a means to allow for a convenient and ubiquitous access during the ride to content as well as various services and products.
Exploring Use Cases and User Perception of a Proactive Voice Assistant 173
STATE-OF-THE-ART Concepts and use cases for proactive in-car voice assistants were already investigated for the car, however mainly with a focus on supporting the dri- ving task in a manual ride. Schmidt et al. (2019) present an investigation of proactive use cases for a VA during manual driving, showing that proactive offers for parking options or gas refueling receive high approval among users. Further studies about use cases for proactive VA were mainly conducted out- side the vehicle domain. A study by Reicherts et al. (2021) investigates user reactions to different scenarios for proactive smart speakers at home, such as a proactive reminder about a meeting or proactively setting up an appoin- tment. With an increasing level of vehicle automation, the role of the user in the car shifts from the driver towards the passenger, enabling the user to per- form non-driving related activities (NDRA) (Pfleging et al., 2016). Therefore, current research focuses on the intelligent and adaptive interaction with the user in the car, considering different levels of driving automation (Diederichs et al., 2022). To our knowledge, research about user’s perception of potential use cases for a proactive VA during automated driving is lacking. Thus, the goal of this paper is to present how we derived and evaluated different use cases as well as user’s perception of a proactive in-vehicle VA, focusing on non-driving relevant use cases during an automated ride.
METHOD
We took a multi-step approach to derive use cases and understand users’ feed- back on the proposed use cases (see Fig. 1). We started out with the collection of use cases in an expert workshop, followed by a scenario-based online study with users. Finally, users experienced prioritized use cases in a driving simu- lator setup and evaluated them using a Retrospective Think-Aloud (RTA). In the following, we will describe the procedure and analysis of each step in more detail.
Expert Workshop Our basis for the collection of relevant use cases was a persona and user journey. We focused on the persona of Matthias, 49 years old, external sales employee, who spends a lot of time in his car due to his job. He has a medium level of technology affinity and a probability to get car-sick when not looking up while riding (KARLI project, 2022). His user journey describes a business ride, starting out in manual mode at home in the town and changing into an automated ride SAE Level 4 (SAE International, 2021), first on a country road and finally on a highway. Based on the given persona and user journey, experts brainstormed ideas for potential use cases with a focus on non-driving relevant use cases, i.e., use cases for a proactive VA during SAE Level 4. In
Figure 1: Approach for use case derivation and evaluation.
Figure available in the original PDF paper.
the brainstorming workshop, N = 5 experts took part, with background in automotive human factors (N = 4) and/or computational linguistics (N = 2). A virtual whiteboard was used to present the persona and user journey, then collect the ideas for potential use cases on an individual basis and secondly, to group them into different use case clusters.
User Evaluation in Online Study Following that, the use cases were rated in online survey by in total N = 80 participants. Each use case was presented within a short scenario description to give participants an immersive description of the use case, as already used in similar studies, such as in Völkel et al. (2021). Each use case was rated with the 6 items presented in Schmidt et al. (2019) to evaluate pro- active use cases and with one additional item on efficiency. Table 1 shows an exemplary scenario description and the corresponding items from the online questionnaire.
Driving Simulator Study Based on the results of the online study, the 10 most favored use cases were selected for a driving simulator study. In the study, N = 32 experienced a Wizard-of-Oz setup with a proactive VA during an automated ride. The data on the use cases presented in this paper was collected within a larger study. Each participant started out with a short manual ride, then the car switched to automated driving SAE Level 4 (SAE International, 2021). While in automa- ted mode, the participant had a NDRA in form of a reading task for approx. 5 minutes. After that, the participant did not have a specific task. During the total ride time of 40 minutes, participants were addressed by the VA every 2–6 minutes proactively with one prompt. One prompt was formulated for each use case according to the same structure of an introductory sentence followed by a yes/no question. For example, for the scenario shown above in Table 1, the corresponding prompt was: “The coming route section cau- ses increased motion sickness. Would you like to interrupt your activity for 5 minutes?” The linguistic formulation of each prompt was based on the principles for VA output derived in Meck and Precht (2021). Prompts were recorded using a commercial text-to-speech engine with a female voice and were during the study automatically played at predefined route points.
Table 1. Exemplary scenario description and items used in online user study (translated from German original).
Table available in the original PDF paper.
Scenario description Items (rated on 5-point Likert scale) During the automated journey, you want to use the time and read on your tablet or in a book. After a few minutes, the VA warns you that the section ahead of you may lead to increased motion sickness while reading, due to the route. Therefore, it suggests interrupting your reading activity for 5 minutes.
1) I find the function useful. 2) I would use the function regularly. 3) The function excites me. 4) The function improves my safety. 5) The function increases my comfort. 6) The function is innovative. 7) The function increases my efficiency.
Exploring Use Cases and User Perception of a Proactive Voice Assistant 175
To evaluate participants’ perception of the speech prompts, we conducted a video-based Retrospective Think-Aloud (RTA) (Elbabour et al., 2017) to derive participant’s feedback and their assessment of each use case. For this, the experimenter showed the participant the situation of the prompt on the recording and then asked “What did you think about the offered function in this moment?” (translated from German original). In the following, par- ticipants were asked about their general perception of the proactive VA in a semi-structured interview. The participants’ answers were recorded and tran- scribed. A qualitative content analysis according to Mayring and Fenzl (2019) was applied to analyze the data.
RESULTS
Brainstorming and Clustering of Use Cases In total, 27 possible use cases for the selected user journey were collected in the expert workshop. In a second step, the findings were clustered into the five main topics office work, personalization, entertainment, knowledge, and well-being, shown in Table 2. The category office work was included, as the persona Matthias could use the ride time to prepare meetings or check the agenda while on his way to a customer. Other categories like entertainment or knowledge could also be suitable for other personas. The experts judged use cases in the categories personalization and well-being as relevant for Mat- thias, since he spends a lot of time in his car and therefore could benefit from an interior adapted to his needs and supporting his well-being.
Evaluation and Prioritization of Use Cases After consolidation of similar use cases from the expert workshop, 24 distinct use cases were included in the online study. Figure 2 presents the cumulative average of the ratings for all items per use case. The overall rating shows that the two office work-related use cases meeting preparation and to-do list are rated best. In contrast, several use cases from the cluster entertainment receive less approval from participants, while still ranging in an average scale
Table 2. Topic cluster for use cases derived from expert workshop.
Table available in the original PDF paper.
Cluster Description Office work (O) VA makes offers regarding office work related topics, such as reading out the to-do list or agenda. Personalization (P) VA makes offers to better tailor the ride experience to user’s preferences, interests and needs, such as saving particular settings to the user profile. Entertainment (E) VA offers options of entertainment during the ride such as music, videos, games. Knowledge (K) VA makes offers with option to gain information expanding the user’s personal knowledge, such as learning vocabulary or listening to a podcast. Well-being (W) VA supports the user during the ride with offers beneficial to health and well-being, e.g., a breathing or movement exercise.
Figure 2: Cumulated averages for each use case on a Likert scale from 1 (strongly disagree) to 5 (strongly agree). Error bars show +/−1 SEM.
area. Figure 3 depicts the averages per item of the three top and three worst rated use cases. Both office-work use cases receive high average values for the items usefulness (M = 4.61, SD = 0.50 for meeting preparation; M = 4.43, SD = 0.51 for to-do list) and efficiency (M = 4.43, SD = 0.59 for meeting preparation; M = 4.48, SD = 0.67 for to-do list; see Figure 3). The movement exercise from the cluster well-being is also rated favorably by participants, evaluated high for excitement (M = 3.82, SD = 1.25) but also for comfort (M = 3.91, SD = 1.14). As shown in Figure 3, the three entertainment use cases on gaming, user interests and small talk receive lower ratings on all items. For small talk, especially the item on regular use is rated worse in comparison to the other use cases (M = 2.26, SD = 1.23).
Assessment of Use Cases in Driving Simulator Study The results of the RTA show how users evaluated the top ten selected use cases after experiencing them during the automated ride from a qualitative point of view. Results from the performed content analysis are shown in Table 3, depicting the category inferred from the interview material and their corre- sponding relative frequency based on the number of comments. We can see that the majority of the feedback for all use cases is positive. The most frequ- ent word to describe any of the use cases was “useful”: “I found this a useful offer, I also used to do that when riding the bus in the past”(P15). For the two well-being use cases, participants pointed out that it increases their well-being (9%, respectively 10% of the comments): “That makes one feel good if you move a bit.” (P25). However, for the motion sickness use case, 17% of the comments also state that this use case it not useful for them. Participants
Exploring Use Cases and User Perception of a Proactive Voice Assistant 177
Figure 3: Individual average ratings of all items used on a Likert scale from 1 (strongly disagree) to 5 (strongly agree) for the 3 top-rated and the 3 worst-rated use cases.Table 3. Qualitative evaluation of the use cases in the driving simulator study. Percentages depict the frequency of each category in relation to the total sum of comments per use case.
Table available in the original PDF paper.
also find it helpful to be nudged by the system, e.g., when it comes to the use cases business contact (24%) or meeting preparation (17%): “That was good, I always forget that. And when you’re sitting in the car, you have the time.” (P31). Using the ride time in a beneficial way matters to participants, which they mention in a positive way for instance for the to-do-list and the reminder. For the personalization use case to save the mirror settings to the user profile, 16% of the comments refer to it as useless. Some also explicitly
state that they prefer to adjust the settings themselves rather than saving it to a profile (16%).
User Reflections of a Proactive In-Vehicle VA Overall, participants also pointed out requirements and expectations for their acceptance of proactive use cases during the RTA. First of all, some partici- pants mentioned that they would prefer a personalization of topics the VA proactively engages the user. One participant explains: “I would like to be able to specify [it]. I don’t get motion sickness at all, for example, so I would want to disable it […]” (P19). Also, participants liked about the interaction that they were able to either accept or decline what the VA offered: “[This function] was great. I have the option to say ‘yes’ or ‘no’, or ‘to do’ or ‘not to do’.” (P28). In addition, some participants would have preferred to get more explanation on selected proactive interactions, e.g., what is adapted exactly: “Then I asked myself, what will be adjusted now? So maybe it would be a bit more helpful to give some more examples of functions in general.” (P09). Furthermore, they expect that the VA adapts to their habits with the proa- ctive interactions, e.g., P01 says “If this was my own vehicle, I would wish that I enter it in the garage in the morning and start it, that [this function] might then be addressed relatively at the beginning.”
Furthermore, we analysed user’s general perception of the proactive VA after the ride from a corresponding interview question, which also gives insi- ghts to the perceived benefit from user’s point of view (see Table 4). From in total 50 coded statements, 86% contained positive associations of the proa- ctive VA, often pointing out that the assistant was perceived as pleasant and useful. “It was pleasant […] I didn’t feel stressed in any way or pushed into anything I didn’t want to do.” (P28). Some comments also refer to it as increa- sing the value of travel time or mention that it takes over work from them: “I think it’s practical when you’re actually just an observer in the car, then you don’t have to worry about something like that yourself.” (P03). In contrast, 14 % of the comments described the experience as unfamiliar (6%) or that users prefer to do things themselves (8%): “I don’t think this is something in the long run. I’ll have my own workflow […] Maybe if there’s a function to say, this trip there won’t be proactive interaction” (P29).
Table 4. User’s perception of the proactive VA clustered into positive and negative associations from a total of 50 coded comments.
Table available in the original PDF paper.
DISCUSSION
Our approach focused on deriving use cases for a proactive VA during auto- mated driving and receiving user feedback on prioritized use cases. While use case studies often include an evaluation on a conceptual level with scenario
Exploring Use Cases and User Perception of a Proactive Voice Assistant 179
descriptions, we additionally included a user evaluation with a Wizard-of-Oz proactive VA in a driving simulator ride to let users experience the use cases and collect their qualitative feedback using a RTA.
Relevance of New Use Cases The results show that the prioritized use cases in this study receive high approval by participants. Especially use cases which enable the use of tra- vel time for office-work related tasks or which increase well-being, such as a guided movement exercise or a motion sickness warning, are among the top- rated use cases. Given the comparably low rating for some entertainment use cases, also including small talk, participants overall preferred use cases with a clear benefit such as being useful to them, increasing their efficiency or their comfort. This might be due to the fact that current in-vehicle VA are mostly perceived as tools to perform goal-directed tasks, such as receiving naviga- tion directions or writing a message. With an increasing intelligent behavior of VA in the future, their role is expected to change more towards a compa- nion (Biundo et al., 2016; Lugano, 2017). Given a changing relation between user and VA, the scope of use cases will also be affected, so that use cases like small talk might become more common.
Challenges of Proactivity Proactive voice interaction is a sensitive topic, as user concerns regarding the appropriateness of a proactive behavior in certain contexts have been rai- sed in previous studies. In the current study, at least few participants rated proactive use cases negatively because they prefer to do things themselves and feel patronized by the VA. This fear of a loss of control resonates with previous studies on proactive VA (e.g., Reicherts et al., 2021). As shown by the qualitative feedback, it also matters for users how the proactive intera- ctions is phrased, e.g. as yes/no question, and when it occurs, e.g. according to their routine. These are among the key questions for proactivity pointed out in previous research, namely when and how a user should be addressed by the system (Nothdurft et al., 2015). Thus, future research is needed to investigate these questions for proactive in-vehicle assistants in more depth. We also conclude from the qualitative feedback that for the acceptance of a proactive VA, adaptivity to the users’ preferences and routines is crucial. This requires a learning system which collects information about the user in the vehicle in the long-term (Diederichs et al., 2022).
Limitations Since our approach started out with one specific persona and user journey, the results only present a selection of possible use cases of a proactive in-vehicle VA. Given the user journey “business ride”, several office-work related use cases were in focus. However, the approach presented can be easily repeated with a different persona and user journey to collect more potential use cases. We also would like to point out that the very positive results of the driving simulator study might be influenced by the fact that the proactive VA was a
new feature that participants had not experienced before. Therefore, evalua- tions might differ when assessing use cases and interactions in the long run, especially with a fading fascination for a new product (Petterson, 2017).
CONCLUSION
Proactivity for in-vehicle VA has the potential to become a central feature to make assistants more interactive. We derived a set of prioritized use cases based on a selected user journey, which received a high acceptance from users after having experienced them. The approach can be applied to derive further use cases for different personas and user journeys. The findings show that use cases which provide a benefit in efficiency and comfort are preferred by users. Furthermore, participants’ feedback shows that adaptivity of the pro- active interaction to individual users is important, as well as the content of the proactive address. Based on these initial results, we aim to validate the use cases during a real road study with a Wizard-of-Oz vehicle as well as to investigate concrete proactive interaction concepts with users.
ACKNOWLEDGMENT
This study was conducted within the project KARLI (“Artificial Intelligence for an adaptive, responsive and level conform interaction with future vehi- cles”), receiving funding from the Federal Ministry of Economic Affairs and Climate Action.
REFERENCES
Biundo, S., Höller, D., Schattenberg, B. and Bercher, P. (2016) ‘Companion- Technology: An Overview’, KI - Künstliche Intelligenz, Vol. 30, no. 1, pp. 11–20. Diederichs, F., Wannemacher, C., Faller, F., Mikolajewski, M., Martin, M., Voit,
M., Widlroither, H., Schmidt, E., Engelhardt, D., Rittger, L., Hashemi, V., Sah- akyan, M., Romanelli, M., Kiefer, B., Fäßler, V., Rößler, T., Großerüschkamp, M., Kurbos, A., Bottesch, M., Immoor, P., Engeln, A., Fleischmann, M., Sch- weiker, M., Pagenkopf, A., Mathis, L.-A. and Piechnik, D. (2022) ‘Artificial Intelligence for Adaptive, Responsive, and Level-Compliant Interaction in the Vehicle of the Future (KARLI)’, in Stephanidis, C., Antona, M. and Ntoa, S. (eds) HCI International 2022 Posters, Cham, Springer International Publishing, pp. 164–171. Elbabour, F., Alhadreti, O. and Mayhew, P. (2017) ‘Eye Tracking in Retrospective Think-Aloud Usability Testing: Is There Added Value?’, Journal of Usability Studies, Vol. 12, no. 3, pp. 95–110. KARLI project (2022) KARLI Personas [Online]. Available at https://karli- projekt.de/wp-content/uploads/2022/09/KARLI-Personas-Infographic-scaled.jpg. Lugano, G. (2017) ‘Virtual assistants and self-driving cars’, Proceedings of 2017 15th
International Conference on ITS Telecommunications (ITST): May 29-31, 2017, Warsaw, Poland. Warsaw, Poland, 29.05.2017 - 31.05.2017. [Piscataway, NJ], IEEE, pp. 1–5. Mayring, P. and Fenzl, T. (2019) ‘Qualitative Inhaltsanalyse’, in Baur, N. and Blasius, J. (eds) Handbuch Methoden der empirischen Sozialforschung, Wiesbaden, Springer Fachmedien Wiesbaden, pp. 633–648.
Exploring Use Cases and User Perception of a Proactive Voice Assistant 181
Meck, A.-M. and Precht, L. (2021) ‘How to Design the Perfect Prompt: A Linguistic Approach to Prompt Design in Automotive Voice Assistants – An Exploratory Study’, 13th International Conference on Automotive User Interfaces and Intera- ctive Vehicular Applications. Leeds United Kingdom, 09.09.2021. New York, NY, USA, ACM, pp. 237–246. Nothdurft, F., Ultes, S. and Minker, W. (2015) ‘Finding Appropriate Interaction Stra-
tegies for Proactive Dialogue Systems—An Open Quest’, in K. Jokinen and M. Vels (eds) Proceedings of the 2nd European and the 5th Nordic Symposium on Multimodal Communication, August 6-8, 2014, Tartu, Estonia. Linköping Uni- versity Electronic Press, 2015. S. 73-80, Linköping University Electronic Press, pp. 73–80. Petterson, I. (2017) ‘Travelling from Fascination to New Meanings: Understanding User Expectations Through a Case Study of Autonomous Cars’, International Journal of Design. 11(2), pp. 1–11. Pfleging, B., Rang, M. and Broy, N. (2016) ‘Investigating user needs for non-driving-
related activities during automated driving’, Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia - MUM ‘16. Rovaniemi, Finland, 12.12.2016 - 15.12.2016. New York, New York, USA, ACM Press, pp. 91–99. Reicherts, L., Zargham, N., Bonfert, M., Rogers, Y. and Malaka, R. (2021) ‘May I Interrupt? Diverging Opinions on Proactive Smart Speakers’, CUI 2021 - 3rd Conference on Conversational User Interfaces. Bilbao (online) Spain, 27 07 2021 29 07 2021. New York, NY, United States, Association for Computing Machinery, pp. 1–10. SAE International (2021) ‘J3016 Taxonomy and Definitions for Terms Related to
Driving Automation Systems for On-Road Motor Vehicles’, SAE International - Recommended Practice. Schmidt, M., Stier, D., Werner, S. and Minker, W. (2019) ‘Exploration and assessment of proactive use cases for an in-car voice assistant’, Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung, 148–155. Völkel, S. T., Buschek, D., Eiband, M., Cowan, B. R. and Hussmann, H. (2021)
‘Eliciting and Analysing Users’ Envisioned Dialogues with Perfect Voice Assi- stants’, Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Yokohama Japan, 08 05 2021 13 05 2021. New York, NY, United States, Association for Computing Machinery, pp. 1–15.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Transportation
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-71-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional