Hatice Kirkici, Paula Carneiro, Paulo Pedrosa, Sandra Melo, Ana Colim (2024). User experience assessment with high level autonomous vehicle console – a pilot test. In: Human Systems Engineering and Design (IHSED2024): Future Trends and Applications. AHFE Open Access, vol. 158, pp. 127–133. https://doi.org/10.54941/ahfe1005536
To gain broad approval, autonomous vehicles must not only be seen as practical but also as safe and effective. The interior design of both current and future vehicles is characterized by increasingly smooth and seamless surfaces. By integrating ergonomic design principles that prioritize the use of lightweight materials and aerodynamically efficient forms in autonomous vehicles, there is potential to reduce energy consumption, making a significant contribution to the overall sustainability of transportation systems. This study focuses on evaluating the user experience with an autonomous vehicle console prototype. The primary goal is to gain valuable insights into usability, efficiency, and overall user satisfaction regarding the console prototype. The participants engaged with the console interface in a controlled laboratory setting, with each interaction lasting less than 10 minutes. Initially, participants conducted a series of trial interactions with the autonomous vehicle's console, guided by the primary investigator, for a period of 3 minutes. Following this, they completed a questionnaire consisting of 12 items, including the System Usability Scale (SUS). It's worth noting that the majority of the participants, accounting for 60% of the total, were familiar with Level 3 and Level 4 autonomous vehicle concepts. The overall assessment score, determined through the System Usability Scale (SUS), reached an impressive 84.5. Consequently, the console prototype received a favorable verdict regarding its acceptability, further supported by an outstanding qualitative rating. In summary, the empirical findings collectively support the idea that the usability features of the autonomous vehicle console currently meet acceptable standards. Future research efforts will focus on an expanded participant pool, and center on the evaluation of haptic feedback, backlight effects, and capacitive touch sensations pertaining to the buttons.
Autonomous vehicles (AVs) have the potential to profoundly transform our transportation system, positively affecting vehicle safety, traffic flow, and how people travel. The main goals of autonomous vehicles are to improve
road quality, safety, efficiency, and comfort. With high automation, drivers will be freed from the duty of driving, enabling them to engage in activities that weren’t feasible in manually operated vehicles. These new in-vehicle activities will necessitate redesigning vehicle information and functions, introducing fresh interaction methods (Jorlöv et al., 2017). Introducing new in-vehicle activities requires updating the vehicle’s information and functions, creating new ways for interaction. Automation changes how drivers process information and perform tasks, replacing or changing task requirements. Any changes should match human thinking, emphasizing safety and efficiency during autonomous driving. The redesigned system should support a collaborative effort between the driver and automation for top-notch performance (Yamani and Horrey, 2018). The realization of advantages offered by autonomous vehicles, such as enhanced safety, convenience, fuel efficiency, and reduced emissions, hinges on the acceptance and comfort of consumers with the vehicle’s design (Tang et al., 2020). Designing innovative products like autonomous vehicles requires the assessment of their user-friendliness. User evaluation is indispensable to guarantee that the design aligns with the requirements and preferences of the target audience (Angeleska et al., 2022). According to ISO (The International Organization for Standardization), usability is an extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use (ISO 9241-11:2018, 2018). Usability tests place the user at the center of the design process. They help ensure that products or interfaces are created having in mind the needs, preferences, and expectations of the users (Gibson et al., 2016). Usability tests reveal issues and pain points users may encounter while interacting with a product. These issues can include confusing interfaces, frustrating interactions, or features that don’t work as expected. By identifying and addressing usability problems, usability tests contribute to a better overall user experience. This, in turn, can lead to higher customer satisfaction and loyalty. Early identification and correction of usability issues can save significant costs that might be incurred if problems are discovered after a product has been launched or is in widespread use. Usability evaluation is used to assess the extent to which a system’s human-machine interface (HMI) complies with the various usability criteria that are applicable in its specific context of use (Bevan, 1995). Usability evaluation gauges how well a system’s interface aligns with the specific usability standards relevant to its intended use. The findings from such evaluations can predict a product’s success in its target market, facilitate comparisons between similar products, offer insights for design improvements, and help estimate potential training needs for the product (Harvey et al., 2011). According to a literature review carried out by Almasi et al. (2023), the most widely used questionnaires were the System Usability Scale (SUS), Technology Acceptance Model (TAM), Situation Awareness Rating Technique (SART), Questionnaire for User Interaction Satisfaction (QUIS), Unified Theory of Acceptance and Use of Technology (UTAUT). According to previous studies, among these questionnaires, SUS emerges as the most frequently used and the most validated tool for appraising the usability of products or services (Li et al.,
User Experience Assessment With High-Level Autonomous Vehicle Console – A Pilot Test 129
2017; Almasi et al., 2023). The objective of this research is to assess user experience with a prototype of an autonomous vehicle console to gather valuable insights into its usability, efficiency, and overall user satisfaction.
METHODOLOGY
The prototype is one autonomous vehicle console that includes, a touch display, three capacitive touch buttons (dimming, ambient color, volume), and one capacitive slider in the center of the console screen, as well as four buttons (driving mode, park button, up arrow, and down arrow) on the console control component all buttons are with backlighting and haptic feedback (Figure 1).
Figure 1: Autonomous vehicle console prototype.
The interaction with the console was performed in a laboratory context and the experimental procedure took less than 10 minutes per participant (Figure 2).
Figure 2: Experimenting the console prototype.
Involvement in the study was voluntary, and all data was examined and stored in a manner that preserved anonymity. First, the participants tested the different trials with the autonomous vehicle console for 3 minutes with the help of the main researcher.
Following that, they answered a 12-item questionnaire that also included a System Usability Scale (SUS) (Brooke, 2020). It consists of a 10-item questionnaire with a 5-point Likert scale for each one, ranging between “Strongly agree” to “Strongly disagree”. Then, a global score is calculated, which can range from 0 to 100, with 68 indicating that the system’s usability is good. SUS items were adapted for the autonomous vehicle console prototype. The term “system” was replaced with the term “console” in the statements. The SUS score was calculated based on Brooke’s (2020) study. Figure 3 shows SUS (10-item questionnaire with a 5-point Likert scale).To calculate the SUS score, firstly the score contributions from each item were summed. Each item’s score contribution ranged from 0 (strongly disagree) to 4 (strongly agree).
Figure 3: Adapted SUS for autonomous vehicle console prototype.
In addition, the questionnaire includes a category of questions related to demographic data (e.g., gender, age, and the highest level of education), allowing the sample characterization. The questions mentioned above are based on a closed-form response system. The participants were required to choose between certain given options. If these options did not match their opinions, they could freely express them as open comments about positive and/or negative aspects associated with the console interaction.
User Experience Assessment With High-Level Autonomous Vehicle Console – A Pilot Test 131
RESULTS
Demographic Characteristics of the Participants The autonomous vehicle console prototype was tested by 15 participants (53% were male and 47% female). Relating to age, 40% of the participants were in the 26–35 years old group, 20% in the 18–25 group,13% were in the 46–55 group and 27% were in the 36–45 group. Therefore, the sample was slightly younger with a slightly higher representation of the male participants. Of all the respondents, 53% had a master’s degree, 27% of them had a bachelor’s degree and 20% had completed a PhD. Only one participant was left-handed.
Technology Knowledge To understand the technical knowledge of the participants, the questionnaire contains questions about the self-use of the smartphone and familiarity with Level 3 and Level 4 autonomous vehicle concepts. The majority of the participants (60%) are familiar with Level 3 and Level 4 autonomous vehicle concepts (Figure 4).
Figure 4: Familiarity with autonomous vehicle concept.
All the participants have a smartphone, and the most used features are sending messages, calls, and social media (Figure 5).
During SUS score calculation firstly the score contributions from each item were summed. Each item’s score contribution ranged from 0 to 4. For items 1, 3, 5, 7, and 9, the score contribution was the scale position minus 1. For items 2, 4, 6, 8 and 10, the contribution was 5 minus the scale position. Multiplying the sum of the scores by 2.5, we obtain the overall value of SUS. After obtaining the SUS score of each participant, the sum of all the scores was divided by 15, thus obtaining the final SUS value which is 84.5. As shown in Figure 6, the autonomous vehicle console prototype was acceptable with an excellent adjective rating.
Figure 5: Most used features of a smartphone.Figure 6: SUS evaluation graphic.
CONCLUSION
This research aimed to investigate the user experience with a prototype of an autonomous vehicle console. The questionnaire results revealed that the usability test was conducted with participants largely familiar with the concept of autonomous vehicles. Furthermore, all participants possessed smartphones and proficiently utilized various features. Thus, it can be deduced that the individuals involved in the pilot test, assessing the usability of the autonomous vehicle console, had the necessary knowledge and capability to perform the test effectively. Overall results showed that the usability of the autonomous console is already in the acceptable range. A
User Experience Assessment With High-Level Autonomous Vehicle Console – A Pilot Test 133
positive trend is particularly evident in comparison with the values of other evaluated innovations (Bangor et al., 2009; Harvey et al., 2011). Usability problems regarding the console were more related to the touch feeling of the arrow buttons. Future investigations may involve an expanded participant pool, and center on evaluation of haptic feedback, backlight effects, and capacitive touch sensations pertaining to the buttons.
ACKNOWLEDGMENT
This work has been supported by FCT – Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020.
REFERENCES
Almasi, S. et al. (2023) ‘Usability Evaluation of Dashboards: A Systematic Literature Review of Tools’, BioMed Research International, 2023. Available at: https://doi. org/10.1155/2023/9990933. Angeleska, E. et al. (2022) ‘Inclusive Autonomous Vehicle Interior Design (IAVID)
Platform’, Design for Inclusion, 45, pp. 54–62. Available at: https://doi.org/10. 54941/ahfe1001869. Bangor, A., Kortum, P. and Miller, J. (2009) ‘Determining what individual SUS scores mean; adding an adjective rating’, Journal of usability studies, 4(3), pp. 114–23. Bevan, N. (1995) ‘Human-computer interaction standards’, Advances in Human Factors/Ergonomics, 20(B), pp. 885–890. Available at: https://doi.org/10.1016/S0921-2647(06)80326-6. Brooke, J. (2020) ‘SUS: A “Quick and Dirty”Usability Scale’, Usability Evaluation in
Industry, (November 1995), pp. 207–212. Available at: https://doi.org/10.1201/ 9781498710411-35. Gibson, Z., Butterfield, J. and Marzano, A. (2016) ‘User-centered Design Criteria in Next Generation Vehicle Consoles’, Procedia CIRP, 55, pp. 260–265. Available at: https://doi.org/10.1016/j.procir.2016.07.024. Harvey, C. et al. (2011) ‘A usability evaluation toolkit for In-Vehicle Information
Systems (IVISs)’, Applied Ergonomics, 42(4), pp. 563–574. Available at: https: //doi.org/10.1016/j.apergo.2010.09.013. ISO 9241-11:2018 (2018) Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts. Li, R. et al. (2017) ‘Effects of interface layout on the usability of In-Vehicle
Information Systems and driving safety’, Displays, 49, pp. 124–132. Available at: https://doi.org/10.1016/j.displa.2017.07.008. S. Jorlöv, K. Bohman and A. Larsson (2017) ‘Seating Positions and Activities in Highly Automated Cars - A Qualitative Study of Future Automated Driving Scenarios’, pp. 119–120. Tang, P., Sun, X. and Cao, S. (2020) ‘Investigating user activities and the
corresponding requirements for information and functions in autonomous vehicles of the future’, International Journal of Industrial Ergonomics, 80, p. 103044. Available at: https://doi.org/10.1016/j.ergon.2020.103044. Yamani, Y. and Horrey, W. J. (2018) ‘A theoretical model of human-Automation interaction grounded in resource allocation policy during automated driving’, International Journal of Human Factors and Ergonomics, 5(3), pp. 225–239. Available at: https://doi.org/10.1504/ijhfe.2018.10016994.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Systems Engineering and Design (IHSED2024): Future Trends and Applications
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-34-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional