Waldemar Titov, Seyedehfatemeh Ayoubi, Carl Friedrich Gerlach, Thomas Schlegel (2022). Evaluation of a Mobile and Seamless Travel Companionship in Smart Public Transport. In: Human Interaction & Emerging Technologies (IHIET 2022): Artificial Intelligence & Future Applications. AHFE Open Access, vol. 68, pp. 357–364. https://doi.org/10.54941/ahfe1002753
Introduction:In the SmartMMI (https://smartmmi.de/project/) project we combine a smart public transport data platform, a mobile application and a SmartWindow implemented in a public transport enabling the use of contextaware passenger information. Prior we evaluated the prototypes of our mobile application based on the method of paper prototypes using a multi-device evaluation approach introduced in [1]. In the current work, we go one step further and evaluate the high fidelity and fully developed SmartMMI mobile application through an online survey and interview in which 24 persons participated. We conducted two different questionnaires that were used to evaluate user interface satisfaction (QUIS) and user experience (UEQ) and personal interviews.Through the development process, several evaluation phases resulted in gradually improved prototype. The iterative evaluation procedure is shown in [2]. However, how would be the user interaction with such companion technology in case of access to both the SmartWindow and mobile app interface? Would they be satisfied with this ability? This matter would be the focus of this research to evaluate the travel companion during their journey.Conducted Analysis: UEQ Analysis, QUIS and Our Questions Analysis: From quality aspects point of view, hedonic quality items (simulation and originality) assess the lowest mean, among others. Benchmark analysis show that SmartMMI app scores high compared to the products in the benchmark which this can indicate that users generally found the product’s user experience (UX) satisfactory except the Novelty and Simulation, which are indicated again as the weakness of the app by participants.Results:Many positive comments were made on the prototype in general. It was simply structured, minimalistic designed and quickly to learn. In addition, they acknowledged the door-to-door routing, the icon-based choice of transportation mode, the integrated map, and the systematic support during the journey. Some liked the simplification, and many stated it was better to be sufficient and useful for maintaining orientation during the trip. Marking icons was intuitively usable. However, some participants felt confused with the 4-digit number and green numerical sign in the left side of each transition which the latter means the delay in fact. Finally, suggestions for more recognizable map icon received. Participants also liked the idea of having mobility restriction filter in the setting and bookmark option of the routes. Overall, it is interesting to note that SmartMMI app was one of the simplest and easiest applications among other tracking apps which its main role is assisting the passenger through the journey. This was the main result of our evaluation.Outlook:In the full paper, we will describe our analysis and evaluation approaches that we used conducting the studies in more detail. We report on our experiences, the results and the designs of studies implemented, as well as give insights into lessons learned while developing and evaluating Seamless Travel Companionship in Smart Public Transport.References:[1] Titov, W., Tran, H., Keller, C., & Schlegel, T. (2020, July). A Multi-device EvaluationApproach of Passenger Information Systems in Smart Public Transport. InInternational Conference on Human-Computer Interaction (pp. 340-358). Springer,Cham.[2] Keller, C., Titov, W., Sawilla, S., & Schlegel, T. (2019). Evaluation of a smart publicdisplay in public transport. Mensch und Computer 2019-Workshopband.
This paper accumulates findings retrieved during the research project Smar- tMMI that was funded by the mFund research initiative of the German Federal Ministry of Transport and Digital Infrastructure. As Keller et al. introduced, the whole SmartMMI system combines a smart public tran- sport data platform, a mobile application and a SmartWindow and enables context-aware passenger information (Keller et al. 2019). Beyond the design of the interactive SmartWindow, they also research the interplay of mobile application and SmartWindow for situational passenger information. To
develop a travel companionship and interactivity, the passenger information system needs to be aware of the passenger’s expectations and needs. To do this, we evaluated SmartMMI app through an online survey and interview in which 24 persons participated. We conducted two different questionnaires that were used to evaluate user interface satisfaction (QUIS) and user experi- ence (UEQ), and structured interviews. Through the evaluation process, the result of each study in each evaluation method resulted in practical feedba- cks, which is mentioned at the end of this paper. However, how would be the user interaction with such companion technology in case of access to both smart window and mobile app interface? Would they be satisfied with this facility? This matter would be the focus of this research to evaluate the travel companion (TC) during their journey.
RELATED WORK
De Amorim et al. report on a usability study testing of a mobile application for public transport ticketing that applied a check-in/be out scheme, using near field communication, Bluetooth, and GPS (De Amorim et al. 2019). The study was implemented as a field test in the city of Porto, using a pro- totype of the developed application. In this study, eight users tested the app in real public transport, performing given number of tasks. The users started at a station they chose, and their trip required at least one interchange. The study included a pre-test questionnaire and during performing the tasks, the users were asked to think aloud. All tasks were performed using the mobile application. After the tasks were finished, the authors conducted a structured interview, focusing on the app’s usability.
Habermann et al. describes the usability evaluation of a smartphone appli- cation that was developed as part of a project concerned with multi-modal transport (Habermann et al. 2015). A platform supporting mobility with several modes of transport was also developed in this project. The core functions of the applications were routing and booking of trips with public transport or other transport modes, such as car sharing. The authors report on the evaluation of an initial prototype for the smartphone application. They evaluated the perceived ease of navigation and interface quality in a questionnaire that was filled out by participants after performing the tasks.
The smartphone application that Chowdhury and Giacaman report about was also evaluated in the field (Chowdhury et al. 2015). The application enables the user to plan trips with one or more destinations and supports the user during their trip. The authors performed a study with 21 engineering students that were asked to travel four routes in total, of which the parti- cipants should travel one set of two routes using the given application and one set of two routes without the application, using information sources they normally would use during such a task. The study was unsupervised, and the participants were given forms to record at which time they started their trip, how long they waited and how long it took them to board a vehicle, as well as their time of arrival and how long it took them to plan their trips. The partici- pants were also asked to fill out a questionnaire focused on the application’s usability, after completing their tasks.
Evaluation of a Mobile and Seamless Travel Companionship in Smart Public Transport 359
Stopka was dealing with user requirements for mobile application to sup- port door-to-door mobility in public transport (Stopka et al. 2014). Transport operators for coordination of timetables, synchronizing arrival and depar- ture times between the different transportation modes, and the traveler information system, often create these applications (Ezzedine, et al. 2008).
METHODOLOGY
An efficient and inexpensive method to measure user experience from diffe- rent user group’s points of view is thus the usage of validated questionnaires. It has the advantage that the questionnaires split the general notion of user experience into several simple quality criteria, which describe distinct and relatively well-defined aspects of user experience that can be measured independently (Schrepp et al. 2014).
We decided to conduct our own questionnaire and two other literature published ones that we believed could be adapted to evaluate the mobile app more precisely. The questionnaires we used includes as follow:
1. UEQ (User Experience Questionnaire) – The main goal of the UEQ is to allow a fast and immediate measurement of user experience. The UEQ considers aspects of pragmatic and hedonistic quality. The original Ger- man version of the UEQ was created 2005 by a data analytical approach to ensure a practical relevance of the constructed scales, which corre- spond to distinct quality aspects (Schrepp et al. 2014). The results of some studies showed a sufficiently high reliability of the scales (measured by Cronbach’s Alpha) (Laugwitz et al. 2008). 2. QUIS (Questionnaire for User Interface Satisfaction) – It was developed by Chin et al. in 1988. A usability-testing instrument has been designed to measure the satisfaction of users about their interaction with the compu- ter interface. QUIS consists of 27 items in five sections. The first section measures the overall satisfaction, and the four others measure user sati- sfaction regarding screen, terminology and information, learning, and system capabilities aspects. In this paper, we used the short version of QUIS 5.5. (Chin et al. 1988). 3. Our Questionnaire – This is composed of some statements about user’s experience with navigation, map visualization, battery consumption, their opinion about ads and level of satisfaction with the app.
Participants The survey was conducted with 24 persons working with two different inter- faces of virtual form of smart window and the mobile app. Respondents comprising undergraduate/ postgraduate students, employees, parents with children, and tourists are involved in the evaluation of the relevant features of the SmartMMI app. Figure 1 shows selected snapshots of the SmartMMI app used in our evaluation.
The ages of the participants ranged from 21 to 36 years. Thirteen partici- pants were students of different majors, 6 participants employed and five tourists. Fifteen participants stated to use public transport regularly, five
Figure 1: Screenshots of the SmartMMI app interface during intermodal Journey. Left: start page. Middle: choosing from recommended trips. Right: trip details of an individual track.
persons stated one to three times a month and four other participants use public transport less frequently than once a month. All subjects stated to have a high technical affinity, were familiar with the operating technology, and could participate in the study without extensive technical introduction.
Study Procedure The questionnaire was answered online through the generated URLs list that sent to the participants via e-mail individually. The questionnaire was divi- ded into two parts. After a few general questions, the first study design section contained about the attractiveness, perspicuity, efficiency, dependa- bility, simulation, and novelty based on standard UEQ method. The second part asked questions about the user interface satisfaction and technical aspe- cts such as the navigation, map visualization, learning of the app and app capabilities.
The results give a good insight into the varying preferences of different pas- sengers. At the end, we held an online interview as well to collect information on the following:
What information is important for passengers during a train journey?
Which information should be displayed on the SmartMMI app, specifi- cally?
What are the needs experienced by the users in connection with the app?
What factors are relevant for each interaction point and how could they influence the passenger acceptance and use of the app?
With this approach, we expected that users bring up new ideas for information that could be considered for further development.
Evaluation of a Mobile and Seamless Travel Companionship in Smart Public Transport 361
Table 1. The mean and variance of UEQ scales and coefficient of Alpha per item.
Conducted Analyses The evaluation approaches for mobile app were analyzed according to the characteristics defined in the section above. Thus, the overall results for each research question and attributed extracted from the evaluation approaches are discussed in subsequent sections. Carrying out the whole survey took around 13 minutes averagely per person. When working with the SmartW- indow and then moving to the app, participants spent on average 65 seconds on completing the task and 203 seconds for questionnaire during the remote online meeting.
UEQ Analysis Several studies have shown a sufficient reliability (measured by Cronbach’s Alpha) and construct validity of the scales (Laugwitz et al. 2008). Inter-item correlation alpha values varied by only 0.05. The items are scaled from −3 to +3. Thus, −3 represents the most negative answer, 0 a neutral answer, and +3 the most positive answer. The mean ratings varied between 0.427 and 1.082, while standard deviations ranged from 0.836 to 1.479. Thus, it is no surprise that the mean value is above the neutral value (i.e., 0) of the 7-point Likert scale.
A close inspection of the mean ratings in Table 1 reveals that the ratings of items within each travel need factor align close to one another, although novelty items are negatively skewed toward the higher end of the 7-point agreement scale. The UEQ scales distribution of responses are displayed in Figure 2. The average rating of the entire sample is above or close to the mean and the largest mean difference was found in novelty features of the app. From quality aspects point of view, hedonistic quality items (simulation and originality) assess the lowest mean, among others.
With the availability of a benchmark, it is relatively easy to decide if a new product has sufficient user experience to be successful in the market. A com- parison of the results for the different scales with the results of the products in the benchmark allows then conclusions about the relative strengths and weaknesses of the product.
Figure 2 shows that SmartMMI app scores high compared to the products in the benchmark which this can indicate that users generally found the pro- duct’s user experience (UX) satisfactory except the Novelty and Simulation, which are indicated again as the weakness of the app by participants.
Figure 2: Left: The distribution of UEQ scales answers; Right: Benchmark graph from UEQ Analysis.Figure 3: Left: Distribution of overall reaction to the app; Right: Capabilities Compari- son of SmartMMI app.
QUIS and Our Added Questions Analysis Summary data for each question in the short version of QUIS 5.5 are cate- gorized bellow. Each of the six general satisfaction questions, and 22 specific questions are listed by question number.
overall reaction to the app (shown in figure 3 left)
app screen: All participants agreed on easiness of reading characters in the app and just 20% of them found the organization of the information confusing.
terminology and app information: participants stated that the understan- ding of contents was consistent but there was no availability of error messages and progress of the task during navigating.
learning: They were mostly satisfied with the interaction with the app and showed that it was very easy to learn features.
App capabilities (shown in figure 3 right).
RESULTS AND DISCUSSION
We implemented two different questionnaires and interviews to analyze the satisfaction, usability, interaction, design, and companionship of
Evaluation of a Mobile and Seamless Travel Companionship in Smart Public Transport 363
functionalities of the SmartMMI app profoundly. Some caveats need to be pointed out about the interpretation of these data. First, due to the COVID- 19 pandemic, it was limited to invite participants on site and give them more impression to reach results that are more reliable. It is also possible that the results could have been somewhat different if we had been able to collect data from more participants using the questionnaire and the obstacle was that the app was available only for android users. Keeping all those caveats in mind, it is interesting to note that the SmartMMI app was one of the simplest applications among other tracking apps which its main role is assisting the passengers through the journey while using SmartWindow complimentary. Additionally, it was concluded that the app was successful in applying tra- vel companion during the passenger’s journey. These were the main result of our evaluation that counts as an exclusive feature among other independent complex applications in the market.
CONCLUSION
Remarkably, mobile apps are growing in daily life and the search for quality of these applications grows in proportion. Despite this evolution, we were able to observe that there are still needs for studies on mobile apps usability evaluation and travel companion technologies. This paper presents the outco- mes of results within SmartMMI mobile app, which focuses on the “Analysis of the user experience and interaction in using the TC capabilities”. As a first step, the travel companion has been ‘deconstructed’ into its consumer- oriented capabilities and interaction points, i.e., all those situations in which the travel companion may assist the user in different phases of the travel experience. The main consumer interaction points identified are user iden- tity, preferences, planning, adding tickets, information, disruption, after trip, and on-going communication. For each interaction point, a series of assum- ptions have been formulated regarding factors (incentives, needs, constraints, barriers) that could (positively or negatively) influence the consumer uptake of the TC approach. To validate these assumptions, questionnaires and inte- rviews were conducted with 24 users from different perspectives to collect further information and to better understand the factors that could influ- ence the uptake and use of the travel companion. The questionnaires and interviews have provided valuable feedback on the design and recommended the functionalities, which the travel companion should have. Nevertheless, more potentials for further improvements and studies can be reached with the public implementation of the project in the city of Karlsruhe through the vari- ous feedbacks. Human Systems Integration (HSI) is becoming a critical piece of complex systems to help resolve system designs. This proposal has presen- ted a growing body of knowledge for HSI and new technologies that are being developed to capture critical aspects of HSI. The development of a framework for Human Systems Integration with Systems Modeling Language (SysML) will enable teams to collaborate better by providing a common language and process to distribute models and share information. The Human Systems Integration component in systems engineering will be able to recognize the
human as an integral element of every system by representing behaviors, constraints, states, and goals through-out the entire lifecycle.
ACKNOWLEDGMENT
This work was conducted within the scope of the research project “Smar- tMMI - model- and context-based mobility information on smart public displays and mobile devices in public transport” and was funded by the Ger- man Federal Ministry of Transport and Digital Infrastructure as part of the mFund initiative (Funding ID: 19F2042A). Refer to SmartMMI homepage for more information: https://smartmmi.de/project/. We would like to thank Seyedehfatemeh Ayoubi for her excellent contribution to this project.
REFERENCES
Amorim, D. M. D., Dias, T. G., & Ferreira, M. C. (2018, October). Usability evaluation of a public transport mobile ticketing solution. In International Conference on Human Systems Engineering and Design: Future Trends and Applications (pp. 345–351). Springer, Cham. Chin, J. P., Diehl, V. A., & Norman, K. L. (1988, May). Development of an instrument
measuring user satisfaction of the human-computer interface. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 213–218). Chowdhury, S., & Giacaman, N. (2015). En-route planning of multi-destination public-transport trips using smartphones. Journal of Public Transportation, 18(4), 3. Ezzedine, H., Bonte, T., Kolski, C., & Tahon, C. (2008). Integration of traffic
management and traveller information systems: basic principles and case study in intermodal transport system management. International Journal of Computers Communications & Control, 3(3), 281–294. Habermann, A. L., Kasugai, K., & Ziefle, M. (2015, October). Mobile app for public transport: a usability and user experience perspective. In International Internet of Things Summit (pp. 168–174). Springer, Cham. Keller, C., Titov, W., Sawilla, S., & Schlegel, T. (2019). Evaluation of a smart public
display in public transport. Mensch und Computer 2019-Workshopband. Laugwitz, B., Held, T., & Schrepp, M. (2008, November). Construction and evaluation of a user experience questionnaire. In Symposium of the Austrian HCI and usability engineering group (pp. 63–76). Springer, Berlin, Heidelberg. Schrepp, M., Hinderks, A., & Thomaschewski, J. (2014, June). Applying the user
experience questionnaire (UEQ) in different evaluation scenarios. In International Conference of Design, User Experience, and Usability (pp. 383–392). Springer, Cham. Stopka, U. (2014, June). Identification of user requirements for mobile applications to support door-to-door mobility in public transport. In International Conference on Human-Computer Interaction (pp. 513–524). Springer, Cham.
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-44-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional