Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

I don’t understand you – error-handling as a key aspect of conversational design for chatbots

University of Applied Sciences for Administration and Service Rehmkamp 10, 24161 Altenholz, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe100878

Abstract

Misunderstandings can occur in any dialogue. This applies to communication between people just as it does to communication between humans and chatbots. Resolving these misunderstandings and continuing the dialogue effectively is called error handling. Good error handling makes conversations with chatbots more helpful, convincing and successful and has great impact on user experience. The aim of this paper is to contribute to successful chatbot implementation projects. To this end, the paper will focus on the process of planning and implementing chatbot projects, with a particular emphasis on human-centred design (i.e. ISO 9241). A special role is played by the conversational design aspects with special consideration of how error handling should look like.Generalised, it can be said that an analysis for the start of a chatbot project should run (at least) through the following phases: Stakeholders and Value Proposition, Current solutions and channels, Context and conversational tasks, Personality of the chatbot, Background tasks, Error handling and fallbacks.In the field of public services, chatbot systems are becoming more and more common in public administration contexts. Compared to business use cases, where conversion rates are the ultimate KPI, here the benefits are more towards 24/7 support services, availability and scalability. For the conversational design, users should be able to enjoy talking to the virtual assistant. Hence the dialogue design must be adapted to a (defined) target group depending on the context and area of application. The chatbot is a (virtual) representative of a company or administrative body, and it therefore represents it in conversations with user or customers. Consequently, the language of the chatbot should be adapted to the Coroporate Identity, values and language.Challenges and opportunities of chatbot design, especially looking at the user experience, give you on the one hand a great potential for engaging with users at scale. On the other hand there are (potential) usability problems, i.e. users need to read a lot, can’t just easily scan and skip to their relevant content and probably a limited ability of chatbot systems to understand natural language.So what is the cause of possible misunderstandings between humans and chatbots? There can be various reasons, apart from a lack of the aforementioned understanding. The natural language processing in terms of AI comprises of two important aspects: detection of (a users’) intent as well as entities. In that context the error or misunderstanding occurs when a chatbot does not understand a request or assigns the wrong concern to the request. Other reasons might be that the chatbot knows the user's concern but has not understood the wording correctly, or the chatbot has not (yet) learned a concern that is actually relevant. Otherwise the concern might not belong to the chatbot's area of expertise.So what are the options for dealing with unknown questions and misunderstandings? A often used approach is that the chatbot describes the misunderstanding and asks the user to formulate their question differently. This is called a fallback answer. It often looks like: "I'm sorry, I didn't understand you. Can you describe your request again in other words?". Recommended alternatives to this standard behaviour might be, that the chatbot says that it may not yet have learned anything about this issue and offers the user a selection of topics that it has already mastered. Or that the chatbot refers to additional information (i.e. websites) where information on a recognised keyword can be found. Even more convenient is an intelligent search as a background function that provides relevant entries to the question directly in the chat. Last exit can be an offer to forward to a human employee (e.g. callback service).

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

INTRODUCTION

In over 50 years (the very first text chatbots were Eliza at MIT, 1966 and Parry at Stanford, 1972) , the evolution of chatbots has progressed to the point where different variants of voice-assisted assistants shape our everyday lives. If we follow the lines of development, we can say that chatbots have existed for decades, but have not been widely adopted. However, a revolution only occurred with the launch of Apple's Siri on the iPhone 4s (Siri was introduced by Apple on 4 October 2011 in a product presentation of the iPhone 4s). Whereas previously the function of a communication interface was fulfilled by chatbots, a variety of user commands could now be realised directly through the virtual voice assistant Siri - including performing phone actions, scheduling events, checking basic information, reminders, handling device settings, navigating areas, searching the Internet as well as engaging with iOS-integrated applications.

The revolutionary maturation of artificial intelligence (driven i.e. by IBM Watson winning Jeopardy 2011, among others) in terms of natural language processing did the rest. Amazon introduced Alexa (Initial release 2013 as Echo smart speaker) where Google presented Dialog Flow (2017, became later part of Google Cloud Platform) as a natural language understanding platform for conversational user interfaces for web apps, devices, interactive voice response systems and the like. The essential components for conversation design were thus available and are now well established: natural language understanding as well as dialog management.

Today, a large number of leading companies use chatbots to increase their efficien- cy. Communication channels can be offered 24/7 and can be directly connected through frameworks - from Facebook to Twitter, Skype, Cortana, Alexa or Slack and many more (Chatbot statistics 2021 – landbot.io).

As an example of use in the public sector, in 2019 the first pilot applications were implemented in Germany's northernmost federal state, Schleswig-Holstein (chatbot Ina for the Integration Office of the Ministry of Social Affairs, Integrationsamt 2021). In addition, chatbots and digital voice assistants were defined as important fields of application in the federal state’s digitalisation programme (The Prime Minster of Schleswig-Holstein 2021) as well as the AI strategy. For further expansion, it was decided to set up a state infrastructure for chatbots in Schleswig-Holstein (based on the popular RASA open source framework for chat and voice-based virtual assistants, see https://rasa.com/). So if we assume that chatbots will find wider use in businesses and the public sector, it becomes clear that good conversational design will play an essential role.

The aim of this paper is to contribute to the user centered design of chatbots with a special focus on conversational design and error handling. The concrete motivation for the authors is to analyse the preliminary steps of an ongoing project and to plan and, if necessary, optimise its further implementation. The methodological approach

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 is combining Design Science Research (Hevner et. al. 2004, Peffers et. al. 2007) and Information System Development (Nunamaker et. al. 1990). The structure is as follows: first we present our case study background and requirements. Next, we discuss process models and recommendations of practice in literature, especially from the context of Human Centred Design, Conversational Design and Error Handling. We close with a conclusion and an outlook on the next steps in our project and give the direction of our further research approach.

A CHATBOT AS CONTINUING EDUCATION ADVISOR FOR PUBLIC ADMINISTRATION EMPLOYEES The background for this paper is provided by a current research and development project of the authors (Digital Competence Navigator). As part of the development of a continuing education portal on digital competencies for administrative employees, three access points to courses will be implemented: A detailed search function for courses, individual matchmaking after an anonymous competence test as well as a self-assessment through a dialogue option with a chatbot. Herein, the chatbot is in- tended to be the central interface for presenting the portal and its functions and ex- plaining all questions about it. In addition, structured chatbot dialogues should lead users to individual course recommendations.

Compared to existing chatbot projects within the context of administration in Schleswig-Holstein, it should be noted that a large and diverse target group (all employees of the public administration at federal state and municipal level) is addressed whereas the thematic scope is also quite large.

Generalised, it can be said that at the start of each chatbot project an analysis of the following aspects should be considered: stakeholders and value proposition, current solutions and channels, context and conversational tasks, personality of the chatbot, background tasks, error handling and fallbacks (Deibel and Evanhoe 2021, Hathaway and Hathaway 2021).

In the field of public services, chatbot systems are becoming more and more com- mon in public administration contexts. Compared to business use cases, where con- version rates count as the ultimate KPI, here the benefits are more towards 24/7 sup- port services, availability and scalability (Chatbot statistics 2021 – landbot.io).

From conversational design perspective, users should be able to enjoy talking to the virtual assistant. Hence, the dialogue design must be adapted to a (defined) target group depending on the context and area of application. The chatbot is a (virtual) representative of a company or administrative body, and therefore represents them in conversations with users or customers. Consequently, the language of the chatbot

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 should be adapted to the corporate identity, values and language (Chatbot statistics 2021 – landbot.io, Deibel and Evanhoe 2021, Hathaway and Hathaway 2021).

Challenges and opportunities of chatbot design, especially looking at the user experience, give you on the one hand a great potential for engaging with users at scale. On the other hand, there are (potential) usability problems, i.e. users need to read a lot and can’t just easily scan and skip to their relevant content.

With regard to chatbot types, the presented example relies on the aforementioned federal state’s infrastructure and allows for AI or NLP based chatbots (as opposed to simpler, rule-based chatbots). The underlying technology of machine learning allows the chatbot to learn from user requests, in order to become more and more knowl- edgeable and intelligent. It builds upon the open-source conversational AI stack RASA and integrates the chatbot’s content management-platform botario (see https://botario.com/en/), in order to provide automatic processing of requests.

USER-CENTRICITY, CONVERSATIONAL DESIGN AND ERROR HANDLING After an initial literature review, the question can be asked, whether a discipline like Conversational UX Design is already established as a distinct discipline . However, it can be said that Conversational Design falls under the umbrella of UX Design and shares common processes, principles and techniques. Accordingly, a number of principles and guidelines for the development of chatbots can be considered (Hoehn 2019, Moore and Arar 2019, Shevat 2017, Sugisaki and Bleiker 2020).

The basic understanding of the general UX Design Process (for systems) also applies here. The fundamental steps (cp. ISO 9241-210:2019) within an iteration cycle are “understanding & specifying context of use”, “specifying user requirements”, “production of design solutions” and “evaluating design solutions against requirements”. Well-established evaluation methods for the last step are thereby Usability Testing and Heuristic Evaluation.

Ren et. al. (2019) aimed to identify the state-of-the-art in chatbots usability and the applied Human–Computer Interaction (HCI) techniques. They analysed the state-of- the-art of usability in the development of chatbots, and collected usability methods for evaluating overall chatbot usability. According to this, the most frequently applied research methods were quantitative surveys (and qualitative interviews), with a specific focus on the following parameters: “[…] among efficiency, effectiveness and satisfaction, satisfaction is the one that has been evaluated more frequently. User satisfaction is directly impacted by chatbot usability and hence it is considered as the primary measure in most studies.” (Ren et. al. 2019).

On the other hand, within an administrative project in Schleswig-Holstein at the be- ginning of the year for a municipal vehicle registration office, the “correctness” or

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

“expected conformity of answers” from a chatbot was used as a measure of success (which could be increased from 69% to almost 90% in about 4 weeks – Petersen 2021).

Höhn and Bongard-Blanchy (2021) proposed “12 Heuristics for Conversational UX Analysis” based on Nielsen’s ten heuristics for User Interface Design (Nielsen 2005) and elaborated that “this evaluation framework is applicable to chatbots of different conversation styles”. Table 1 gives an overview of Heuristics and assigned Sub- heuristic.

Table 1: 12 Heuristics for Conversational UX Analysis [17] (Folds et al. 2008)
Original PDF Table 1:

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

CONCLUSIONS AND OUTLOOK The literature research has yielded several essential results for us and our chatbots project. We can clearly confirm that prototyping conversational interfaces is different or “stranger” compared to GUI-based interactions and more iteration cycles are recommendable in development (Deibel and Evanhoe 2021).

We have also found that we need to optimise our project plan in terms of chatbot training and associated iteration cycles (we will extend this project phase by approx. 6 weeks, as found in Petersen 2021). We will try to involve as large a number of test users (at least 100+) as possible in order to provide for a good training base.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Finally, we will apply the Heuristics for Conversational UX Analysis to our proto- types and first development increments. We assume that we have found a good work- ing basis here, which also represents comparative values for a different application context.

Since we have not yet found much standardisation for the UX design of chatbots, we also want to incorporate our approach and templates for this project and future research into the digital design system (ITVSH 2021) of the municipal IT Association of the federal state of Schleswig-Holstein.

REFERENCES

Chatbot Statistics 2021 (March 2021): State of the Market & Opportunities, Website: https://landbot.io/blog/chatbot-statistics-compilation Deibel, D., Evanhoe, R.: Conversations with things – UX Design for Chat and

Voice, Rosenfeld, New York 2021 Google Assistant (November 18, 2021) Conversation Design – Errors - Google Developer, Website: https://developers.google.com/assistant/conversation-design/errors Hathaway, T., Hathaway, A.: Chatting with Humans: User Experience Design (UX)

for Chatbots, BA-Experts, 2021 Hevner, A.R.; March, S.T.; Park, J.; Ram, S. (2004): Design Science in Information Sys-tems Research. In: MIS Quarterly (Vol. 28 No. 1), S. 75–105. Hoehn, S.: Artificial Companion for Second Language Conversation. Springer

(2019) Hoehn, S., Bongard-Blanchy, K. (2021): Heuristic Evaluation of COVID-19 Chatbots, In: Følstad A. et al. (eds) Chatbot Research and Design. CONVERSATIONS 2020. Lecture Notes in Computer Science, vol 12604. Springer, Cham. https://doi.org/10.1007/978-3-030-68288-0_9 IBM Conversational UX Design CHI 2017 Workshop, Website:

https://researcher.watson.ibm.com/researcher/view_group.php?id=7539 Integrationsamt (November 18, 2021) Chatbot Ina - new digital assistance of the Integration Office (Chatbot Ina - neue digitale Assistenz des Integrationsamtes), Website: https://www.schleswig- holstein.de/DE/Landesregierung/IAMT/Projekte/teilhabe_Chatbot_Fachinhalt. html ISO 9241-210:2019, Ergonomics of human-system interaction — Part 210: Human-

centred design for interactive systems, Website: https://www.iso.org/standard/77520.html ITVSH (November 2021) Design System | ITVSH - Design System, Website: https://www.design-system.sh/ Moore, R.J., Arar, R.: Conversational UX Design: A Practitioner’s Guide to the

Natural Conversation Framework. ACM Books (2019)

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Nielsen, J. (2001): Error Message Guidelines, Website: https://www.nngroup.com/articles/error-message-guidelines/ Nielsen, J. (2005): Ten usability heuristics, Website:

http://www.nngroup.com/articles/ten-usability-heuristics/ Nunamaker, J.F.; Chen, M.; Purdin, T.D.M. (1990): Systems Development in Information Systems Research. In: Journal of Management Information Systems 7 (3), S. 89–106. Peffers , K., Tuunanen, T. , Rothenberger, M.A. & Chatterjee, S. (2007): A Design

Science Research Methodology for Information Systems Research, Journal of Management Information Systems, 24:3, 45-77, DOI: 10.2753/MIS0742- 1222240302 Petersen, L., Höhn, T., A. Höhn (2021): Chatbots: Digital assistants for citizencentred administration (Chatbots: Digitale Assistenten der bürgernahen Verwaltung) In: Die Gemeinde – Zeitschrift für die kommunale Selbstverwaltung Schleswig-Holstein, 04/2021, pp 96-100, ISSN 0340-3653 Ren, R., Castro, J.W., Acuna, S.T. and de Lara, J. (2019): Evaluation Techniques

for Chatbot Usability: A Systematic Mapping Study, International Journal of Software Engineering and Knowledge Engineering, Vol. 29, Nos. 11&12 (2019) 1673–1702, DOI: 10.1142/S0218194019400163 Shevat, A. (2017): Designing Bots: Creating Conversational Experiences. O’Reilly Sugisaki, K., Bleiker, A. (2020): Usability guidelines and evaluation criteria for conversational user interfaces: a heuristic and linguistic approach. In: Proceedings of the Conference on Mensch und Computer. pp. 309–319 The Prime Minister of the State of Schleswig-Holstein (Der Ministerpräsident des

Landes Schleswig-Holstein), Digitalisierungsprogramm 2021/2022 (March 2021) Website: https://www.schleswig- holstein.de/DE/Fachinhalte/D/Digitalisierung/ExterneLinks/digitalisierungspro gramm2122.pdf?__blob=publicationFile&v=1

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8989-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →