STARTING POINT AND RESEARCH QUESTIONS The intention of the EU project “Career Intelligence”(Kröll & Burova-Keßler, 2022) is to further develop the learning platform “Career 4.0”, which has been developed and successfully tested throughout Europe to promote young people’s entrepreneurial and digital skills, into an adaptive learning environ- ment with the help of an AI-based learning assistant. The project is being funded by the EU for 2.5 years. The central aim of the “Career 4.0” learn- ing platform is to support young people in developing sustainable career prospects based on their respective life situations, skills and personal interests. In addition to the development and specification of a personal development plan, this also involves the creation of innovative services that can be pursued by those affected, if required and interested.
In this context, it is crucial to build on the research findings on adap- tive learning recommendations and the associated comprehensive discussion
© 2024. Published by AHFE Open Access. All rights reserved. 114
AI-Based Learning Recommendations - Possibilities and Limitations 115
about the possibilities and limitations of using AI-based solutions in the edu- cation sector. The integration of data and modelling serves as a key tool for analyzing and improving learning processes (Kerres & Buntins, 2020). Due to the wide variety of different professional and educational biographies, individualization is playing an increasingly important role and this increases the need to develop, establish and specifically promote individual learning paths (Cook & Artino Jr., 2016). It should be borne in mind that the demand for shaping the digital transformation in organizations and economic sectors leads to new requirements in terms of necessary skills in the world of work and corresponding qualification efforts. The organization and establishment of feedback processes is key to initiating corresponding learning processes and enabling learning success. The Botty learning assistant developed in the EU “Career Intelligence” project has the task of stimulating and promoting such feedback processes. Ensuring the quality of these feedback processes is of particular relevance (Hattie & Donoghue, 2016). The collaboration among the virtual learning assistant, the mentors and the learning tasks in the learn- ing platform is intended to promote the digital and entrepreneurial skills as well as the reflection skills of young people.
Using the “Career 4.0” learning platform as an example, the current find- ings on AI-based learning recommendations are taken up and examined. The aim is to explore the extent to which this offers a possible solution for pro- moting the learning process and where its limitations may lie. The following questions are central in this context: What are the advantages and challenges of using a virtual learning assistant, e.g. to promote interaction between stakeholders using a learning platform? What needs to be considered when developing dialogs that are carried out with the help of the virtual learning assistant? What design principles can be used in this context? How can the quality of the dialogs be guaranteed?
The answers to these questions were and are used in the EU project “Career Intelligence” to design the interaction between the young people and the vir- tual learning assistant in a way that promotes learning. Ultimately, the aim is to professionalize young people’s personal development plans.
THEORETICAL FRAME OF REFERENCE In the current state of research on AI-based solutions in the education sec- tor, the adaptivity of learning systems is emphasized as a possible potential, especially with regard to learning recommendations (e.g. Bäsler and Sasaki, 2020; Biel et al., 2019). This includes (a) making learning opportunities more conducive to learning, (b) recording and evaluating learning processes and results using AI-based solutions, (c) providing personalised recommen- dations for the learner, (d) enabling the further development of the relevant skills and (e) increasing the probability of achieving the learning objectives (Bäsler & Sasaki, 2020; Biel et al., 2019). In addition, it is possible to refer to the concept of “machine learning” in the context of promoting the learning process. This concept creates the advantage that current learning processes can be monitored and adjustments can be made in real time. The latter is also referred to as “adaptive feedback”. This can take place at both personal
116 Kröll and Burova-Keßler
and task level (Ninaus & Sailer, 2022). In addition, publications such as Seufert et al. (2020) emphasize the benefits of using chatbots in terms of learning success. One basis for this is the use of virtual learning platforms, such as the Career 4.0 learning platform (Kröll & Burova-Keßler, 2023). However, the type of interaction on the learning platform is crucial for a positive result.
In the “Career Intelligence” project, the interaction of young people (mentees) in the context of the learning platform is to be improved by devel- oping a virtual learning assistant. Particular attention is paid to the dialogs and recommendations of the virtual learning assistant. To ensure that these dialogues are accepted, it makes sense to actively involve young people in the development of the dialogues.
According to Bäsler and Sasaki (2020), the use of these new didactic meth- ods can be described as a “didactic shift”. As a result, teachers are taking on a new role in the learning process, as they are increasingly supporting the young people’s learning process as guides and coaches. However, the aim is not to replace teachers, but to relieve the burden on teaching staff and improve the quality of teaching and learning processes (Seufert et al., 2020). Since learning platforms such as “Career 4.0”are primarily aimed at self-regulated learning, the focus of the learners also changes in this case. The young people them- selves can select learning paths or learning sprints through which they gain further knowledge. The self-selected learning paths offer the opportunity to achieve better learning outcomes, as empirical studies show (Bäsler & Sasaki, 2020). The following two theories represent a further element of the theo- retical frame of reference: Self-determination theory and the expectation- disconfirmation paradigm. In the self-determination theory according to Deci and Ryan (1990), the feeling of autonomy is central. The aim is to create the feeling that autonomous decisions can be made, with a corresponding sense of responsibility and effectiveness at the centre (Vansteenkiste et al., 2004). The expectation-disconfirmation paradigm, on the other hand, pos- tulates that user satisfaction is a central predictor of the intention to continue using the system (Liao et. al., 2009). In this context, reference is also made to the model of that places user satisfaction in relation to perceived per- formance and fulfilment of previous expectations (Bhattachcheriee, 2001; Oliver, 1980).
In the “Career Intelligence” project, the development of the “Career 4.0” skills model will form the basis for personalized learning recommendations by linking learning units in the learning platform with the skills promoted. The personal development plan is another starting point for personalized learning recommendations and a tool for skills management.
THE USE OF AI TECHNOLOGIES - BENEFITS AND CHALLENGES In the following chapter, the advantages of the “Career 4.0”learning platform and the potential of using chatbots in the learning process are discussed first. This is followed by a description of the challenges involved in using chatbots. In addition, the chapter concludes with a concrete explanation of the virtual learning assistant “Botty” and its possibilities.
AI-Based Learning Recommendations - Possibilities and Limitations 117
With regard to the advantages of the “Career 4.0”learning platform, inter- nationalization can be mentioned on the one hand, which means that it is a multilingual platform with the best practices of different countries. This gives young people from one EU country the opportunity to work together with mentors from another EU country. On the other hand, self-organized learning and the possibility of promoting this learning play a superordinate role. The (further) development of the personal development plan and various tools for self-organization, which are available in the learning platform, are important tools here. The learning sprints promote both self-organised and collaborative learning. With regard to the latter point, the mentees and men- tors are provided with tools for teamwork, for example. The mentors are also responsible for ensuring that the participants receive constructive feedback from them and from other experts. However, the mentees’ self-assessments also play a special role here. The structure of the platform offers mentees a high degree of flexibility. It also creates the opportunity for mentees to ini- tiate their own projects and to obtain support from mentors and/or experts, for example, if required.
With regard to the advantages of chatbots in the learning process, it can be seen that communication with the chatbot offers various potentials, but also challenges. The chatbot creates the possibility of temporal and spatial flexibility for the user, whereby supportive learning environments can be selected and real-time information can be provided. In addition, multimodal- ity is possible in the transfer of information and context-related conversation initiation.
The challenge here is the discrepancy between user expectations and the actual experience. The fears of potential users regarding the use of chatbots must also be taken into account. In some cases, this leads to a refusal to use the chatbot. With regard to challenges in dealing with chatbots during the learning process, the following further aspects are central: users only have limited flexibility to control the interaction themselves. In addition, the vir- tual learning assistant’s ability to adapt to user habits is limited. There may also be a loss of control over personal data (Annamalai et al., 2023; Diederich et al., 2022; Seymour et al., 2018; Yang and Aurisicchio, 2021). In addition, a lack of knowledge about the functions of virtual learning assistants can make users feel insecure. Last but not least, the problem can arise that the virtual learning assistant only offers the user information of low quality. At the same time, there is a risk that the user loses control over what contextual informa- tion has been retrieved by the learning assistant (Chiu et al., 2023; Yang and Aurisicchio, 2021). The above-mentioned points can lead to frustration, dis- appointment, cognitive stress and low commitment on the part of the user - according to the results of the scientific publications mentioned above. In addition, mistrust can occur and the use of AI can lead to a significant increase in data protection risk.
Despite these challenges, the use of a virtual learning assistant offers many advantages in the specific case of using the “Career 4.0” learning platform, which are discussed below. First of all, a central goal of using the virtual learning assistant is to intensify the interaction between the potential users
118 Kröll and Burova-Keßler
(in this case: the young people who are dealing with their career orienta- tion, the mentors and experts as well as the organizers of corresponding activities) and the learning platform. The promotion of interaction is seen as a decisive game changer for the use of a learning platform and the pos- sibilities for learning with the help of the learning platform. In addition, the virtual learning assistant Botty can specifically help users familiarise themselves with the functions of the “Career 4.0” learning platform by pro- viding context-related answers to frequently asked questions. This leads to an improvement in the user-friendliness of the learning platform. In addition, the virtual learning assistant supports users in a dialogue-oriented man- ner in completing the various aspects of their personal development plan. Another function is that the virtual learning assistant gives recommenda- tions on learning sprints and tasks in the learning platform to promote the individual competence development of the users. The virtual learning assistant can also evaluate the user’s learning processes and achievement of goals. The learning assistant can support the user in pursuing their learning strategy. The previous points make it clear that the virtual learning assis- tant can contribute to the quality of learning activities within the learning platform.
EMPIRICAL APPROACH-RESEARCH METHODOLOGY The research methodology of the EU project is based on a comprehensive, iterative approach that is based on numerous feedback loops. To this end, several consecutive workshops, expert discussions and interviews were con- ducted. The aim of this approach was to enable a well-founded scientific analysis of the research questions and tasks of the EU project. As part of the four-phase research design in the form of a mixed methods design, both qualitative and quantitative research methods were used. After a two- day introductory workshop, two three-day teaching-learning activities (LLA) were carried out. Workshop participants were the members of the project teams from the EU countries (Hungary, Greece, Finland, Bulgaria and Ger- many) as well as experts from the respective EU countries. The potential users of the learning platform and the virtual learning assistant, i.e. the young peo- ple and the mentors, were also involved in the development process. Their feedback is considered crucial as they are the ones who will later interact with the virtual learning assistant and/or possibly take up the virtual learn- ing assistant’s recommendations or break off the dialogue with the virtual learning assistant for whatever reason. In this context, a test phase was con- ducted in Hungary, Bulgaria, Greece and Finland and, among other things, the dialogues with the virtual learning assistant and the chatbot persona itself were evaluated. The results of the feedback phase were then documented and evaluated and taken up in the further development process, including the dialogues.
The development of criteria that are of particular importance for the design of learning recommendations within the framework of the “Career 4.0” platform was another focus of the workshop series. The aim was to work out and assess the most important criteria for the development of learning
AI-Based Learning Recommendations - Possibilities and Limitations 119
recommendations that are addressed to users with the help of the virtual learning assistant. The learning recommendations related to the learning units available in the Career 4.0 learning platform.
The research methodology was supplemented by further activities: on the one hand, existing research approaches to the use of AI-based solutions in the education sector (e.g. CBR, learning analytics, adaptive learning systems) were taken up. In addition, networking took place with “good practice” projects that work with the use of AI in education (“BMBF project BeSt F:IT”, “AI Coach VICI”, “SMART LEARNING TAXES”, etc.). Last but not least, a potential resistance analysis on the use of AI-based tools in the learning plat- form was carried out as an empirical study. In addition, the project is based on both a process-orientated and a participatory approach. Cross-functional development teams were established to model the central processes in the learning platform. The exchange with experts from projects dealing with the use of AI-based in the education sector as well as the experts of the project partners was also promoted.
Rasa technology was used to develop the virtual learning assistant. The main function of this open source technology is to ensure natural language processing, extract entities and create a dialogue flow design. The basis for the development and establishment of recommendations was the open source technology “My CBR”. Based on an experience-based approach and with reference to previously successful solutions for similar problems, the learning assistant can be enabled to give personalised recommendations with regard to learning sprints and learning tasks.
RESEARCH RESULTS The current research results with regard to the development of learning recommendations are presented below.
a) Reasons for discontinuing the learning process During the expert workshops in the five EU countries (Bulgaria, Germany, Finland, Greece and Hungary), the following assessments were made regard- ing possible reasons for dropout, both in connection with chatbots in general and with the virtual learning assistant and the learning platform: (1) Topic is difficult to understand (dropout can be encouraged by a topic that is too com- plex or difficult to understand), (2) Topic is boring/uninteresting (the learner quickly loses interest if the topic is not captivating), (3) Unclear learning pro- cess (the learner can quickly lose interest in the learning process if the topic is not captivating).), (4) Topic is boring/uninteresting (The learner quickly loses interest in the learning process if the topic is not engaging), (5) Unclear learning process (Demotivation can be increased by a lack of understand- ing of how the learning process is structured and how progress is made), (6) Unfulfilled expectations (Premature termination can occur, (7) Feedback too general/too broad (Learner may not feel sufficiently supported if they receive unspecific or unhelpful feedback), (8) Topic is not relevant (There is a lack of motivation to continue if the learning content is perceived as not relevant to
120 Kröll and Burova-Keßler
their own goals or interests), (9) No trust between mentor and mentee (Moti- vation to learn can be affected by a lack of trust in the relationship between learner and teacher), (10) Demotivating feedback (Motivation to learn can be significantly reduced by negative or demotivating feedback), (11) Lack of transparency about lessons learned (The learner may lose sight of the goal if there is a lack of clarity about what is being gained from the learning process, (12) Content too text-heavy (It can be tiring for the learner if there is an exces- sive amount of text material without variety), (13) Learners find the process difficult (Learners can become frustrated if there are obstacles in the learning process, such as lack of resources or poor teaching materials), (14) Learning takes too long (There is a risk that motivation will wane if the learning pro- cess is perceived as too time-consuming.), (15) Learning needs are not taken into account (It can lead to dissatisfaction if individual learning needs and styles are not taken into account) and (16) Underestimation of own com- petencies (The learning process can be terminated prematurely if there is a feeling of being overwhelmed). Based on these results, a questionnaire with quantitative and qualitative questions was developed in the further course of the project to record resistance and potential when communicating with the chatbot “Botty”. A comprehensive study is to be carried out on this basis.
b) Criteria for the development of recommendations The central aim of a second series of workshops with experts from the five EU countries was to work out criteria for the development of learning recommendations by the chatbot “Botty” and to ensure the quality of the learning recommendations. The focus was on finding out which criteria users consider to be particularly important when it comes to the design of learning unit recommendations by the virtual learning assistant Botty. The aspects of personal talents, personal learning goals, fit of the learning sprint to the mentee, personal interests, goals of the learning unit and the language of the learning unit (DE, EN, IT, ES, EL, BE and HU) were given very high priority by the respondents. The criteria, age, proximity of the learning sprint to the search terms, competence metadata of the learning sprints, format (self- study/group work) and the input from the personal development plan were rated as moderately relevant. In contrast, the aspects of the mentee’s learning style, duration of the learning sprint and gender tended to be given a low priority.
In addition, the criteria described were assessed in terms of how they are captured by the learning platform (automatic vs. interactive). The result was that all criteria that were given a high priority can also be recorded as interac- tive, i.e. through interaction with the virtual learning assistant. In the second category of moderate relevance, only the aspects “proximity of the learning sprint to search terms”, “competence metadata of the learning sprints” and “input from the personal development plan” were assigned to the automatic type. The other criteria were assigned to the interactive type. With regard to the low priority category, all criteria were also rated as interactive. This again shows the importance of interaction with the virtual learning assistant.
AI-Based Learning Recommendations - Possibilities and Limitations 121
This result also makes it clear that research into potential and resistance can help to promote interaction between young people and the learning assis- tant. The results of the series of workshops with the experts represent the first step in an iterative cycle for recommending learning units. The aim is to further evaluate the results in the future course of the EU project and to coordinate them with the target group of young people.
c) Design principles and quality criteria for the digital learning assistant In the scientific debate, various principles are pointed out in relation to the design of virtual learning assistants (Strohmann et. al., 2022): One is the human likeness of the persona of the virtual learning assistant. The aim here is for the virtual learning assistant to have a humorous character, for example, to use emojis when necessary and to admit mistakes. It is also considered advantageous if the dialog takes place at eye level and using language suitable for young people. In general, it is advantageous if there is an open atmosphere in the conversation and the language and appearance of the avatar adapt to the different countries and recognize the different roles. In addition, the behavior of the virtual learning assistant should be both reactive (answering questions) and proactive (obtaining feedback). In addition, the aim is for the bot to build a relationship as a supportive assistant, which is particularly encouraged by the appropriate level of communication. It makes sense for the virtual learning assistant Botty to be open (i.e. on an equal footing), friendly, trustworthy and fresh, i.e. entertaining and relaxed. Overall, it makes sense for the learning content to have a connection to the user’s everyday working life. These quality criteria were developed in 5 expert workshops in different countries, expert feedback and from the pre-test phases.
The development of a personal development plan (PDP) is a particular focus. The bot introduces the young people to the process and helps them to complete a rudimentary PDP. Based on the PDP, the bot makes recom- mendations for learning sprints and learning tasks. Self-organized learning takes place through the development of a successful learning strategy and can be promoted by encouraging self-reflection. Based on this, the personal motivation profile is also supported in order to set appropriate and functional motivational incentives (e.g. networking with peers). Another important task of Botty is to explain the functions and the transparency of data usage.
OUTLOOK In the next step, the EU project “Carreer Intelligence” deals with the fur- ther development of learning recommendations that are suggested by the virtual learning assistant. In this context, the following further research questions arise: What content-related, methodological-didactic and technical conditions should be met so that the virtual learning assistant can provide recommendations that promote learning? What points or design principles need to be observed in order to develop beneficial learning recommendations? What criteria can be used to ensure the quality of learning recommendations?
For future research projects, it is worth investigating the extent to which AI-based solutions (in particular learning recommendations provided by the
122 Kröll and Burova-Keßler
virtual learning assistant) represent a beneficial aid for mentees and mentors. It is particularly relevant to recognise dropout triggers and to develop strate- gies for coping with them in the next step. In this way, premature terminations can be minimised.
ACKNOWLEDGMENT
The “Career Intelligence” project is funded by the EU. This article builds on German research by Kröll, Burova-Keßler & Fischer (2024).
REFERENCES
Annamalai, N.; Eltahir, M. E.; Zyoud, S. H.; Soundrarajan, D.; Zakarneh, B.; Al Salhi, N. R. Exploring English language learning via Chabot: A case study from a self determination theory perspective. Comput. Educ. Artif. Intell. 2023, 5. https://doi.org/10.1016/j.caeai.2023.100148 Bäsler, S. A. & Sasaki, F. (2020). Interaktive Lernmedien: Gestaltung von digi-
talen Bildungsmedien mit Künstlicher Intelligenz. Information-Wissenschaft & Praxis, 71(1), 39–42. https://doi.org/10.1515/iwp-2019-2059 Bhattacherjee, A. (2001). Understanding information system continuance: An expectation-confirmation model. MIS Quarterly, 25(3), 351–370. Biel, C., Brandt, P. & Hellmich, C. (2019). Lern-Empfehlungen von der Mas-
chine. weiter bilden, 2019(04), 22–25. Chiu, T. K. F., Moorhouse, B. L., Chai, C. S., & Ismailov, M. (2023). Teacher support and student motivation to learn with Artificial Intelligence (AI) based chatbot. Interactive Learning Environments, Advanced online publication. https://doi.org/ 10.1080/10494820.2023.2172044. Cook, D. A., & Artino Jr, A. R. (2016). Motivation to learn: an overview of
contemporary theories. Medical education, 50(10), 997–1014. Cooperate Learning Trends & Innovationen (2020). Praxiseinsatz Künstliche Intelligenz - Wie lernen wir mit Maschinen?. Deci, Edward L.; Ryan, Richard M.: Die Selbstbestimmungstheorie der Motivation
und ihre Bedeutung für die Pädagogik - In: Zeitschrift für Pädagogik 39 (1993) 2, S. 223–238 - URN: urn: nbn: de:0111-pedocs-111739 - DOI: 10.25656/01:11173. Diederich, S., Brendel, A., Morana, S., & Kolbe, L. (2022). On the design of and interaction with conversational agents: an organizing and assessing review of human-computer interaction research. Journal of the Association for Information Systems, 23(1), 96–138. https://doi.org/10.17705/1jais.00724 Digel, S., Krause, T. & Biel, C. (2023). Enabling Individualized and Adaptive Learn-
ing – The Value of an AI-Based Recommender System for Users of Adult and Continuing Education Platforms. In N. Wang, G. Rebolledo-Mendez, V. Dim- itrova, N. Matsuda & O. C. Santos (Hrsg.), Communications in Computer and Information Science. Bd. 1831, 797–803. Springer. Drachsler, H., Verbert, K., Santos, O. C. & Manouselis, N. (2015). Panorama of recommender systems to support learning. Recommender systems handbook, 421–451. Hattie, J. A., & Donoghue, G. M. (2016). Learning strategies: A synthesis and
conceptual model. npj Science of Learning, 1(1), 1–13. Kerres, M. & Buntins, K. (2020). Recommender in AI-enhanced learning: An assessment from the perspective of instructional design. Open Education Studies, 2(1), 101–111.
AI-Based Learning Recommendations - Possibilities and Limitations 123
Kröll, M. & Burova-Keßler, K. (2022). Use of AI tools in learning platforms and the role of feedback for learning. In: Nazir, S. et al. (Eds.): Advances in Human Factors in Training, Education, and Learning Sciences. Springer. Kröll, M. & Burova-Keßler, K. (2023): KI-Tools und Feedbackprozesse in der
beruflichen Bildung. In: GfA, Sankt Augustin (Hrsg.): Nachhaltig Arbeiten und Lernen, Frühjahrskongress 2023, Hannover, C.9.8. Kröll, M. & Burova-Keßler, K. (2024). KI-basierte Lernempfehlun- gen – Möglichkeiten und Grenzen. GfA, Sankt Augustin (Hrsg.): Frühjahrskongress 2024, Stuttgart “Arbeitswissenschaft in-the-loop: Mensch- Technologie-Integration und ihre Auswirkung auf Mensch, Arbeit und Arbeitsgestaltung”. Liao, Chechen & Palvia, Prashant & Chen, Jian-Liang. (2009). Information technology adoption behavior life cycle: Toward a Technology Continuance Theory (TCT). International Journal of Information Management - INT J INFORM MANAGE. 29. 309–320. 10.1016/j.ijinfomgt.2009.03.004. Ninaus, M. & Sailer, M. (2022). Zwischen Mensch und Maschine: Kün- stliche Intelligenz zur Förderung von Lernprozessen. Lernen und Lernstörungen. https://doi.org/10.1024/2235-0977/a000386. Oliver, R. L. (1980). A cognitive model for the antecedents and consequences of
satisfaction. Journal of Marketing Research, 17, 460–469. Seufert, S., Guggemos, J. & Sonderegger, S. (2020). Digitale Transformation der Hochschullehre: Augmentationsstrategien für den Einsatz von Data Analytics und Künstlicher Intelligenz. Zeitschrift für Hochschulentwicklung, 15(1), 81–101. Seymour, M., Riemer, K., & Kay, J. (2018). Actors, avatars and agents: Potentials and
implications of natural face technology for the creation of realistic visual presence. Journal of the Association for Information Systems, 19. Strohmann, T., Siemon, D., Khosrawi-Rad, B. & Robra-Bissantz, S. (2022).
Toward a design theory for virtual companionship. Human-Computer Interaction. 10.1080/07370024.2022.2084620. Terblanche, N. (2020). A design framework to create Artificial Intelligence Coaches. International Journal of Evidence Based Coaching & Mentoring, 18(2). https://doi.org/10.24384/b7gs-3h05. Vansteenkiste, M., Simons, J., Lens, W., Sheldon, K. M., & Deci, E. L. (2004).
Motivating learning, performance, and persistence: the synergistic effects of intrin- sic goal contents and autonomy-supportive contexts. Journal of personality and social psychology, 87(2), 246. Yang, Xi & Aurisicchio, Marco. (2021). Designing Conversational Agents: A Self- Determination Theory Approach. 10.1145/3411764.3445445.