Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Evaluation of Online Education Service System Based on User Emotional Valence

Shaanxi University of Science and Technology, Shannxi, Xi’an 710016, China ↗
Publication year
Digital Object Identifier10.54941/ahfe1001734

Abstract

In the post-epidemic era, online education has developed rapidly. In order to improve the experience of online education services, this research uses user emotional valence as an evaluation variable to explore the evaluation methods of online education service systems. First, use the interactive contact matrix to deconstruct the system service interface, and set the evaluation indicators of the online education service system from the user and the user, the user and the system. Secondly, taking Tencent Classroom as the research object, based on the user's emotional evaluation, an online education service system evaluation experiment is designed. Taking learners' emotional pleasure, arousal, dominance, satisfaction and importance as measurement indicators, the multi-dimensional evaluation of the online education service system is completed. Finally, an online education service system evaluation model is constructed to provide a basis for system iterative optimization. At the same time, it provides effective ideas and feasible suggestions for enhancing the interest of online education services and users' learning initiative and enthusiasm.

INTRODUCTION

As a new normal in the post-epidemic era, online education is manifested in that users learn knowledge, improve skills, and broaden their horizons through the Internet platform. It is an opportunity and focus for the transfor- mation of traditional education models (Lee, 2010). Online education service evaluation is a process of deconstructing, analyzing and optimizing the online education system, which plays a decisive role in enhancing users’ learning effi- ciency and learning interest (Pawan et al., 2003). Since the online education service process involves many types of elements such as platform interface, learning resources, and operating users, how to effectively sort out and evalu- ate service elements, lock key opportunities and output corresponding design strategies is the key and prerequisite for improving online education services (Waldner et al 2012).

At present, domestic and foreign scholars’ evaluation research on online education services is mainly divided into two categories: subjective evaluation and objective evaluation. Among them, subjective evaluation mainly relies on the experience of education industry experts or designers for platform

© 2022. Published by AHFE Open Access. All rights reserved. 395

396 He and Song

structure and optimization, which is prone to large errors. Therefore, the research mostly adopts the objective evaluation method, which is divided into two aspects: system analysis and user research.

(1) System analysis mainly focuses on the analysis of platform interface design and functional modules. It mainly includes the rationality analysis of website structure, platform interactive experience design, online education model research and online education platform virtual space analysis (Chen, et al 2020; Benigno; Trentin, 2000).

(2) User research mainly focuses on the research on user behavior and user psychological changes. The research methods involve flow theory, distributed cognitive perspective, user collaborative participation design, gamification design method, user behavior chain, situation analysis and so on. Compared with system analysis, user research pays more attention to the change of user needs and behavior changes, so it is more flexible (Huiping, Aisen 2019; Heyang. 2020).

Although the above-mentioned research on online education services effe- ctively makes up for the shortcomings of subjective evaluation methods, it improves the objectivity of service evaluation research under certain conditi- ons. However, they are all based on user behavior or explicit needs, and lack research on users’ emotional and psychological needs. Based on this, this paper proposes an online education service evaluation method based on user emotional valence. Try to combine psychological research with service evalu- ation, and explore the relationship between service evaluation elements and user emotions from different dimensions, so as to output service strategies in a more in-depth and detailed manner.

ESTABLISHMENT OF RESEARCH PROCESS FOR ONLINE EDUCATION SERVICE EVALUATION BASED ON USER EMOTIONAL VALENCE Research on online education service evaluation based on user emotional valence follows the basic principles of user research and service evaluation, and takes user emotion as the core element and entry point of the service system. Compared with other online education service evaluation methods, the research is not limited to user behavior and user needs. Instead, it combi- nes user pleasure, arousal, and dominance with satisfaction and importance through emotional analysis and psychological exploration of users. Finally, a comprehensive evaluation model is constructed to realize the evaluation and optimization of the service touchpoints of the online education platform.The research process of online education service evaluation based on user emo- tional valence is shown in Figure 1, which can be divided into the following three steps:

(1) Build a service evaluation index system. Firstly, use the relevant theory of service design to analyze user behavior, collect service touchpoints in the target system and output the interactive contact matrix. Secondly, the obtai- ned service contacts are subjected to cluster analysis and transformed into corresponding service evaluation indicators. Finally, through expert analysis and user research, a reasonable evaluation index system is output.

Evaluation of Online Education Service System Based on User Emotional Valence 397

Figure 1: Research process of online education service evaluation.
Original PDF Figure 1:

(2) An experimental study of emotional evaluation was carried out. Select 20-30 online education platform users to conduct sentiment analysis and psychological exploration of the online education system respectively. On the one hand, with the help of the PAD emotional model, the emotional evaluation criteria are determined as three items of pleasure, arousal and dominance, and according to the evaluation index system, the correspon- ding questions are designed to complete the emotional analysis. On the other hand, with the help of satisfaction and importance questionnaires, a deeper psychological exploration of users is carried out, and users’ comprehensive evaluation of the current education system service indicators is obtained from the two aspects of user needs and user expectations.

(3) Complete the iterative optimization of the service system. Summarize and test the data obtained from the emotional evaluation experiment, build an online education system service evaluation model, and conduct a compre- hensive analysis of the corresponding service contacts based on the obtained data. Using comparative analysis and univariate analysis methods, the rela- tionship between user emotion, satisfaction and importance is explored, so as to propose effective iterative strategies for the current online educa- tion system, optimize user learning experience and improve their learning efficiency.

CONSTRUCTION OF ONLINE EDUCATION SERVICE EVALUATION INDEX SYSTEM

3.1 Service System Deconstruction and Contact Clustering

In the existing literature research, the establishment of the evaluation indi- cators of the online education service system is mainly considered from the perspective of platform design and expert analysis, and less analysis is car- ried out from the perspective of user experience and user behavior. This paper uses the service design research method, takes the platform user as the center, and constructs the interactive contact matrix of the online education plat- form according to the user behavior trajectory and system composition, as shown in Figure 2.

398 He and Song

Figure 2: Interactive contact matrix of online education platform.
Original PDF Figure 2:

3.2 Establishment of Service Evaluation Index System

In order to obtain more detailed service evaluation indicators, qualitative research methods such as data query, in-depth interviews, and group discus- sions are used to obtain service touchpoints and convert them into evaluation indicators, as shown in Table 1.

Establishment of Service Evaluation Index System In order to obtain more detailed service evaluation indicators, qualitative research methods such as data query, in-depth interviews, and group discus- sions are used to obtain service touchpoints and convert them into evaluation indicators, as shown in Table 1.

EXPERIMENTAL RESEARCH ON EMOTIONAL EVALUATION Evaluation of Experimental Design

Through a comprehensive investigation of factors such as cumulative down- loads, registered users, click-through rate, and functional integrity of existing online education platforms, Tencent Classroom was selected as the experi- mental target of this emotional evaluation. Twenty-five college students were selected as experimental subjects to complete the experimental research. The selected subjects were all users of Tencent Classroom and had a good under- standing and manipulation of the platform. The specific experimental scheme is shown in Table 2.

User Emotion Analysis The essence of sentiment analysis for online education service platform users is to seek relevant variables to measure user sentiment, so as to realize the quantitative expression of user sentiment. This paper introduces the PAD emotional model, a commonly used research tool in the field of psychology, and divides user emotions into three dimensions: pleasure (P), arousal (A), and dominance (D), which are measured separately. The process is as follows:

(1) Sentiment analysis questionnaire design. In order to enhance the rea- dability and validity of the questionnaire, the SAM image sentiment scale

Evaluation of Online Education Service System Based on User Emotional Valence 399

Table 1. Online education platform service evaluation index system.

Table available in the original PDF paper.

Target Project Indicator Indicator Explanation Number

T2

Visual guide

UI Design Interface color matching, icon design T1 Navigation Settings

Navigation bar location and course classification T4 Course Reminder Class schedule design, class reminders T3 AD Recommendation Platform course recommendation, carousel advertisement

User and System

Teacher courseware or PPT T6

Course Video Course quality and course types T5 Courseware Information

Learning Resources

T8

Bibliography Course related books T7 Extended Reading Course related articles or similar courses Exercises or cases involved in the lesson T9 After-school Material

T12

After-school Assessment

Unit Test Test after each unit T10 Homework Tests or exercises after each measure T11 Final Assessment Post-course assessments or final assignments Find Answer After-school exercises text answers T13 Student Notes In-class or after-class key records T14

Users and Users

T17

interactive feedback

Course Forum Course related discussion groups T15 Help Share Course link or course poster sharing T16 Homework Commentary

Homework accuracy and completion analysis T19 Teacher Q&A Teachers answer questions online T18 Questionnaire Course experience and satisfaction survey

Table 2. Evaluation experimental design.

Table available in the original PDF paper.

Evaluation Item Emotion Analysis Psychological Exploration evaluation System Tencent Classroom Tencent Classroom Evaluation Object College students College students Evaluation Method

PAD emotion model, SAM emotion scale

Satisfaction, Importance

Satisfaction questionnaire, AHP matrix Evaluation Content

Pleasure, Activation, Domi- nance Data Range 1-9 1-5, 0-1

was introduced to analyze user sentiment. Help the subjects to evaluate their emotions more quickly and intuitively, as shown in Figure 3.

(2) Output of emotional evaluation results. Twenty-five college students aged 20-25 were invited to conduct the index emotion test using the task method. The results are shown in Table 3.

400 He and Song

Figure 3: Legend of the online education sentiment evaluation questionnaire.
Original PDF Figure 3:
Table 3. Emotional measurement results.

Table available in the original PDF paper.

Number Pleasure Arousal Dominance Number Pleasure Arousal Dominance T1 6.14 5.00 7.00 T11 6.23 6.01 4.67 T2 5.05 5.26 6.57 T12 6.87 5.45 2.10 T3 5.29 5.57 2.81 T13 3.00 2.99 2.05 T4 2.05 3.00 2.10 T14 3.00 5.67 7.43 T5 7.24 6.86 7.05 T15 3.42 2.65 4.51 T6 3.52 4.62 2.95 T16 7.23 6.87 7.50 T7 3.81 3.52 6.71 T17 2.43 6.71 3.05 T8 3.10 2.82 5.76 T18 7.85 6.78 7.54 T9 5.67 6.70 7.00 T19 5.03 3.56 2.13 T10 4.54 6.83 3.51

Table 4. Satisfaction with online education service touchpoints.

Table available in the original PDF paper.

Number Satisfaction Number Satisfaction Number Satisfaction Number Satisfaction T1 3.24 T6 1.86 T11 3.38 T16 4.10 T2 3.19 T7 1.92 T12 2.45 T17 1.57 T3 2.52 T8 1.83 T13 2.14 T18 2.16 T4 1.76 T9 3.65 T14 1.98 T19 2.65 T5 3.43 T10 3.54 T15 2.11

User Psychological Discovery (1) Touchpoint satisfaction study. Combined with 19 indicators, a satisfa- ction questionnaire was designed. The questionnaire used the Likert 5-level scale method, and 25 users from the previous experiment were invited to complete it. The average value of user satisfaction for each indicator is calcu- lated by the mean value method, and the calculation results are shown in Table 4.

Evaluation of Online Education Service System Based on User Emotional Valence 401

Table 5. Importance of online education service touchpoints.

Table available in the original PDF paper.

Number Importance Number Importance Number Importance Number Importance T1 0.0110 T6 0.1201 T11 0.0616 T16 0.0297 T2 0.0200 T7 0.0469 T12 0.0236 T17 0.1065 T3 0.0171 T8 0.0187 T13 0.0446 T18 0.1190 T4 0.0024 T9 0.0445 T14 0.0095 T19 0.0155 T5 0.2485 T10 0.0324 T15 0.0336

Figure 4: Online education system service evaluation model.
Original PDF Figure 4:

(2) Touchpoint importance study. Invite 5 college teachers to form an expert group, use the 1-9 evaluation scale to evaluate and score the 19 eva- luation elements of the online education platform in pairs, and take the arithmetic mean of the scores of each evaluation element as the final score. With the help of MATLAB software, the weight of each element is obtained, as shown in Table 5.

RESULTS AND DISCUSSION

Integration of Online Education Service System Evaluation Results Integrate the experimental results of emotional evaluation, and compare the user’s emotional pleasure, arousal, and dominance with satisfaction and importance, as shown in Table 6. In order to facilitate the observation of the correlation between different factors, a visualization method is used to pro- cess the obtained data to build an online education service evaluation model, as shown in Figure 4. Try to enhance the visualization degree of the chart. The model divides the importance index into 5 levels according to the weight, and uses different colors to distinguish each index, which is convenient for users to observe the correlation between the indicators, and provides convenience for evaluation results discussion and strategy output.

402 He and Song

Table 6. Comprehensive indicators of online education service evaluation.

Table available in the original PDF paper.

Number Pleasure Arousal Dominance Satisfaction Importance Importance Ranking T1 6.14 5.00 7.00 3.24 0.0110 18 T2 5.05 5.26 6.57 3.19 0.0200 13 T3 5.29 5.57 2.81 2.52 0.0171 15 T4 2.05 3.00 2.10 1.76 0.0024 19 T5 7.24 6.86 7.05 3.43 0.2485 1 T6 3.52 4.62 2.95 1.86 0.1201 3 T7 3.81 3.52 6.71 1.92 0.0469 6 T8 3.10 2.82 5.76 1.83 0.0187 14 T9 5.67 6.70 7.00 3.65 0.0445 8 T10 4.54 6.83 3.51 3.54 0.0324 10 T11 6.23 6.01 4.67 3.38 0.0616 5 T12 6.87 5.45 2.10 2.45 0.0236 12 T13 3.00 2.99 2.05 2.14 0.0446 7 T14 3.00 5.67 7.43 1.98 0.0095 17 T15 3.42 2.65 4.51 2.11 0.0336 9 T16 7.23 6.87 7.50 4.10 0.0297 11 T17 2.43 6.71 3.05 1.57 0.1065 4 T18 7.85 6.78 7.54 2.16 0.1190 2 T19 5.03 3.56 2.13 2.65 0.0155 16

Discussion on Evaluation Results of Online Education Service System By analyzing the changes of each index in the online education system service evaluation model, the following conclusions are drawn:

(1) In the process of online education service, the user’s emotional pleasure, arousal, and dominance are basically consistent with the change trend of user satisfaction, which is positively correlated. It can be seen that there is a positive relationship between user satisfaction and user emotional valence. Designers can start from user emotions and optimize the lower points of user emotions, so as to improve user satisfaction.

(2) Users pay the highest attention to the learning resource touchpoints, among which T5 (course videos), T6 (courseware materials), T7 (refere- nce lists), and T9 (after-school materials) are all important touchpoints, indicating that the quality of courses on the online education platform is com- parable to that of courses. The class experience is the primary factor affecting its service quality, and it is also the core touch point that attracts users to the system.The corresponding emotional valence and satisfaction values show that in the current system service contacts, only T5 (course videos) and T9 (after-class materials) have higher user emotional valence and satisfaction, while the remaining three touchpoints are in the below the median. There- fore, the subsequent iteration of the service system should firstly optimize the learning resource touchpoints to meet the needs of users to the greatest extent and improve user satisfaction.

(3) Users pay the least attention to visual guidance touchpoints. T1 (UI design) and T2 (navigation settings) have higher emotional valence and sati- sfaction, indicating that most users are satisfied with the current system

Evaluation of Online Education Service System Based on User Emotional Valence 403

interface design and visual navigation , which can be maintained in subsequ- ent designs. The user’s emotional pleasure and arousal degree of T3 (course reminder) are above the average, but the user’s emotional dominance and satisfaction are lower. This result shows that the current course reminder function makes users feel a strong sense of being manipulated, resulting in low user satisfaction. In the subsequent design, attention should be paid to reducing the compulsion of course reminders, and a more personalized and customizable way to remind users follow the progress of the course.

(4) T14 (student notes) and T18 (teacher answering questions) are special points and can be handled as design opportunity points. The user satisfaction and emotional pleasure of T14 are low, but the emotional activation and dominance are above the average, indicating that the note-taking function is easy to stimulate users’ learning initiative and activity. However, due to the imperfect function of the current system and poor experience, it needs to be optimized and improved in the subsequent design. The user’s emotional pleasure, arousal and dominance of T18 are all at a high level, but the user satisfaction is the lowest, indicating that the current online education plat- form teacher Q&A function module construction is insufficient, only some courses provide teacher Q&A services, and most of them are in the message reply mode . Therefore, users cannot solve the puzzles and difficulties encoun- tered in course learning point-to-point, resulting in poor user satisfaction. In terms of importance, T18>T14, so in the follow-up design, the teacher’s question-answering problem should be solved first, and college notes should be introduced as a new function to create platform differentiation.

CONCLUSION

In view of the fact that the current research on online education platform service evaluation is mostly based on expert evaluation or system analysis, and lack of consideration of user emotion and user psychology research, this paper proposes a research method for online education service evaluation based on user emotional valence. This method considers users and users, users and systems, and establishes an evaluation index system for online education services. Through experimental design, it evaluates the five dimensions of users’ emotional pleasure, arousal, dominance, satisfaction and importance. Ultimately, a more guided service optimization strategy is obtained.

This paper is an extension of the traditional service evaluation analysis method. Applying user sentiment analysis to service evaluation, obtaining the relationship between user emotional changes and satisfaction through experiments, and mining design opportunities are issues that need further research.

REFERENCES

Benigno, V., & Trentin, G. (2000). The evaluation of online courses. Journal of computer assisted learning, 16(3), 259–270. Bo C. (2011). An Analysis of Distance Education Network Platforms from the

Perspective of Distributed Cognition. Journal of Jiangsu Ocean University: Huma- nities and Social Sciences, 9(7), 3.

404 He and Song

Chen, T., Peng, L., Yin, X., Rong, J., Yang, J., & Cong, G. (2020, September).

Analysis of user satisfaction with online education platforms in China during the COVID-19 pandemic. In Healthcare (Vol. 8, No. 3, p. 200). Multidisciplinary Digital Publishing Institute. Heyang C. (2020). Research on the mooc model of light game-based information literacy based on situational stories. Library Science Research (14), 8. Huiping C., Aisen X. (2019). Construction and Application of Online Education

Website User Satisfaction Evaluation Index System. Chongqing Higher Education Research, 7(2), 87–96. Lee, J. W. (2010). Online support service quality, online learning acceptance, and student satisfaction. The internet and higher education, 13(4), 277–283. Mandernach, B. J., Donnelli, E., Dailey, A., & Schulte, M. (2005). A faculty eva-

luation model for online instructors: Mentoring and evaluation in the online classroom. Online Journal of Distance Learning Administration, 8(3), 1–10. Moreno-Ger, P., Burgos, D., Martínez-Ortiz, I., Sierra, J. L., & Fernández-Manjón, B.

(2008). Educational game design for online education. Computers in Human Behavior, 24(6), 2530–2540. Park, J. Y. (2011). Design education online: Learning delivery and evaluation.

International Journal of Art & Design Education, 30(2), 176–187. Pawan, F., Paulus, T. M., Yalcin, S., & Chang, C. F. (2003). Online learning: Patterns of engagement and interaction among in-service teachers. Language Learning & Technology, 7(3), 119–140. Qiangkun H.,Renke H. (2018). Research on interaction design of mobile learning

applications based on flow theory. Packaging Engineering, 39(4), 5. Waldner, L. S., Widener, M. C., & McGorry, S. Y. (2012). E-service learning: The evolution of service-learning to engage a growing online student population. Journal of Higher Education Outreach and Engagement, 16(2), 123–150.

Cover of the proceedings volume Usability and User Experience

Explore the complete digital volume

Usability and User Experience

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-15-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →