Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Determining the Relevance of Quality Dimensions Concerning Video Conferencing Applications Used in Educational Ecosystem

1 Juraj Dobrila University of Pula, Faculty of Informatics, Zagrebačka 30, 52100 Pula, Croatia ↗
2 Polytechnic of Rijeka, Trpimirova 2/V, 51000 Rijeka, Croatia ↗
Publication year
Digital Object Identifier10.54941/ahfe1001135

Abstract

With the emergence of the COVID-19 pandemic, video conferencing applications have become an indispensable asset of educational ecosystems worldwide. These applications nowadays serve as a backbone of all educational activities, including delivering lectures and practical lab sessions as well as conducting exams. It is therefore of great importance to identify to what extent they comply with the quality requirements. The objective of this paper is to examine the psychometric features of the measuring instrument designed for measuring quality in the context of applications used for delivering lectures online and to determine if there are any significant differences among them when quality attributes and items are considered. For this purpose, an empirical study was carried out in which a representative sample of users was composed of students from two Croatian higher education institutions. The research adopted a within-subjects design contrasting applications designed for video interaction and used in educational settings. Reported findings revealed which dimensions of quality are the most relevant in the context of applications meant for video meetings. Implications for both researchers and practitioners were presented and discussed.

INTRODUCTION

Video conferencing applications are the most relevant piece of software used for conducting educational activities during the global COVID-19 pandemic. By using video conferencing software, teachers and students can interact in many ways which makes them an effective synchronous teaching tool (Correia et al. 2020). However, it should be noted that video conferencing applications were not designed to be primarily used in the educational ecosystem (Zou et al. 2020). As a result of the sudden transition from contact to virtual teaching, a large number of educational institutions have chosen video conferencing applications for delivering lectures and conducting exams without considering any or a few quality criteria in that respect. Due to the cost, security issues, and privacy concerns with the use of video conferencing services such as Zoom and Microsoft Teams, some higher education institutions have implemented BigBlueButton, their open-source counterpart.

Many university teachers expressed their intention to continue using the video conferencing software for educational purposes in the post-COVID-19 era. Findings reported in (Nikou 2021) revealed that satisfaction and perceived usefulness are strong predictors of continuance intention to use video conferencing applications by university teachers.

The usability of video conferencing applications is considered to be an important aspect in perceiving the quality of the educational environment but it does not depend on the device from which it is accessed (Pal & Vanijja 2020). On the other hand, the quality of internet connection can affect students’ perception of the video conferencing software quality (Suga 2021). Students' experience in using video conferencing systems is found to be related to the overall success of synchronous e-learning (Rahayu 2020). Results of a study presented in (Ngo et al. 2020) indicate that, when the employment of video conferencing applications is considered, ease of use strongly affects effectiveness in learning and perceived usefulness than it contributes to students’ attitude towards use.

An empirical evaluation of video conferencing applications uncovered significant differences among Zoom, Microsoft Teams, and Jitsi in terms of performance variability and resource utilization (Bieringa et al. 2021). Although the same authors discovered memory leaks (Bieringa et al. 2021) in Zoom, students have not experienced any technical difficulties while using it (Sumarna et al. 2020) and were satisfied with its interface and functionalities such as screen sharing and lecture recording (Minhas et al. 2021). The outcome of a

Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 comparative study (Hwangbo & Kim 2020) has shown that MS Teams has excellent attractiveness, while Zoom has high usability, accessibility, and searchability. According to (Zou et al. 2020), the usability of Microsoft Teams, Cisco Webex, and Zoom is very good in general, but screen sharing is not intuitive enough for novice users. Results of usability evaluation conducted by (Correia et al. 2020) have shown that Microsoft Teams scored high on learnability, but its effectiveness and usefulness are unsatisfactory (e.g., navigation bars are sometimes complicated and confusing). On the other hand, the same study (Correia et al. 2020) revealed that Zoom can be tailored to users’ preferences (e.g., the meeting room can be personalized, system preferences and virtual backgrounds changed, participants’ activities controlled, etc.).

This paper aims to explore the psychometric characteristics of the post-use questionnaire designed for evaluating facets of quality in the context of video conferencing applications used in educational settings.

The remainder of the paper is structured as follows. The next section provides the research framework. The findings of the empirical study are reported in the third section. Conclusions are drawn in the last section.

EMPIRICAL STUDY Procedure. The study was carried out during the winter semester of the academic year 2020/2021. Before the research took place, all students have been using the video conferencing applications which were the subject of evaluation. FIPU (Juraj Dobrila University of Pula, Faculty of Informatics) students employed BigBlueButton which is integrated with learning management system Moodle and Zoom, whereas POLYRI (Polytechnic of Rijeka) students interacted with Adobe Connect that is implemented within learning management system Merlin and Microsoft Teams. The post-use questionnaire was administrated through Google Forms. Participation in the study was voluntary and anonymous.

Framework. Drawing on our prior work on quality evaluation of different pieces of software (Orehovački et al. 2013, Orehovački & Babić 2018, Orehovački et al. 2019, Škorić et al. 2021, Orehovački et al. 2021), we identified 17 quality attributes and adapted their items to the context of this study. Attitude towards use indicates the extent to which students have a positive perception about the use of video conferencing applications in educational settings. Job relevance refers to the degree to which video conferencing applications are suitable for performing educational activities. Ease of use denotes the level to which the employment of video conferencing applications in the educational ecosystem is free of effort. Aesthetics represents the extent to which video conferencing applications applied for educational purposes have a visually appealing user interface. Usefulness indicates the degree to which video conferencing applications are beneficial in the context of conducting educational activities. Effectiveness refers to the level to which video conferencing applications enable

Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 students to execute educational activities accurately and completely. Efficiency denotes the degree to which video conferencing applications enable students to quickly perform educational activities. Outcome indicates the extent to which students believe that the quality of the educational process carried out using the video conferencing applications is sufficient. Reliability implies the degree to which video conferencing applications used for distance learning are bug-free. Task-technology fit refers to the level to which video conferencing applications support students in conducting educational activities. Compatibility denotes the degree to which video conferencing applications employed for distance learning operate properly on various devices and among different environments. Responsiveness signifies the level to which video conferencing applications used for educational purposes respond promptly to students’ actions. Anxiety indicates a psychological state caused by negative experiences or expectations to lose self-esteem when confronting a situation in which users must employ video conferencing applications in educational settings. Uniqueness implies the degree to which video conferencing applications used in the educational ecosystem are distinctive among applications with the same purpose. Confirmation indicates the extent to which video conferencing applications used for distance learning have met students’ expectations. Satisfaction refers to the degree to which users are pleased with the employment of video conferencing applications in educational settings. Loyalty signifies the degree to which students have the intention to continue to use the video conferencing ap- plications for educational purposes and recommend them to their peers. Overall quality indicates the level of comprehensive impression video conferencing applications used for distance learning have left on students.

Apparatus. Educational activities refer to all relevant students’ commitments including attending lectures, active participation in lab sessions, taking part in examinations, etc. The measuring instrument was composed of 10 items related to the demographics of study participants and the usage frequency of video conferencing applications. The second part of the post-use questionnaire was composed of 149 items dealing with the evaluation of 17 aforementioned quality attributes and one single score item designed for examining the overall quality in the context of video conferencing applications used for conducting educational activities. The responses to the items operationalized for measuring perceived quality were modulated on a five-point Likert scale (1 - strongly agree, 5 – strongly disagree). The sum of responses to items assigned to the particular attribute was used as a composite indicator that reflects the quality of video conferencing applications employed for distance learning as perceived by an individual student.

Research Design. Data was collected using the post-use questionnaire that was administrated through Google Forms. Considering the Shapiro–Wilk statistic significantly deviated from a normal distribution (p < .05) for at least one of the items in all conducted comparisons, the non-parametric tests were used for the analysis of data gathered from students. Consequently, all reported findings are expressed as median values. Differences between evaluated applications were explored with the Wilcoxon Signed-Rank Tests, the non-parametric alternative to the dependent t-test. The effect size, as an objective measure

Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 of the effect importance, was estimated by dividing Z-value by the square root of the number of observations. According to (Cohen, 1992), the values of .10, .30, and .50 indicate small, medium, and large effect sizes, respectively.

FINDINGS Participants. A total of 385 subjects took part in the study. Most of them (57.93%) were students at the Faculty of Informatics (FIPU), Juraj Dobrila University of Pula, among whom 71.6% were male and 27.4% female. The remaining 42.07% studied at the Polytechnic of Rijeka (POLYRI) among whom 58.6% were male and 41.4% female. The average age of the POLYRI students was 23.01 (SD=6.519), while FIPU students had 21.67 (SD=3.836) years on average. Most of the study participants in both groups were full-time students (FIPU=86%, POLYRI=76%). While all FIPU students were enrolled in the Informatics study program, in the POLYRI group of students, 77.2% of them studied Informatics and 22.8% of students were enrolled in the Entrepreneurship study program. The majority (87%) of FIPU students used BigBlueButton, while 52% of them used Zoom on daily basis during the winter semester of the 2020/2021 academic year. As much as 61.4% of FIPU students used the BigBlueButton to attend classes between 4 and 10 hours per week, while 46.2% of them used Zoom for the same purpose between 1 and 3 hours per week. The majority (77.2%) of POLYRI students employed Microsoft Teams, while 46.9% of them used Adobe Connect every day within the winter semester of the 2020/2021 academic year. Most POLYRI students (43.2%) used Microsoft Teams to participate in classes between 4 and 10 hours per week, while 35.8% of them used Adobe Connect for the same purpose between 1 and 3 hours per week.

Results. The findings of the study presented in Table 1 indicate that, according to the perception of FIPU students, Microsoft Teams has met the requirements of the identified quality attributes to a significantly greater extent when compared to Adobe Connect. More specifically, it appeared that the difference between these two video conferencing applications was large in size when the overall quality and requirements of the quality attribute loyalty were considered, small in the size difference between Adobe Connect and Microsoft Teams was found when requirements of the quality attribute anxiety were considered, while for all remaining 15 quality attributes determined difference in complying with associated requirements was medium in size.

Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0

Table 1: Perceived quality of video conferencing applications Adobe Connect and
Original PDF Table 1:

Microsoft Teams

On the other hand, we discovered that there was no significant difference between BigBlueButton and Zoom in terms of the degree to which these two video conferencing applications comply with requirements of 16 identified quality attributes nor in the case of indicator on overall quality, as perceived by FIPU students. The only significant difference that was discovered between these two video conferencing applications is related to the

Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 quality attribute responsiveness and was small in size. Findings on the perceived quality of BigBlueButton and Zoom are summarized in Table 2.

Table 2: Perceived quality of video conferencing applications Adobe Connect and
Original PDF Table 2:

Microsoft Teams

When Microsoft Teams and Zoom are considered, findings related to efficiency,

Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 effectiveness, satisfaction, and usefulness are in line with those reported in (Correia et al. 2020). Based on the overall difference that was found between evaluated video conferencing applications, the proposed set of attributes can be classified into five different groups: (1) essential attributes (loyalty, aesthetics, confirmation, and satisfaction) whose requirements video conferencing applications have to meet because otherwise their perceived quality will be significantly decreased; (2) sufficient attributes (job relevance, attitude towards use, ease of use, outcome, and reliability) which are also very important but failing to meet their requirements will be penalized less severe than in the case of essential attributes; (3) desired attributes (task-technology fit, usefulness, uniqueness, and responsiveness) whose relevance is significantly lower than those of previous two types of quality attributes but video conferencing applications still needs to comply with their requirements to some degree; (4) optional attributes (compatibility, efficiency, and effectiveness) that have similar role as desired ones, but their impact on perceived quality is lower than the those of desired attributes; and (5) not relevant attribute (anxiety) which can be but does not have to be considered when measuring perceived quality of video conferencing applications.

CONCLUSIONS

This paper aimed to determine to what extent the identified set of attributes is relevant for evaluating the perceived quality of video conferencing applications. For this purpose, an empirical study was carried out in which each of two groups of students used two video conferencing applications to perform various educational activities due to the COVID-19 pandemic. As an outcome of data analysis, we found that video conferencing applications employed by VELERI students differ significantly in every of 17 dimensions of perceived quality while in the case of video conferencing applications used by FIPU, a significant difference was discovered only when requirements of the quality attribute responsiveness were examined. Drawing on the study results, we classified the proposed set of quality attributes into five categories. Findings reported in this paper can be used by researchers as a foundation for conducting similar studies while practitioners can use it as a guideline when designing video conferencing applications. However, reported findings should be interpreted carefully due to the limitation of an empirical study dealing with the homogeneity of the sample of users. Participants in our study were students from two higher educational institutions. In that respect, other groups of stakeholders in the educational ecosystem (e.g., teachers, pupils, elementary and secondary school students) should be also involved in the study to make sound generalizable conclusions. Given that the findings presented in this paper are part of ongoing research, in our future work we are planning to examine the interplay among the proposed set of quality attributes.

Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0

REFERENCES

Bieringa, R., Radhakrishnan, A., Singh, T., Vos, S., Donkervliet, J., and Iosup, A. (2021), “An Empirical Evaluation of the Performance of Video Conferencing Systems.” Companion of the ACM/SPEC International Conference on Performance Engineering. Rennes: ACM, 65-71. Cohen, J. (1992), “A power primer.” Psychological Bulletin, 112 (1), 155–159. Correia, A-P., Liu, C., and Xu, F. (2020), “Evaluating video conferencing systems for the

quality of the educational experience.” Distance Education, 41(4), 429-452. Hwangbo, Y., and Kim, S. I. (2020), “A study on video conferencing program user experience-Focused on Zoom and MS Teams.” Journal of Digital Convergence, 19(2), 339-344. Minhas, S., Hussain, T., Ghani, A., and Sajid, K. (2021), “Exploring students online learning:

a study of Zoom application.” Gazi University Journal of Science, 34(2), 171-178. Ngo, T. T., Nguyen, T. T. T., and Tran, T. G. (2020), “Influence of Learning by Using Video Conferencing Tools on Perceptions and Attitude of Vietnamese Female Students in COVID-19 Pandemic”, http://dx.doi.org/10.2139/ssrn.3697029 Nikou, S. A. (2021), “Web-based video conferencing for teaching online: Continuance

intention to use in the post-COVID-19 period.” Interaction Design and Architecture, 47(Winter), 123-143. Orehovački, T., and Babić, S. (2018), “Identifying the relevance of quality dimensions contributing to universal access of social Web applications for collaborative writing on mobile devices: an empirical study.” Universal Access in the Information Society, 17(3), 453-473. Orehovački, T., Babić, S., and Etinger, D. (2021), “Perceived Quality of Software Used for

Distance Learning During COVID-19 Pandemic.” 44th International Convention on Information and Communication Technology, Electronics and Microelectronics. Opatija: IEEE, 856-861. Orehovački, T., Etinger, D., and Babić, S. (2019), “Modelling an interplay of adoption determinants with respect to social Web applications used in massive online open courses.” Universal Access in the Information Society, 18(3), 469-487. Orehovački, T., Granić, A., and Kermek, D. (2013), “Evaluating the perceived and estimated

quality in use of Web 2.0 applications.” Journal of Systems and Software, 86(12), 3039–3059. Pal, D., and Vanijja, V. (2020), “Perceived usability evaluation of Microsoft Teams as an online learning platform during COVID-19 using system usability scale and technology acceptance model in India.” Children and youth services review, 119 (December), https://doi.org/10.1016/j.childyouth.2020.105535. Rahayu, D. (2020), “Students' E-Learning Experience through a Synchronous Zoom Web

Con-ference System.” Journal of ELT Research: The Academic Journal of Studies in English Language Teaching and Learning, 5(1), 68-79. Suga, H. (2021), “A comparison of bandwidth consumption between proprietary web conference services and BigBlueButton, an open-source webinar system.” Bioresource Science Reports, 13, 1-11.

Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0

Sumarna, N., Kansil, Y. E., and Hamid, R. (2020), “The influence of online learning platform models during the COVID-19 outbreak on college student satisfaction levels in Southeast Sulawesi.” International webinar on education 2020, 72-81. Škorić, I., Orehovački, T., and Ivašić-Kos, M. (2021), “Task-Technology Fit and

Continuance of Use of Web-Based Programming Tool: A Pilot Study.” In W. Karwowski et al. (eds.). Human Systems Engineering and Design III. Cham: Springer, 57-62. Zou, C., Zhao, W., and Siau, K. (2020), “COVID-19 Pandemic: A Usability Study on Platforms to Support eLearning.” In C. Stephanidis et al. (eds.). HCI International 2020 – Late Breaking Posters. Cham: Springer, 333-340.

Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0

Cover of the proceedings volume Human Systems Engineering and Design (IHSED2021): Future Trends and Applications

Explore the complete digital volume

Human Systems Engineering and Design (IHSED2021): Future Trends and Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8987-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →