Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

An automated machine learning approach for early identification of at-risk maritime students

1 Department of Maritime Operations, University of South-Eastern Norway, 3184, Horten, Norway ↗
2 Bangladesh Marine Academy, Chattogram 4206, Bangladesh ↗
Publication year
Digital Object Identifier10.54941/ahfe1003150

Abstract

Machine Learning (ML) presents a significant opportunity for the field of education, including Maritime Education and Training (MET). Unfortunately, the benefits of ML have yet to be fully realized within MET. By integrating ML and Artificial Intelligence (AI)-powered methods into maritime education, institutions can better prepare future seafarers while providing accurate, state-of-the-art education tailored to individual student needs. Early identification of areas for improvement can help students and teachers enhance educational outcomes within MET. This study showcases the potential of automated ML (AutoML) platforms for predicting future performance as well as for identifying at-risk maritime students at the initial stages of their degree program. By enabling early identification, institutions can more efficiently plan and execute education strategies while building confidence in AutoML as a decision-aid among maritime stakeholders.

INTRODUCTION

Machine Learning (ML) and Artificial Intelligence (AI) have facilitated data- backed decision-support system for educators across all domains (Chen et al., 2020). It involves training models on data to make predictions or decisions without the need for human intervention. Nowadays ML methods are being used for objective analysis of learners’ performance (Alkadri et al., 2021), providing adaptive learning content to the students (Edwards et al., 2018), even psychological evaluation of learners (Liu et al., 2020) in a variety of contexts.

A novel application of ML and AI in education includes identifying stu- dents who may need extended supervision or are at-risk of dropout, i.e., iden- tification of at-risk students during the early stages of their studies (Chui et al., 2020; Waheed et al., 2020; Xing & Du, 2019). ML algorithms are a perfect fit for this purpose especially due to their inherent capacity to identify pat- terns and trends in students’ data along with making predictions based on the data (Chen et al., 2020). These characteristics enable ML algorithms to be used for monitoring students’ development and analysing the present and

© 2023. Published by AHFE Open Access. All rights reserved. 47

predicted performances in a specific learning context. However, there is a lack of literature employing these methods in the maritime education context.

Maritime education, or more specifically seafaring education is generally divided into two separate streams, i.e., engineering and navigation. Maritime Education and Training (MET) providers offer educational components to prospective seafarers satisfying the International Convention on Standards of Training, Certification and Watchkeeping for Seafarers (STCW, 2011). Depending on the country or administration, universities may offer mari- time degrees (e.g., Bachelor’s in Marine Engineering or Bachelor’s in Nautical Science etc.) that include relevant course components along with on-the-job training at sea and a few post-sea study components (IAMU, 2023). With the increasing demand for quality education as well as for workplace-relevance of seafarers’ training, it has become crucial to understand which factors affect the learning trajectory of seafarers during the early stages of their degree education.

This study demonstrates an application of ML enabling early prediction of specific courses that instructors at maritime institutes may use to identify areas where their students need improvement. By providing instructors with targeted insights into students’ strengths and weaknesses, this system can help them tailor their teaching approaches and offer additional support where necessary. Ultimately, this can help maritime degree students to succeed in their studies and prepare them for successful careers in the maritime industry.

RELATED STUDIES Studies related to the application of ML in differing educational contexts ranging from administration, instruction and learning have increased signifi- cantly over the past few years (Chen et al., 2020). A recent review by Luan & Tsai (2021) identified a wide spectrum of ML applications in educatio- nal context ranging from diagnosis (e.g., introverts, extroverts), prediction (e.g., dropout, performance), intervention (e.g., plan-making intervention, value-relevance intervention), prevention and recommendation (e.g., perso- nalized learning paths, learning contents). In addition, state-of-the-art AI tools have been employed in safety-critical industries, e.g., in surgical trai- ning for performance categorization as well as for real-time feedback (Alkadri et al., 2021; Yilmaz et al., 2022), in assessing mental workload during pilot training (Jiang et al., 2022) as well as a few prospective use cases in maritime simulator training (Tusher et al., 2022).

Type of Variables and Methods Used In learning context, varying ML algorithms are operationalized such as logi- stic regression, decision trees, Artificial Neural Networks (ANN), Support Vector Machines (SVM), Bayesian networks, gradient boosting machine, K-nearest neighbour along with other available methods (von Gerich et al., 2022). However, manual data pre-processing to utilize these algorithms require substantial computing and human resources if done manually. In such cases, automated machine learning (AutoML) is being utilized as a low-cost alternative where the steps of building the ML model such as data

An Automated Machine Learning Approach for Early Identification 49

pre-processing, feature engineering, model selection and tuning is automa- ted (Feurer et al., 2015; He et al., 2021). This method is already being used in a plethora of contexts which includes predicting academic performance (Tsiakmaki et al., 2021) and evaluation of simulator training exercises in healthcare domain (Smith et al., 2022).

Scope of This Study Recent literature reviews (2020-2022) related to the application of ML in differing educational contexts include identifying the influential factors to predict academic success (Alyahyan & Dustegor, 2020); students’ performa- nce prediction in e-learning platforms (Albreiki et al., 2021; Ouyang et al., 2022); various use cases of educational data mining in humanities and social science fields (Charitopoulos et al., 2020), in K-12 education (Du et al., 2020); identification of gaps and remedies of applying ML methods in per- formance prediction (Sekeroglu et al., 2021) as well as in exam preparation (Kaddoura et al., 2022). However, the lack of evidence in literature related to ML applications in maritime education context forms the scope of this study.

METHOD

This study employed an AutoML platform, DataRobot (datarobot.com) for identifying at-risk maritime students based on their early exam grades, i.e., to predict their final graduation grades based on their initial semester exam gra- des. DataRobot facilitates the analysis of data through a number of models and proposes the best model for deployment based on performance metrics.

Data Collection We collected a pseudo-anonymous dataset of all exam results of a 4-years Bachelor’s in Marine Engineering program consisting of a 3-phase, 08 seme- ster course plan, i.e., (a) phase 1: two-year pre-sea training (with 04 seme- sters), (b) phase 2: one-year on-the-job training at sea (with 02 semesters) and (c) phase 3: one-year post-sea study (with 02 semesters). Each student of this program takes a total of 45 different courses (see Figure 1) during the first two-year of phase 1 and receives a BSc in marine engineering degree with a final GPA (grade point average) after four (04) years upon the completion of phase 3. It is crucial to note for the analysis that the number of phase 3 gradu- ates in a given year is lower than the number of phase 1 graduates due to the staggered completion of required on-the-job training placements in phase 2.

A total of 416 samples of phase 1 graduated students and 137 samples of phase 3 graduated students from three (03) consecutive batches of a maritime institute are included in the analysis (see Table 1).

Model Performance Based on the properties of the data, 09 different ML models are operationa- lized by DataRobot. The best performing model was selected based on their RMSE (Root Mean Square Error) and R-Squared values of validation and cross-validation samples (see Table 2).

Figure 1: All pre-sea courses in phase 1 (total 04 semester).
Original PDF Figure 1:
Table 1. Details of students.

Table available in the original PDF paper.

Student batches

Number of phase 1 graduated students

Number of phase 3 graduated students A 156 39 B 119 62 C 141 36 Total 416 137

Table 2. Ranking of best performing ML model.

Table available in the original PDF paper.

Rank Model name RMSE R Squared validation cross validation validation cross validation

0.0702 0.0668 0.8956 0.8909

1 Light Gradient Boosting on Elastic-Net Predictions

0.0617 0.0697 0.9195 0.8784

2 Light Gradient Boosting on Elastic-Net Predictions

0.0642 0.0741 0.9129 0.8629

Mathematical equation reproduced from the original PDF
Equation reproduced from the original PDF.

In addition to the accuracy measures (e.g., RMSE, R-squared values), we refer to the lift chart (see Figure 2) to assess the model performance. The lift chart depicts actual and predicted final GPA scores in 10 bins. Each bin inclu- des an equal number of observations and shows prediction performance by comparing the average actual and predicted final GPA. Here, we see that the actual and predicted values for each bin fit well, indicating a good predictive performance of the selected models. We also observe that overfitting is not

An Automated Machine Learning Approach for Early Identification 51

Figure 2: Lift chart.
Original PDF Figure 2:

an issue in the model since the actual and predicted curves intersect four (04) times (see Figure 2).

Therefore, we utilized the model Light Gradient Boosting on ElasticNet Predictions for estimating the feature effects (i.e., semester courses) on the final GPA as well as differing inter-connected association of features.

Subsequently, the K-Means Clustering algorithm was utilized to analyze the data in an unsupervised manner, without any predetermined target vari- ables (e.g., final GPA). This approach allowed the identification of natural patterns and groupings within the data, which potentially provide useful insights into the students’ scores.

RESULTS

Identification of Feature Impact and At-Risk Students The analysis suggests that Engineering Watchkeeping (EWK_4) and Control Engineering-I (CE-I_4) from 4th semester, Basic Electro-Technology (BET_1) and Workshop Process & Materials (WMP_1) from 1st semester are top four (04) subjects in decreasing order of importance with the highest fea- ture effect on the final GPA. In addition, Applied Heat (AH_3) from 3rd

semester, English (ENG_1) from 1st semester, Applied Physics (AP-II_2) from 2nd semester and Applied Mechanics Sessional (AMCS_4) from the 4th seme- ster bear the least feature effect on the final GPA (see Figure 3). While the actual values in the figures are based on the given data, the predicted values are generated by the best performing ML model. Therefore, the tight coupling between the actual and the predicted values of the feature effect on the target (final GPA) depicts good model fit (see Figure 3b, 3c, 3d). On the other hand, partial dependence takes the effect of all features into account.

The K-Means Clustering revealed four (04) clusters where students scoring an average of 62.22, 61.86, 55.89 and 64.66 in EWK_4, in CE-I_4, in BET_1 and WPM_1, respectively, are at-risk of scoring the lowest in their final GPA (see Figure 4a, 4b, 4c & 4d).

Figure 3: Feature impact on final GPA (a) along with feature pair association for top three features (b, c and d).
Original PDF Figure 3:
Figure 4: Cluster analysis of top four (04) features [(a) EWK_4; (b) CE-I_4; (c) BET_1; (d) WPM_1] in four different clusters.
Original PDF Figure 4:

An Automated Machine Learning Approach for Early Identification 53

DISCUSSION

The study suggests that ML techniques hold promise for enhancing mari- time education. Specifically, the utilization of such tools may aid in the early detection of students who may struggle in their coursework and potentially achieve lower GPAs in their final assessments. This research carries significa- nce for both academic institutions and industry stakeholders in the maritime domain.

The results identified four (04) different courses (i.e., two courses from 4th semester and two courses from the 1st semester) which have the highest feature effect in the final GPA of a marine engineering degree program. In addition, the cluster analysis identified specific groups of students taking those four courses who may need extensive care at an early stage. Courses with the least feature effect indicate that a they hold less power in predicting the final GPA. It is important to note that a few courses are left out of analysis by the AutoML program either due to their duplicating effect or for minimal feature effect. Nevertheless, these courses could still be important; for exam- ple, the English course from 1st semester holds low feature effect (1.90%) in predicting overall GPA but it bears crucial significance for overall profes- sional success as seafarers (Shi & Fan, 2021). This instance conforms with the findings of Zawacki-Richter et al., (2019) highlighting the importance of human intervention and critical reflection in utilizing automated decision- support systems in education. Thus, the role of instructors in determining the learning strategies in maritime domain remain unscathed.

Considering the utility of ML in providing automated insights into the future performance of seafarers, maritime employers could use these tools to identify candidates with high potential for success in real workplaces. Our proposed model utilizing AutoML could assist industry stakeholders during on-campus recruitment or early selection of candidates for internships and employment opportunities. A deeper analysis may be performed using ML if data related to external factors such as age of students, their socio-economic status, past school grades etc. is available depending on the goal and scope of the investigation.

CONCLUSION

This study highlights the potential benefits of ML in detecting group of maritime students who may struggle with their coursework early in the stu- dies which might lead to a lower GPA in their final exam. The importance of critical reflection and human intervention are stressed along with other automated decision-support system in determining educational strategies.

Future studies could explore the potential of utilizing ML in assessing seafarers in differing MET contexts, e.g., in simulator training to predict their future performance in real workplaces. Additionally, exploring exter- nal factors that might be associated with seafarers’ education as well as their professional success holds potential value.

ACKNOWLEDGMENT

The first and the fourth author acknowledge the support of the European Union’s Horizon 2020 research and innovation programme under the Marie

Skłodowska-Curie grant agreement No 823904 (Project: ENHANCing Human Performance in Complex Socio-Technical SystEms, ENHANCE).

REFERENCES

Albreiki, B., Zaki, N., & Alashwal, H. (2021). A Systematic Literature Review of Student’ Performance Prediction Using Machine Learning Techniques. Education Sciences, 11(9), 552. https://doi.org/10.3390/educsci11090552. Alkadri, S., Ledwos, N., Mirchi, N., Reich, A., Yilmaz, R., Driscoll, M., & Del

Maestro, R. F. (2021). Utilizing a multilayer perceptron artificial neural network to assess a virtual reality surgical procedure. Computers in Biology and Medicine, 136, 104770. https://doi.org/10.1016/j.compbiomed.2021.104770. Alyahyan, E., & Dustegor, D. (2020). Predicting academic success in higher education: Literature review and best practices. International Journal of Educational Technology in Higher Education, 17(1), 3. https://doi.org/10.1186/s41239-020- 0177-7. Charitopoulos, A., Rangoussi, M., & Koulouriotis, D. (2020). On the Use of Soft

Computing Methods in Educational Data Mining and Learning Analytics Resea- rch: A Review of Years 2010–2018. International Journal of Artificial Intelligence in Education, 30(3), 371–430. https://doi.org/10.1007/s40593-020-00200-8. Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review.

Ieee Access, 8, 75264–75278. Chui, K. T., Fung, D. C. L., Lytras, M. D., & Lam, T. M. (2020). Predicting atrisk university students in a virtual learning environment via a machine learning algorithm. Computers in Human Behavior, 107, 105584. https://doi.org/10.1016/ j.chb.2018.06.032. Du, X., Yang, J., Hung, J.-L., & Shelton, B. (2020). Educational data mining: A

systematic review of research and emerging trends. Information Discovery and Delivery, 48(4), 225–236. https://doi.org/10.1108/IDD-09-2019-0070. Edwards, C., Edwards, A., Spence, P., & Lin, X. (2018). I, teacher: Using artificial intelligence (AI) and social robots in communication and instruction. COMMU- NICATION EDUCATION, 67(4), 473–480. https://doi.org/10.1080/03634523. 2018.1502459. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., & Hutter, F.

(2015). Efficient and robust automated machine learning. Advances in Neural Information Processing Systems, 28. He, X., Zhao, K., & Chu, X. (2021). AutoML: A survey of the state-of-the-art.

Knowledge-Based Systems, 212, 106622. https://doi.org/10.1016/j.knosys.2020. 106622. IAMU | International Association of Maritime Universities. (2023). https://iamu-edu .org/. Jiang, G., Chen, H., Wang, C., & Xue, P. (2022). Mental Workload Artifi-

cial Intelligence Assessment of Pilots’ EEG Based on Multi-Dimensional Data Fusion and LSTM with Attention Mechanism Model. International Journal of Pattern Recognition and Artificial Intelligence, 36(11). https://doi.org/10.1142/ S0218001422590352. Kaddoura, S., Popescu, D. E., & Hemanth, J. D. (2022). A systematic review on machine learning models for online learning and examination systems. Peerj Computer Science, 8, e986. https://doi.org/10.7717/peerj-cs.986. Liu, Y., Lan, Z., Cui, J., Krishnan, G., Sourina, O., Konovessis, D., Ang, H. E., &

Mueller-Wittig, W. (2020). Psychophysiological evaluation of seafarers to improve training in maritime virtual simulator. Advanced Engineering Informatics, 44, 101048. https://doi.org/10.1016/j.aei.2020.101048.

An Automated Machine Learning Approach for Early Identification 55

Luan, H., & Tsai, C. (2021). A Review of Using Machine Learning Approaches for Precision Education. EDUCATIONAL TECHNOLOGY & SOCIETY, 24(1), 250–266. Ouyang, F., Zheng, L., & Jiao, P. (2022). Artificial intelligence in online higher

education: A systematic review of empirical research from 2011 to 2020. Educa- tion and Information Technologies, 27(6), 7893–7925. https://doi.org/10.1007/ s10639-022-10925-9 Sekeroglu, B., Abiyev, R., Ilhan, A., Arslan, M., & Idoko, J. B. (2021). Systematic Literature Review on Machine Learning and Student Performance Prediction: Critical Gaps and Possible Remedies. Applied Sciences-Basel, 11(22), 10907. https://doi.org/10.3390/app112210907. Shi, J., & Fan, L. (2021). Investigating Teachers’ and Students’ Perceptions of

Online English Learning in a Maritime Context in China. Sage Open, 11(3), 21582440211040800. https://doi.org/10.1177/21582440211040800. Smith, R., Julian, D., & Dubin, A. (2022). Deep neural networks are effective tools for assessing performance during surgical training. Journal of Robotic Surgery, 16(3), 559–562. https://doi.org/10.1007/s11701-021-01284-7. STCW, I. (2011). International Convention on Standards of Training, Certifica-

tion and Watchkeeping for Seafarers, (STCW) 1978, as amended in 1995/2010. International Maritime Organisation, London, UK. Tsiakmaki, M., Kostopoulos, G., Kotsiantis, S., & Ragos, O. (2021). Fuzzy-based active learning for predicting student academic performance using autoML: A step-wise approach. Journal of Computing in Higher Education, 33(3), 635–667. https://doi.org/10.1007/s12528-021-09279-x. Tusher, H. M., Nazir, S., Mallam, S., & Munim, Z. H. (2022). Artificial Neural

Network (ANN) for Performance Assessment in Virtual Reality (VR) Simula- tors: From Surgical to Maritime Training. 2022 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), 0334–0338. https://doi.org/10.1109/IEEM55944.2022.9989816. von Gerich, H., Moen, H., Block, L. J., Chu, C. H., DeForest, H., Hobensack, M., Michalowski, M., Mitchell, J., Nibber, R., Olalia, M. A., Pruinelli, L., Ronquillo, C. E., Topaz, M., & Peltonen, L.-M. (2022). Artificial Intelligence -based techno- logies in nursing: A scoping literature review of the evidence. International Journal of Nursing Studies, 127, 104153. https://doi.org/10.1016/j.ijnurstu.2021.104153. Waheed, H., Hassan, S.-U., Aljohani, N. R., Hardman, J., Alelyani, S., & Nawaz,

R. (2020). Predicting academic performance of students from VLE big data using deep learning models. Computers in Human Behavior, 104, 106189. https://doi. org/10.1016/j.chb.2019.106189. Xing, W., & Du, D. (2019). Dropout Prediction in MOOCs: Using Deep Learning for Personalized Intervention. Journal of Educational Computing Research, 57(3), 547–570. https://doi.org/10.1177/0735633118757015. Yilmaz, R., Winkler-Schwartz, A., Mirchi, N., Reich, A., Christie, S., Tran, D. H.,

Ledwos, N., Fazlollahi, A. M., Santaguida, C., Sabbagh, A. J., Bajunaid, K., & Del Maestro, R. (2022). Continuous monitoring of surgical bimanual expertise using deep neural networks in virtual reality simulation. Npj Digital Medicine, 5(1), 54. https://doi.org/10.1038/s41746-022-00596-8. Zawacki-Richter, O., Marin, V., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education— Where are the educators? INTERNATIONAL JOURNAL OF EDUCATIONAL TECHNOLOGY IN HIGHER EDUCATION, 16(1). https://doi.org/10.1186/ s41239-019-0171-0.

Cover of the proceedings volume Training, Education, and Learning Sciences

Explore the complete digital volume

Training, Education, and Learning Sciences

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-85-8

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →