Comparing machine learning models for academic success prediction in online learning

Sofiia V. Velychko; Olena Y. Chepurna; Olena H. Fedorenko

doi:10.55056/cte.1576

Authors

Sofiia V. Velychko National University Odesa Law Academy https://orcid.org/0000-0002-2728-6796
Olena Y. Chepurna National University Odesa Law Academy https://orcid.org/0000-0002-1432-0799
Olena H. Fedorenko Donbas State Pedagogical University https://orcid.org/0000-0002-1897-874X

DOI:

https://doi.org/10.55056/cte.1576

Keywords:

learning analytics, academic success prediction, student risk, grouped cross-validation, online learning

Abstract

Early-warning models must balance the detection of unsuccessful outcomes with the workload of student support. We compare logistic regression, a decision tree, a random forest, and gradient boosting using 32,593 records from the Open University Learning Analytics dataset. Sixteen features are evaluated at 30, 60, and 90 days and over the full course using a student-grouped holdout and repeated grouped cross-validation. Gradient boosting has the highest cross-validation macro-F1; holdout macro-F1 is 0.706, 0.751, and 0.775 in the early windows. Assessment features provide substantial predictive information. Target risk recalls of 0.80-0.90 require flagging 48.3-73.3% of records with precision 0.541-0.662. These operating points are selected on holdout outcomes and require independent validation. Full-course performance is retrospective and uses a different population. Approximate statistical comparisons and unresolved feature-availability times limit inference. The results motivate a prospective evaluation of advisor support rather than the establishment of an effective deployment policy.

Downloads

Download data is not yet available.

Abstract views: 69 / PDF views: 22

References

Ahmed, W., Wani, M.A., Plawiak, P., Meshoul, S., Mahmoud, A. and Hammad, M., 2025. Machine learning-based academic performance prediction with explainability for enhanced decision-making in educational institutions. Scientific Reports, 15(1), 26879. Available from: https://doi.org/10.1038/s41598-025-12353-4. DOI: https://doi.org/10.1038/s41598-025-12353-4

Althibyani, H.A., 2024. Predicting student success in MOOCs: a comprehensive analysis using machine learning models. PeerJ Computer Science, 10, e2221. Available from: https://doi.org/10.7717/peerj-cs.2221. DOI: https://doi.org/10.7717/peerj-cs.2221

Bouckaert, R.R. and Frank, E., 2004. Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms. In: H. Dai, R. Srikant and C. Zhang, eds. Advances in Knowledge Discovery and Data Mining. Berlin, Heidelberg: Springer, Lecture Notes in Computer Science, vol. 3056, pp.3–12. Available from: https://doi.org/10.1007/978-3-540-24775-3_3. DOI: https://doi.org/10.1007/978-3-540-24775-3_3

Breiman, L., 2001. Random Forests. Machine Learning, 45(1), pp.5–32. Available from: https://doi.org/10.1023/A:1010933404324. DOI: https://doi.org/10.1023/A:1010933404324

Breiman, L., Friedman, J.H., Olshen, R.A. and Stone, C.J., 1984. Classification and Regression Trees. Belmont: Wadsworth. Available from: https://doi.org/10.1201/9781315139470. DOI: https://doi.org/10.1201/9781315139470

Dietterich, T.G., 1998. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation, 10(7), pp.1895–1923. Available from: https://doi.org/10.1162/089976698300017197. DOI: https://doi.org/10.1162/089976698300017197

Efron, B., 1979. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1), pp.1–26. Available from: https://doi.org/10.1214/aos/1176344552. DOI: https://doi.org/10.1214/aos/1176344552

Friedman, J.H., 2001. Greedy function approximation: a gradient boosting machine. The Annals of Statistics, 29(5), pp.1189–1232. Available from: https://doi.org/10.1214/aos/1013203451. DOI: https://doi.org/10.1214/aos/1013203451

Herodotou, C., Rienties, B., Verdin, B. and Boroowa, A., 2019. Predictive Learning Analytics ‘At Scale’: Guidelines to Successful Implementation in Higher Education. Journal of Learning Analytics, 6(1), pp.85–95. Available from: https://doi.org/10.18608/jla.2019.61.5. DOI: https://doi.org/10.18608/jla.2019.61.5

Holm, S., 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), pp.65–70. Available from: https://www.jstor.org/stable/4615733.

Hosmer, D.W., Lemeshow, S. and Sturdivant, R.X., 2013. Applied Logistic Regression, Wiley Series in Probability and Statistics. 3rd ed. Hoboken: Wiley. Available from: https://doi.org/10.1002/9781118548387. DOI: https://doi.org/10.1002/9781118548387

Kohavi, R., 1995. A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI 1995). vol. 2, pp.1137–1143. Available from: https://www.ijcai.org/Proceedings/95-2/Papers/016.pdf.

Kuzilek, J., Hlosta, M. and Zdrahal, Z., 2015. Open University Learning Analytics dataset. Available from: https://doi.org/10.24432/C5KK69.

Kuzilek, J., Hlosta, M. and Zdrahal, Z., 2017. Open university learning analytics dataset. Scientific data, 4(1), 170171. Available from: https://doi.org/10.1038/sdata.2017.171. DOI: https://doi.org/10.1038/sdata.2017.171

Lundberg, S.M. and Lee, S., 2017. A Unified Approach to Interpreting Model Predictions. In: I. Guyon, U. von Luxburg, S. Bengio, H.M. Wallach, R. Fergus, S.V.N. Vishwanathan and R. Garnett, eds. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pp.4765–4774. Available from: https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html.

McNemar, Q., 1947. Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika, 12(2), pp.153–157. Available from: https://doi.org/10.1007/BF02295996. DOI: https://doi.org/10.1007/BF02295996

Nadeau, C. and Bengio, Y., 2003. Inference for the Generalization Error. Machine Learning, 52(3), pp.239–281. Available from: https://doi.org/10.1023/A:1024068626366. DOI: https://doi.org/10.1023/A:1024068626366

Omarbekova, A., Nazyrova, A., Bekmanova, G., Lamasheva, Z., Turebayeva, R. and Abzal, Z., 2026. Comparative Evaluation of Machine Learning Models for Early Prediction of Student Academic Performance Using the Open University Learning Analytics Dataset. International Journal of Interactive Mobile Technologies (iJIM), 20(17), pp.110–136. Available from: https://doi.org/10.3991/ijim.v20i17.62574. DOI: https://doi.org/10.3991/ijim.v20i17.62574

Paul, R., Hussain, S., Kannan, K.D., Mondal, B.K., Baruah, A.K. and Gaftandzhieva, S., 2025. Interpreting the Machine Learning Approaches to Predict CGPA of the University Students. TEM Journal, 14(4), pp.3438–3447. Available from: https://doi.org/10.18421/TEM144-50. DOI: https://doi.org/10.18421/TEM144-50

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., VanderPlas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M. and Duchesnay, E., 2011. Scikit-learn: Machine Learning in Python. The Journal of Machine Learning Research, 12, pp.2825–2830. Available from: https://doi.org/10.5555/1953048.2078195.

Pronina, O.I. and Pertsev, Y.A., 2026. Comparative analysis of machine learning algorithms in decision support tasks for academic success. Reporter of the Priazovskyi State Technical University. Section: Technical sciences, 53(1), pp.61–68. Available from: https://doi.org/10.31498/2225-6733.53.1.2026.359776. DOI: https://doi.org/10.31498/2225-6733.53.1.2026.359776

Raj, N.S. and Renumol, V.G., 2022. Early prediction of student engagement in virtual learning environments using machine learning techniques. E-Learning and Digital Media, 19(6), pp.537–554. Available from: https://doi.org/10.1177/20427530221108027. DOI: https://doi.org/10.1177/20427530221108027

Tkach, V.S., 2025. Povedinkovi modeli otsiniuvannia uspishnosti studentiv u smart-klasakh [Behavioral models for assessing student success in smart classrooms]. Master’s thesis. National Technical University of Ukraine Igor Sikorsky Kyiv Polytechnic Institute, Kyiv, Ukraine. Available from: https://ela.kpi.ua/items/21594864-0f65-470b-81a7-59957a67a6cd.