Physics on autopilot: exploring the use of an AI assistant for independent problem-solving practice
DOI:
https://doi.org/10.55056/etq.671Keywords:
large language models, physics education, chatbots, artificial intelligence, problem-solvingAbstract
This study investigates the efficacy of large language model (LLM)-powered chatbots in guiding physics problem-solving, examining whether they can effectively supplement teacher-led learning. A customised chatbot was developed leveraging ChatGPT to provide step-by-step assistance through a structured problem-solving algorithm. Its impact was evaluated via an experimental study with 12th-grade physics students (N = 24) randomly assigned to a teacher-guided or chatbot-guided group for problem-solving practice. A Mann-Whitney U test revealed no significant differences in problem-solving competency between conditions. Qualitative analysis of conversational logs indicates the chatbot successfully emulated key teacher scaffolding behaviours. Our findings suggest AI tutors can deliver personalised, interactive support akin to human teachers, offering viable supplements to augment physics learning. Further research should explore optimising LLM training, human-chatbot balances, and impacts across diverse educational settings.
Downloads
References
Ausat, A.M.A., Massang, B., Efendi, M., Nofirman, N. and Riady, Y., 2023. Can Chat GPT Replace the Role of the Teacher in the Classroom: A Fundamental Analysis. Journal on Education, 5(4), pp.16100–16106. Available from: https://jonedu.org/index.php/joe/article/view/2745.
Barbas, M.P., Vieira, A.T. and Branco, P.D., 2024. The Importance of Chat GPT Training for Higher Education: Case Study. In: N. Martins and D. Brandão, eds. Advances in Design and Digital Communication IV. Cham: Springer Nature Switzerland, Springer Series in Design and Innovation, vol. 35, pp.695–705. Available from: https://doi.org/10.1007/978-3-031-47281-7_57. DOI: https://doi.org/10.1007/978-3-031-47281-7_57
Bitzenbauer, P., 2023. ChatGPT in physics education: A pilot study on easy-to-implement activities. Contemporary Educational Technology, 15(3), p.ep430. Available from: https://doi.org/10.30935/cedtech/13176. DOI: https://doi.org/10.30935/cedtech/13176
Budzianowski, P., Ultes, S., Su, P.H., Mrkšić, N., Wen, T.H., Casanueva, I., Rojas-Barahona, L.M. and Gašić, M., 2017. Sub-domain Modelling for Dialogue Management with Hierarchical Reinforcement Learning. In: K. Jokinen, M. Stede, D. DeVault and A. Louis, eds. Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. Saarbrücken, Germany: Association for Computational Linguistics, pp.86–92. Available from: https://doi.org/10.18653/v1/W17-5512. DOI: https://doi.org/10.18653/v1/W17-5512
Chi, M.T.H., 2005. Commonsense Conceptions of Emergent Processes: Why Some Misconceptions Are Robust. The Journal of the Learning Sciences, 14(2), pp.161–199. Available from: https://www.public.asu.edu/~mtchi/papers/EmergJLSpdf.pdf. DOI: https://doi.org/10.1207/s15327809jls1402_1
Docktor, J.L., Strand, N.E., Mestre, J.P. and Ross, B.H., 2015. Conceptual problem solving in high school physics. Phys. Rev. ST Phys. Educ. Res., 11, p.020106. Available from: https://doi.org/10.1103/PhysRevSTPER.11.020106. DOI: https://doi.org/10.1103/PhysRevSTPER.11.020106
Fuchs, K., 2023. Exploring the opportunities and challenges of NLP models in higher education: is Chat GPT a blessing or a curse? Frontiers in Education, 8. Available from: https://doi.org/10.3389/feduc.2023.1166682. DOI: https://doi.org/10.3389/feduc.2023.1166682
Gao, J., Galley, M. and Li, L., 2018. Neural Approaches to Conversational AI. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. New York, NY, USA: Association for Computing Machinery, SIGIR ’18, p.1371–1374. Available from: https://doi.org/10.1145/3209978.3210183. DOI: https://doi.org/10.1145/3209978.3210183
Hamaniuk, V.A., 2021. The potential of Large Language Models in language education. Educational Dimension, 5, p.208–210. Available from: https://doi.org/10.31812/ed.650. DOI: https://doi.org/10.31812/ed.650
Jinchuña Huallpa, J., Flores Arocutipa, J.P., Panduro, W.D., Chauca Huete, L., Flores Limo, F.A., Espinoza Herrera, E., Alba Callacna, R.A., Ariza Flores, V.A., Medina Romero, M. Ángel, Merino Quispe, I. and Hernández Hernández, F.A., 2023. Exploring the ethical considerations of using Chat GPT in university education. Periodicals of Engineering and Natural Sciences, 11(4), pp.105–115. Available from: http://pen.ius.edu.ba/index.php/pen/article/view/3770.
Kieser, F., Wulff, P., Kuhn, J. and Küchemann, S., 2023. Educational data augmentation in physics education research using ChatGPT. Phys. Rev. Phys. Educ. Res., 19, p.020150. Available from: https://doi.org/10.1103/PhysRevPhysEducRes.19.020150. DOI: https://doi.org/10.1103/PhysRevPhysEducRes.19.020150
Liang, Y., Zou, D., Xie, H. and Wang, F.L., 2023. Exploring the potential of using ChatGPT in physics education. Smart Learning Environments, 10(1), p.52. Available from: https://doi.org/10.1186/s40561-023-00273-7. DOI: https://doi.org/10.1186/s40561-023-00273-7
Logacheva, V., Malykh, V., Litinsky, A. and Burtsev, M., 2020. ConvAI2 Dataset of Non-goal-Oriented Human-to-Bot Dialogues. In: S. Escalera and R. Herbrich, eds. The NeurIPS ’18 Competition. Cham: Springer International Publishing, The Springer Series on Challenges in Machine Learning, pp.277–294. Available from: https://doi.org/10.1007/978-3-030-29135-8_11. DOI: https://doi.org/10.1007/978-3-030-29135-8_11
Markel, J.M., Opferman, S.G., Landay, J.A. and Piech, C., 2023. GPTeach: Interactive TA Training with GPT-Based Students. Proceedings of the Tenth ACM Conference on Learning @ Scale. New York, NY, USA: Association for Computing Machinery, L@S ’23, p.226–236. Available from: https://doi.org/10.1145/3573051.3593393. DOI: https://doi.org/10.1145/3573051.3593393
Masche, J. and Le, N.T., 2018. A Review of Technologies for Conversational Systems. In: N.T. Le, T. van Do, N.T. Nguyen and H.A.L. Thi, eds. Advanced Computational Methods for Knowledge Engineering. Cham: Springer International Publishing, Advances in Intelligent Systems and Computing, vol. 629, pp.212–225. Available from: https://doi.org/10.1007/978-3-319-61911-8_19. DOI: https://doi.org/10.1007/978-3-319-61911-8_19
Mestre, J.P., 2002. Probing adults’ conceptual understanding and transfer of learning via problem posing. Journal of Applied Developmental Psychology, 23(1), p.9–50. Available from: https://doi.org/10.1016/s0193-3973(01)00101-0. DOI: https://doi.org/10.1016/S0193-3973(01)00101-0
Nkambou, R., Bourdeau, J. and Mizoguchi, R., eds, 2010. Advances in Intelligent Tutoring Systems, Studies in Computational Intelligence, vol. 308. Berlin, Heidelberg: Springer Berlin Heidelberg. Available from: https://doi.org/10.1007/978-3-642-14363-2_1. DOI: https://doi.org/10.1007/978-3-642-14363-2
Oranga, J., 2023. Benefits of artificial intelligence (ChatGPT) in education and learning: Is Chat GPT helpful? International Review of Practical Innovation, Technology and Green Energy (IRPITAGE), 3(3), p.46–50. Available from: https://radjapublika.com/index.php/IRPITAGE/article/view/1250.
Prananta, A.W., Megahati S, R.R.P., Susanto, N. and Raule, J.H., 2023. Transforming Education and Learning through Chat GPT: A Systematic Literature Review. Jurnal Penelitian Pendidikan IPA, 9(11), p.1031–1037. Available from: https://doi.org/10.29303/jppipa.v9i11.5468. DOI: https://doi.org/10.29303/jppipa.v9i11.5468
Rakap, S., 2023. Chatting with GPT: Enhancing Individualized Education Program Goal Development for Novice Special Education Teachers. Journal of Special Education Technology, p.01626434231211295. Available from: https://doi.org/10.1177/01626434231211295. DOI: https://doi.org/10.1177/01626434231211295
Riabko, A.V., Vakaliuk, T.A., Zaika, O.V., Kukharchuk, R.P. and Kontsedailo, V.V., 2023. Chatbot algorithm for solving physics problems. In: T.A. Vakaliuk, V.V. Osadchyi and O.P. Pinchuk, eds. Proceedings of the 2nd Workshop on Digital Transformation of Education (DigiTransfEd 2023) co-located with 18th International Conference on ICT in Education, Research and Industrial Applications (ICTERI 2023), Ivano-Frankivsk, Ukraine, September 18-22, 2023. CEUR-WS.org, CEUR Workshop Proceedings, vol. 3553, pp.75–92. Available from: https://ceur-ws.org/Vol-3553/paper5.pdf.
Semerikov, S.O., Striuk, A.M. and Shalatska, H.M., 2021. AI-assisted language education: critical review. Educational Dimension, 4, p.1–7. Available from: https://doi.org/10.31812/ed.623. DOI: https://doi.org/10.31812/ed.623
Tong, D., Tao, Y., Zhang, K., Dong, X., Hu, Y., Pan, S. and Liu, Q., 2023. Investigating ChatGPT-4’s performance in solving physics problems and its potential implications for education. Asia Pacific Education Review. Available from: https://doi.org/10.1007/s12564-023-09913-6. DOI: https://doi.org/10.1007/s12564-023-09913-6
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I., 2017. Attention is All you Need. In: I. Guyon, U. von Luxburg, S. Bengio, H.M. Wallach, R. Fergus, S.V.N. Vishwanathan and R. Garnett, eds. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. Curran Associates, Inc., vol. 30, pp.5998–6008. Available from: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
Walsh, L.N., Howard, R.G. and Bowe, B., 2007. Phenomenographic study of students’ problem solving approaches in physics. Phys. Rev. ST Phys. Educ. Res., 3, p.020108. Available from: https://doi.org/10.1103/PhysRevSTPER.3.020108. DOI: https://doi.org/10.1103/PhysRevSTPER.3.020108
Zirnstein, B., 2023. Extended context for InstructGPT with LlamaIndex. (Report). Available from: https://www.researchgate.net/profile/Bruno-Zirnstein/publication/371911146.
Downloads
Submitted
Published
Issue
Section
License
Copyright (c) 2024 Andrii V. Riabko, Tetiana A. Vakaliuk
This work is licensed under a Creative Commons Attribution 4.0 International License.
How to Cite
Accepted 2024-01-03
Published 2024-03-20
