Abstracts
Abstract
This mixed-methods study investigated the effectiveness of Generative AI (GenAI) powered intelligent tutoring systems (ITS) in undergraduate physics education, specifically comparing learning outcomes between students using Khanmigo (Khan Academy's AI tutor) and Google search engine. The study involved 69 undergraduate students divided into two groups (Khanmigo and Google search engine), with a third Paper-only group emerging during the experiment. Participants completed pre and posttests using the Lunar Phases Concept Inventory (LPCI) and participated in structured interviews about their learning experiences. Quantitative analysis revealed significant learning gains across all conditions but found no statistically significant differences between groups in terms of learning outcomes. Qualitative findings indicated that students perceived Khanmigo positively, appreciated its step-by-step guidance, practice problems, and personalized interactions. However, students viewed it as a supplementary tool rather than a replacement for traditional instruction. The study's findings suggest that while GenAI-powered tutoring systems can effectively support learning, their immediate impact on learning outcomes may be comparable to traditional methods. However, the short duration of exposure to the AI tutor and the quality of the printed materials may have affected these results.
Appendices
Bibliography
- Ahuja, N. J., & Sille, R. (2013). A critical review of development of intelligent tutoring systems: Retrospect, present and prospect. International Journal of Computer Science Issues, 10(4), 39.
- Aleven, V. A. W. M. M., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer‐based Cognitive Tutor. Cognitive Science, 26(2), 147–179. https://doi.org/10.1207/s15516709cog2602_1
- Alkhatlan, A., and Kalita, J. (2018). Intelligent tutoring systems: a comprehensive historical survey with recent developments. arXiv preprint arXiv:1812.09628. doi: 10.5120/ijca2019918451
- Anderson, J. R., & Gluck, K. (2001). What role do cognitive architectures play in intelligent tutoring systems. In D. Klahr & S. M. Carver (Eds.), Cognition and Instruction: Twenty-five Years of Progress (pp. 227-262). Mahwah,NJ: Erlbaum.
- As’ad, M. (2024). Intelligent tutoring systems, generative artificial intelligence (AI), and healthcare agents: A proof of concept and dual-layer approach. Cureus. https://doi.org/10.7759/cureus.69710
- Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023). Transforming education: A comprehensive review of generative artificial intelligence in educational settings through bibliometric and content analysis. Sustainability, 15(17), 12983. https://doi.org/10.3390/su151712983
- Baltezarević, R., & Baltezarević, I. (2024). Students’ attitudes on the role of artificial intelligence (AI) in personalized learning. International Journal of Cognitive Research in Science, Engineering & Education, 12(2), 387–397. https://doi.org/10.23947/2334-8496-2024-12-2-387-397
- Bassner, P., Frankford, E., & Krusche, S. (2024). Iris: An ai-driven virtual tutor for computer science education. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 (pp. 394–400). https://doi.org/10.1145/3649217.3653543
- Batsaikhan, B. Z., & Correia, A. P. (2024). The effects of Generative Artificial Intelligence on Intelligent Tutoring Systems in higher education: A systematic review. Studies in Technology Enhanced Learning, 4(1).
- Beltozar-Clemente, S., & Díaz-Vega, E. (2024). Physics XP: Integration of ChatGPT and Gamification to Improve Academic Performance and Motivation in Physics 1 Course. International Journal of Engineering Pedagogy, 14(6). https://doi.org/10.3991/ijep.v14i6.47127
- Bloom, B. S. (1984). The 2 Sigma Problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16. https://doi.org/10.3102/0013189X013006004
- Borchers, C., Carvalho, P. F., Xia, M., Liu, P., Koedinger, K. R., & Aleven, V. (2023). What makes problem-solving practice effective? Comparing paper and AI tutoring. In O. Viberg, I. Jivet, P. J. Muñoz-Merino, M. Perifanou, & T. Papathoma (Eds.), Responsive and Sustainable Educational Futures (pp. 44–59). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-42682-7_4
- Chen, B., Zhu, X., & Díaz del Castillo H., F. (2023). Integrating generative AI in knowledge building. Computers and Education: Artificial Intelligence, 5, 100184. https://doi.org/10.1016/j.caeai.2023.100184
- Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.
- Feng, S., Magana, A. J., & Kao, D. (2021). A systematic review of literature on the effectiveness of intelligent tutoring systems in STEM. In 2021 IEEE Frontiers in Education Conference (FIE) (pp. 1–9). IEEE.
- Gligorea, I., Cioca, M., Oancea, R., Gorski, A.-T., Gorski, H., & Tudorache, P. (2023). Adaptive learning using artificial intelligence in e-learning: A literature review. Education Sciences, 13(12), 1216.
- Gökoğlu, S., & Erdoğdu, F. (2025). The effects of GenAI on learning performance: A meta-analysis study. Educational Technology & Society, 28(3). https://doi.org/10.30191/ETS.202507_28(3).TP04
- Graesser, A. C., Chipman, P., Haynes, B. C., & Olney, A. (2005). AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education, 48(4), 612–618.
- Graesser, A. C., Greenberg, D., Frijters, J. C., & Talwar, A. (2022). Using AutoTutor to track performance and engagement in a reading comprehension intervention for adult literacy students. Revista Signos. Estudios de Lingüística, 54(107). http://www.revistasignos.cl/index.php/signos/article/view/776
- Guo, L., Wang, D., Gu, F., Li, Y., Wang, Y., & Zhou, R. (2021). Evolution and trends in intelligent tutoring systems research: A multidisciplinary and scientometric view. Asia Pacific Education Review, 22(3), 441–461. https://doi.org/10.1007/s12564-021-09697-7
- Hsu, M.-H. (2024). Mastering medical terminology with ChatGPT and Termbot. Health Education Journal, 83(4), 352–358. https://doi.org/10.1177/00178969231197371
- Kamalov, F., Santandreu Calonge, D., & Gurrib, I. (2023). New era of artificial intelligence in education: Towards a sustainable multifaceted revolution. Sustainability, 15(16), 12451.
- Kavadella, A., Silva, M. A. D. da, Kaklamanos, E. G., Stamatopoulos, V., & Giannakopoulos, K. (2024). Evaluation of ChatGPT’s Real-Life Implementation in Undergraduate Dental Education: Mixed Methods Study. JMIR Medical Education, 10(1), e51344. https://doi.org/10.2196/51344
- Khan, S. (2024). Brave new words: How AI will revolutionize education (and why that’s a good thing). Penguin.
- Khan Academy. (n.d.). Meet Khanmigo, Khan Academy’s AI-powered teaching assistant & tutor. Khanmigo. https://khanmigo.ai/
- Kong, S.-C., & Yang, Y. (2024). A human-centered learning and teaching framework using generative artificial intelligence for self-regulated learning development through domain knowledge learning in K-12 settings. IEEE Transactions on Learning Technologies, 17, 1562–1573. https://doi.org/10.1109/TLT.2024.3392830
- Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Review of Educational Research, 86(1), 42–78. https://doi.org/10.3102/0034654315581420
- Kumar, H., Musabirov, I., Reza, M., Shi, J., Kuzminykh, A., Williams, J. J., & Liut, M. (2023). Impact of guidance and interaction strategies for LLM use on learner performance and perception. arXiv preprint arXiv:2310.13712 (2023) https://doi.org/10.48550/arXiv.2310.13712
- Lee, Y. F., Hwang, G. J., & Chen, P. Y. (2022). Impacts of an AI-based chatbot on college students’ after-class review, academic performance, self-efficacy, learning attitude, and motivation. Educational Technology Research and Development, 70(5), 1843-1865. https://doi.org/10.1007/s11423-022-10142-8
- Lin, C.-C., Huang, A. Y. Q., & Lu, O. H. T. (2023). Artificial intelligence in intelligent tutoring systems toward sustainable education: A systematic review. Smart Learning Environments, 10(1), 41. https://doi.org/10.1186/s40561-023-00260-y
- Lin, J., Singh, S., Sha, L., Tan, W., Lang, D., Gašević, D., & Chen, G. (2022). Is it a good move? Mining effective tutoring strategies from human–human tutorial dialogues. Future Generation Computer Systems, 127, 194–207.
- Lindell, R. S., & Olsen, J. P. (2002). Developing the lunar phases concept inventory. Proceedings of the 2002 Physics Education Research Conference. New York: Perc Publishing.
- Mayer, R. E. (2024). The past, present, and future of the cognitive theory of multimedia learning. Educational Psychology Review, 36(1), 8. https://doi.org/10.1007/s10648-023-09842-1
- Nwana, H.S. (1990). Intelligent tutoring systems: An overview. Artificial Intelligence Review, 4(4). https://doi.org/10.1007/BF00168958
- Polakova, P., & Klimova, B. (2024). Implementation of AI-driven technology into education–a pilot study on the use of chatbots in foreign language learning. Cogent Education, 11(1), 1-15. https://doi.org/10.1080/2331186X.2024.2355385
- Raslan, G. (2024). The impact of the zone of proximal development concept (scaffolding) on the students problem solving skills and learning outcomes. In K. Al Marri, F. A. Mir, S. A. David, & M. Al-Emran (Eds.), BUiD Doctoral Research Conference 2023 (pp. 59–66). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-56121-4_6
- Saldaña, J. (2021). The coding manual for qualitative researchers. Sage Publications.
- Savolainen, A. (2023, September 29). Sal Khan: "I see AI as an additional tool, but a very powerful one" UNESCO interview The UNESCO Courier. https://courier.unesco.org/en/articles/sal-khan-i-see-ai-additional-tool-very-powerful-one
- Stadler, M., Bannert, M., & Sailer, M. (2024). Cognitive ease at a cost: LLMs reduce mental effort but compromise depth in student scientific inquiry. Computers in Human Behavior, 160, 108386.
- Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106(2), 331.
- VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. https://doi.org/10.1080/00461520.2011.611369
- Wang, L., & Li, W. (2024). The impact of AI usage on university students’ willingness for autonomous learning. Behavioral Sciences (2076-328X), 14(10), 956. https://doi.org/10.3390/bs14100956
- Xiao, J., Bozkurt, A., Nichols, M., Pazurek, A., Stracke, C. M., Bai, J. Y., Farrow, R., Mulligan, D., Nerantzi, C., Sharma, R. C., Lenandlar S., F., I., S., A., H., S., Bond, M., Dron, J., Moore, S., Leng, J., Slagter van Tryon, P.J....Themeli, C. (2025). Venturing into the unknown: Critical insights into grey areas and pioneering future directions in educational generative AI research. TechTrends, 1–16.
- Yannier, N., Hudson, S. E., Chang, H., & Koedinger, K. R. (2024). AI adaptivity in a mixed-reality system improves learning. International Journal of Artificial Intelligence in Education, 34(4), 1541–1558. https://doi.org/10.1007/s40593-023-00388-5