Abstracts
Abstract
This meta-analysis synthesized empirical findings on the influence of ChatGPT on learning achievement. An electronic database search using PRISMA guidelines was conducted with relevant keywords to identify eligible research studies published between November 2022 and December 2024. A total of 22 eligible publications that met our inclusion criteria were reviewed. The overall effect size of ChatGPT's influence on learning achievement was moderate (g= 0.573), suggesting that ChatGPT has the potential to improve learning outcomes. Most participants in the studies were undergraduates (70.9%). However, subgroup analysis revealed that the effect size for middle and high school students (g= 0.928) was larger than that for undergraduates (g= 0.538), although the difference was not statistically significant. This finding highlights the importance for instructors and educational practitioners to consider the applications of ChatGPT in middle and high school settings. No significant statistical differences were found among the three learning domains: cognitive (g= 0.612), affective (g= 0.481), and metacognitive (g= 0.619). Given that nearly half of the studies focused on the cognitive domain, it is important to diversify the application of generative AI across a variety of subjects in different learning domains. The most frequently used instructional approaches with ChatGPT applications were lectures (22.1%) and self-regulated learning (16.3%). The largest effect sizes were observed for self-regulated learning (g= 1.115) and case-based learning (g= 0.836), while the smallest effect size was for game-based learning (g= 0.092, ns). This study was conducted within two years of ChatGPT's emergence, limiting in our ability to analyze a large number of publications. Nevertheless, this study offers meaningful implications for future research on the application of ChatGPT for educational purposes.
Keywords:
- ChatGPT,
- learning achievement,
- generative AI,
- AIED,
- artificial intelligence in education
Appendices
Bibliography
- Alneyadi, S., & Wardat, Y. (2023). ChatGPT: Revolutionizing student achievement in the electronic magnetism unit for eleventh-grade students in Emirates schools. Contemporary Educational Technology, 15(4), Article ep448. https://doi.org/10.30935/cedtech/13417
- Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives: Complete edition. Addison Wesley Longman, Inc.
- Becker, E., Wünsche, J., Veith, J. M., Schrader, J., Bitzenbauer, P. (2025). From cognitive relief to affective engagement: An empirical comparison of AI chatbots and instructional scaffolding in physics education. arXiv. https://doi.org/10.48550/arXiv.2508.06254
- Bloom, B. S. (Ed.). (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York, NY: David McKay.
- Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley & Sons.
- Buchner, J., & Kerres, M. (2023). Media comparison studies dominate comparative research on augmented reality in education. Computers & Education, 195, Article 104711. https://doi.org/10.1016/j.compedu.2022.104711
- Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Rand-McNally.
- Chang, C.-C., & Hwang, G.-H. (2024). Promoting professional trainers’ teaching and feedback competences with ChatGPT: A question-exploration-evaluation training mode. Educational Technology & Society, 27(2), 405–421. https://doi.org/10.30191/ETS.202404_27(2).TP06
- Chang, C.-C., & Hwang, G.-J. (2024). ChatGPT facilitated professional development: Evidence from professional trainers’ learning achievements, self-worth, and self-confidence. Interactive Learning Environments, 33(1), 883–900. https://doi.org/10.1080/10494820.2024.2362798
- Chang, C.-Y., Yang, C.-L., Jen, H.-J., Ogata, H., & Hwang, G.-H. (2024). Facilitating nursing and health education by incorporating ChatGPT into learning designs. Educational Technology & Society, 27(1), 215–230. https://doi.org/10.30191/ETS.202401_27(1).TP02
- Chen, Y.-C., & Hou, H.-T. (2024). A mobile contextualized educational game framework with ChatGPT interactive scaffolding for employee ethics training. Journal of Educational Computing Research, 62(7), 1737–1762. https://doi.org/10.1177/07356331241268505
- Coban, M., Bolat, Y. I., & Goksu, I. (2022). The potential immersive virtual reality to enhance learning: A meta-analysis. Educational Research Review, 36, Article 100452. https://doi.org/10.1016/j.edurev.2022.100452
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
- Council of Europe. (2022). Artificial intelligence and education. https://www.coe.int/en/web/education/artificial-intelligence
- Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Pearson.
- Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20, Article 22. https://doi.org/10.1186/s41239-023-00392-8
- Dalgıç, A., Yaşar, E., & Demir, M. (2024). ChatGPT and learning outcomes in tourism education: The role of digital literacy and individualized learning. Journal of Hospitality, Leisure, Sport & Tourism Education, 34, Article 100481. https://doi.org/10.1016/j.jhlste.2024.100481
- Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
- Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025). Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies. Computers & Education, 227, Article 105224. https://doi.org/10.1016/j.compedu.2024.105224.
- Díaz, B., & Nussbaum, M. (2024). Artificial intelligence for teaching and learning in schools: The need for pedagogical intelligence. Computers & Education, 217, Article 105071. https://doi.org/10.1016/j.compedu.2024.105071
- Dikilitaş, K., Klippen, M. I. F., & Keles, S. (2024). A systematic rapid review of empirical research on students’ use of ChatGPT in higher education. Nordic Journal of Systematic Reviews in Education, 2, 103–125.
- Doo, M. Y., Zhu, M., & Bonk, C. J. (2023). MOOC learners’ self-directed learning and learning outcomes: A meta-analysis. Distance Education, 44(1), 86-105. https://doi.org/10.1080/01587919.2022.2155618
- Driscoll, M. P. (1993). Psychology of learning for instruction (2nd ed.). Allyn & Bacon.
- Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ (Clinical Research Ed.), 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629
- Essel, H. B., Vlachopoulos, D., Essuman, A. B., & Amankwa, J. O. (2024). ChatGPT effects on cognitive skills of undergraduate students: Receiving instant responses from AI-based conversational large language models (LLMs). Computers and Education: Artificial Intelligence, 6, Article 100198. https://doi.org/10.1016/j.caeai.2023.100198
- Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2024). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544
- Gagne, R. M. & Briggs, L. J. (1984). Principles of instructional design. Holt, Rinehart, and Winston.
- Garcia-Varela, F., Beckerman, Z., Nussbuam, M., & Mendoza, M. (2025). Reducing interpretative ambiguity in an educational environment with ChatGPT. Computers & Education, 225, Article 105182. https://doi.org/10.1016/j.compedu.2024.105182
- Garger, J., Jacques, P. H., Gastle, B. W., & Connolly, C. M. (2019). Threats of common method variance in student assessment of instruction instruments. Higher Education Evaluation and Development, 13(1), 2–17. https://doi.org/10.1108/HEED-05-2018-0012
- Gunawardena, M., Bishop, P., & Aviruppola, K. (2024). Personalized learning: The simple, the complicated, the complex and the chaotic. Teaching and Teacher Education, 139, Article 104429. https://doi.org/10.1016/j.tate.2023.104429
- Guo, K., Zhong, Y., Li, D., & Chu, S. K. W. (2023). Effects of chatbot-assisted in-class debates on students’ argumentation skills and task motivation. Computers & Education, 203, Article 104862. https://doi.org/10.1016/j.compedu.2023.104862
- Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. https://doi.org/10.1002/sim.1186
- Hsu, M.-H. (2024). Mastering medical terminology with ChatGPT and Termbot. Health Education Journal, 83(4), 352–358. https://doi.org/10.1177/0017896923119737
- Huesca, G., Martínez-Trevino, Y., Molina-Espinosa, J. M., Sanromán-Calleros, A. R., Martínez-Román, R., Cendejas-Castro, E., & Bustos, R. (2024). Effectiveness of using ChatGPT as a tool to strengthen benefits of the flipped learning strategy. Educational Science, 14, Article 660. https://doi.org/10.3390/educsci14060660
- Jeon, J., Lee, S. & Choe, H. (2023). Beyond ChatGPT: A conceptual framework and systematic review of speech-recognition chatbots for language learning. Computers and Education, 206, Article 104898. https://doi.org/10.1016/j.compedu.2023.104898
- Lee, H.-Y., Chen, P.-H., Wang, W.-S., Huang, Y.-M., & Wu, T.-T. (2024). Empowering ChatGPT with guidance mechanism in blended learning: Effect of self-regulated learning, higher-order thinking skills, and knowledge construction. International Journal of Educational Technology in Higher Education, 21, Article 16. https://doi.org/10.1186/s41239-024-00447-4
- Lee, S. S., & Moore, R. L. (2024). Harnessing generative AI (GenAI) for automated feedback in higher education: A systematic review. Online Learning, 28(3), 82–106. https://doi.org/10.24059/olj.v28i3.4593
- Li, H.-F. (2023). Effects of a ChatGPT-based flipped learning guiding approach on learners’ courseware project performances and perceptions. Australasian Journal of Educational Technology, 39(5), 40–58. https://doi.org/10.14742/ajet.8923
- Liang, H.-Y., Hwang, G.-J., Hsu, T.-Y., & Yeh, J.-Y. (2024). Effect of an AI-based chatbot on students’ learning performance in alternate reality game-based museum learning. British Journal of Educational Technology, 55(5). 2315–2338. https://doi.org/10.1111/bjet.13448
- Liao, X., Zhang, X., Wang, Z., & Luo, H. (2024). Design and implementation of an AI-enabled visual report tool as formative assessment to promote learning achievement and self-regulated learning: An experimental study. British Journal of Educational Technology, 55(3), 1253–1276. https://doi.org/10.1111/bjet.13424
- Lo, C. K., Hew, K. F., & Jong, M. S.-y. (2024). The influence of ChatGPT on student engagement: A systematic review and future research agenda. Computers & Education, 219, Article 105100. https://doi.org/10.1016/j.compedu.2024.105100
- Martin, A. J., Collie, R. J., Kennett, R., Liu, D., Ginns, P., Sudimantara, L. B., ... & Rüschenpöhler, L. G. (2025). Integrating generative AI and load reduction instruction to individualize and optimize students' learning. Learning and Individual Differences, 121, 102723. https://doi.org/10.1016/j.lindif.2025.102723
- Ng, D. T. K., Tan, C. W., & Leung, J. K. L. (2024). Empowering student self-regulated learning and science education through ChatGPT: A pioneering pilot study. British Journal of Educational Technology, 55(4), 1328–1353. https://doi.org/10.1111/bjet.13454
- Nwana, H. S. (1990). Intelligent tutoring systems: An overview. Artificial Intelligence Review, 4, 251–277. https://doi.org/10.1007/BF00168958
- Patac, L. P., & Patac Jr, A. V. (2025). Using ChatGPT for academic support: Managing cognitive load and enhancing learning efficiency–A phenomenological approach. Social Sciences & Humanities Open, 11, 101301. https://doi.org/10.1016/j.ssaho.2025.101301
- Popenici, S. A. D., & Kerr, S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12, Article 22. https://doi.org/10.1186/s41039-017-0062-8
- PRISMA. (2020). PRISMA flow diagram. https://www.prisma-statement.org/prisma-2020-flow-diagram
- Reiser, R. A. (2001). A history of instructional design and technology: Part I: A history of instructional media. Educational Technology Research and Development, 49(1), 53–64. https://doi.org/10.1007/BF02504506
- Reuters. (2024, August 30). OpenAI says ChatGPT’s weekly users have grown to 200 million. https://www.reuters.com/technology/artificial-intelligence/openai-says-chatgpts-weekly-users-have-grown-200-million-2024-08-29/
- Rospigliosi, P. (2023). Artificial intelligence in teaching and learning: What questions should we ask of ChatGPT? Interactive Learning Environments, 31(1), 1–3. https://doi.org/10.1080/10494820.2023.2180191
- Sabzalieva, E., & Valentini, A. (2023). ChatGPT and artificial intelligence in higher education: Quick start guide. UNESCO. https://unesdoc.unesco.org/ark:/48223/pf0000385146
- Saettler, P. (1990). The evolution of American educational technology. Libraries Unlimited.
- Schoenherr, J., Strohmaier, A. R, & Schukajlow, S. (2024). Learning with visualizations helps: A meta-analysis of visualization intervention in mathematics education. Educational Research Review, 45, Article 100639. https://doi.org/10.1016/j.edurev.2024.100639
- Smith, P. L., & Ragan, T. J. (2005). Instructional design (3rd ed.). Wiley.
- Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106(2), 331–347. https://doi.org/10.1037/a0034752
- Sun, L., & Zhou, L. (2024). Does generative artificial intelligence improve the academic achievement of colleges? A meta-analysis. Journal of Educational Computing Research, 62(7), 1673-1713. https://doi.org/10.1177/07356331241277937
- Svendsen, K., Askar, M., Umer, D., & Halvorsen, K. H. (2024). Short-term learning effect of ChatGPT on pharmacy students’ learning. Exploratory Research in Clinical and Social Pharmacy, 15, Article 100478. https://doi.org/10.1016/j.rcsop.2024.100478
- Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4
- Teng, M. F. (2024). “ChatGPT is the companion, not enemies”: EFL learners’ perceptions and experiences in using ChatGPT for feedback in writing. Computers & Education: Artificial Intelligence, 7, Article 100270. https://doi.org/10.1016/j.caeai.2024.100270
- Urban, M., Děchtěrenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024). ChatGPT improves creative problem-solving performance in university students: An experimental study. Computers & Education, 215, Article 105031. https://doi.org/10.1016/j.compedu.2024.105031
- Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540
- Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
- Warnick, B. R., & Burbules, N. C. (2007). Media comparison studies: Problems and possibilities. Teachers College Record, 109(11), 2483–2510. https://eric.ed.gov/?id=EJ820500
- Wu, R., & Yu, Z. (2024). Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis. British Journal of Educational Technology, 55(1), 10–33. https://doi.org/10.1111/bjet.13334
- Xue, Y., Chen, H., Bai, G. R., Tairas, R., & Huang, Y. (2024). Does ChatGPT help with introductory programming? An experiment of using students ChatGPT in CS1. In A. Paiva, R. Abreu, K. Gama, & J. Siegmund (Chairs), ICSE-SEET '24: Proceedings of the 46th international conference on software engineering: Software engineering education and training (pp. 331–341). ACM. https://doi.org/10.1145/3639474.3640076
- Yilmaz, R., & Yilmaz. F. G. K. (2023). The effect of generative artificial intelligence (AI)-based tool use on students’ computational thinking skills, programming self-efficacy and motivation. Computers & Education: Artificial Intelligence, 4, Article 100147. https://doi.org/10.1016/j.caeai.2023.100147
- Zhang, K., & Aslan, A. B. (2021). AI technologies for education: Recent research and future directions. Computers and Education: Artificial Intelligence, 2, Article 100025. https://doi.org/10.1016/j.caeai.2021.100025
- Zhou, W., & Kim, Y. (2024). Innovative music education: An empirical assessment of ChatGPT-4’s impact on student learning experiences. Education and Information Technology, 29, 20855–2088. https://doi.org/10.1007/s10639-024-12705-z