Résumés
Abstract
Background: This study aims to evaluate the effectiveness of the OSCEai, a large language model-based platform that simulates clinical encounters, in enhancing undergraduate medical education.
Methods: A web-based application, OSCEai, was developed to bridge theoretical and practical learning. Following use, medical students from the University of Calgary Class of 2026 completed an anonymized survey on the usability, utility, and overall experience of OSCEai.
Results: A total of 37 respondents answered the anonymized survey. The OSCEai platform was highly valued for its ability to provide data on demand (33/37), support self-paced learning (30/37), and offer realistic patient interactions (29/37). The ease of use and medical content quality were rated at 4.73 (95% CI: 4.58 to 4.88) and 4.70 (95% CI: 4.55 to 4.86) out of 5, respectively. Some participants (8/37) commented that few cases were not representative and needed clarification about app functionality. Despite these limitations, OSCEai was favorably compared to lecture-based teaching methods, with an overall reception rating of 4.62 (95% CI: 4.46 to 4.79) out of 5.
Interpretation: The OSCEai platform fills a gap in medical training through its scalable, interactive, and personalized design. The findings suggest that integrating technologies, like OSCEai, into medical curricula can enhance the quality and efficacy of medical education.
Résumé
Contexte : Cette étude vise à évaluer l'efficacité de l'OSCEai, une nouvelle plateforme basée sur un modèle de langage étendu qui simule des rencontres cliniques, pour améliorer l'enseignement médical de premier cycle.
Méthodes : Une application web, OSCEai, a été créée pour faire le lien entre l'apprentissage théorique et l'apprentissage pratique. Après utilisation, les étudiants en médecine de la promotion 2026 de l'Université de Calgary ont répondu à une enquête anonyme sur la facilité d'utilisation, l'utilité et leur expérience globale de l'OSCEai.
Résultats : La plateforme a été très appréciée pour sa capacité à fournir des données à la demande (33/37), à soutenir l'apprentissage à son propre rythme (30/37) et à offrir des interactions réalistes avec des patients (29/37). La facilité d'utilisation et la qualité du contenu médical ont été évaluées respectivement à 4,73 (IC 95 % : 4,58 à 4,88) et 4,70 (IC 95 % : 4,55 à 4,86) sur 5. Certains participants (8/37) ont indiqué que quelques cas n'étaient pas représentatifs et qu'il fallait apporter des éclaircissements en ce qui a trait aux fonctionnalités de l'application. Malgré ces limites, la plateforme OSCEai a été favorablement comparée aux méthodes d'enseignement traditionnelles, avec une note de réception globale de 4,62 (IC 95 % : 4,46 à 4,79) sur 5.
Interprétation : La plateforme OSCEai comble une lacune dans la formation médicale grâce à sa conception évolutive, interactive et personnalisée. Les résultats suggèrent que l'intégration de technologies, comme OSCEai, dans les programmes d'études médicales peut améliorer la qualité et l'efficacité de l'enseignement médical.
Parties annexes
Bibliography
- Servant-Miklos VFC. Fifty years on: a retrospective on the world’s first problem-based learning programme at McMaster University Medical School. Health Prof Educ. 2019 Mar;5(1):3–12. http://dx.doi.org/10.1016/j.hpe.2018.04.002
- Neville AJ, Norman GR. PBL in the undergraduate MD program at McMaster University: three iterations in three decades: Acad Med. 2007 Apr;82(4):370–4. http://dx.doi.org/10.1097/ACM.0b013e318033385d
- Spaulding WB. The undergraduate medical curriculum (1969 model): McMaster university. Can Med Assoc J. 1969 Apr 12;100(14):659–64. https://www.ncbi.nlm.nih.gov/pubmed/5776441
- Sanson-Fisher R, Hobden B, Carey M, Mackenzie L, Hyde L, Shepherd J. Interactional skills training in undergraduate medical education: ten principles for guiding future research. BMC Med Educ. 2019 May 15;19(1):144. http://dx.doi.org/10.1186/s12909-019-1566-2
- Kurtz SM, Silverman JD. The Calgary-Cambridge Referenced Observation Guides: an aid to defining the curriculum and organizing the teaching in communication training programmes. Med Educ. 1996 Mar;30(2):83–9. http://dx.doi.org/10.1111/j.1365-2923.1996.tb00724.x
- Gordon MS, Ewy GA, DeLeon AC Jr, et al. “Harvey,” the cardiology patient simulator: pilot studies on teaching effectiveness. Am J Cardiol. 1980 Apr;45(4):791–6. http://dx.doi.org/10.1016/0002-9149(80)90123-x
- Wang S, Ren X, Ye J, Wang W, Huang H, Qin C. Exploration of simulation-based medical education for undergraduate students. Med. 2021 May 21;100(20):e25982. http://dx.doi.org/10.1097/MD.0000000000025982
- Al-Elq A. Simulation-based medical teaching and learning. J Fam Community Med. 2010;17(1):35. http://dx.doi.org/10.4103/1319-1683.68787
- Kaplovitch E, Otremba M, Morgan M, Devine LA. Cost-efficient medical education: an innovative approach to creating educational products. J Grad Med Educ. 2019 Dec 1;11(6):713–6. http://dx.doi.org/10.4300/JGME-D-19-00155.1
- Wynter L, Burgess A, Kalman E, Heron JE, Bleasel J. Medical students: what educational resources are they using? BMC Med Educ. 2019 Dec;19(1):36. http://dx.doi.org/10.1186/s12909-019-1462-9
- Abd-Alrazaq A, AlSaad R, Alhuwail D, et al. large language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ. 2023 Jun 1;9:e48291. http://dx.doi.org/10.2196/48291
- Clusmann J, Kolbinger FR, Muti HS, et al. The future landscape of large language models in medicine. Commun Med (Lond). 2023 Oct 10;3(1):141. http://dx.doi.org/10.1038/s43856-023-00370-1
- Wu S, Irsoy O, Lu S, et al. BloombergGPT: a large language model for finance. arXiv [cs.LG]. 2023. http://arxiv.org/abs/2303.17564
- Guo E, Gupta M, Deng J, Park Y-J, Paget M, Naugler C. Automated paper screening for clinical reviews using large language models: data analysis study. J Med Internet Res. 2024 Jan 12;26:e48996. http://dx.doi.org/10.2196/48996
- Unlu O, Shin J, Mailly CJ, et al. Retrieval-augmented generation–enabled GPT-4 for clinical trial screening. NEJM AI. 2024 Jun 17; http://dx.doi.org/10.1056/aioa2400181
- Katz DM, Bommarito MJ, Gao S, Arredondo P. GPT-4 passes the bar exam. Philos Trans A Math Phys Eng Sci. 2024 Apr 15;382(2270):20230254. http://dx.doi.org/10.1098/rsta.2023.0254
- Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya. Improving language understanding by generative pre-training. 2018.
- Biri SK, Kumar S, Panigrahi M, Mondal S, Behera JK, Mondal H. Assessing the utilization of large language models in medical education: insights from undergraduate medical students. Cureus. 2023 Oct;15(10):e47468. http://dx.doi.org/10.7759/cureus.47468
- Safranek CW, Sidamon-Eristoff AE, Gilson A, Chartash D. The role of large language models in medical education: applications and implications. JMIR Med Educ. 2023 Aug 14;9:e50945. http://dx.doi.org/10.2196/50945
- Divito CB, Katchikian BM, Gruenwald JE, Burgoon JM. The tools of the future are the challenges of today: the use of ChatGPT in problem-based learning medical education. Med Teach. 2024 Mar;46(3):320–2. http://dx.doi.org/10.1080/0142159X.2023.2290997
- Cumming School of Medicine. Re-Imagining Medical Education (RIME) 2024. Available from https://cumming.ucalgary.ca/mdprogram/current-students/pre-clerkship-year-1-2/re-imagining-medical-education-rime.
- Egli A. ChatGPT, GPT-4, and other large language models: the next revolution for clinical microbiology? Clin Infect Dis. 2023 Nov 11;77(9):1322–8. http://dx.doi.org/10.1093/cid/ciad407
- Zayyan M. Objective structured clinical examination: the assessment of choice. Oman Med J. 2011 Jul;26(4):219–22. http://dx.doi.org/10.5001/omj.2011.55
- meta-llama/Meta-Llama-3-70B · Hugging Face. Available from https://huggingface.co/meta-llama/Meta-Llama-3-70B [Accessed on Jun 23, 2024].
- Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004 Sep 29;6(3):e34. http://dx.doi.org/10.2196/jmir.6.3.e34
- Kulasegaram KM, Martimianakis MA, Mylopoulos M, Whitehead CR, Woods NN. Cognition before curriculum: rethinking the integration of basic science and clinical learning. Acad Med. 2013 Oct;88(10):1578–85. http://dx.doi.org/10.1097/ACM.0b013e3182a45def
- Bosse HM, Nickel M, Huwendiek S, Schultz JH, Nikendei C. Cost-effectiveness of peer role play and standardized patients in undergraduate communication training. BMC Med Educ. 2015 Oct 24;15:183. http://dx.doi.org/10.1186/s12909-015-0468-1
- Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011 Sep 7;306(9):978–88. http://dx.doi.org/10.1001/jama.2011.1234
- Walsh K, Levin H, Jaye P, Gazzard J. Cost analyses approaches in medical education: there are no simple solutions. Med Educ. 2013 Oct;47(10):962–8. http://dx.doi.org/10.1111/medu.12214
- Emanuel EJ. The inevitable reimagining of medical education. JAMA 2020 Mar 24;323(12):1127. http://dx.doi.org/10.1001/jama.2020.1227
- McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011 Jun;86(6):706–11. http://dx.doi.org/10.1097/ACM.0b013e318217e119
- Kneebone RL, Scott W, Darzi A, Horrocks M. Simulation and clinical practice: strengthening the relationship. Med Educ. 2004 Oct;38(10):1095–102. http://dx.doi.org/10.1111/j.1365-2929.2004.01959.x
- Skelly K, Kim P, Rosenbaum M, Wilbur J. Goldilocks and entrustment: finding the amount of learner autonomy that’s just right. MedEdPORTAL. 2020 Oct 13;16:10987. http://dx.doi.org/10.15766/mep_2374-8265.10987
- Conner SM, Choi N, Fuller J, et al. Trainee autonomy and supervision in the modern clinical learning environment: a mixed-methods study of faculty and trainee perspectives. Res Sq. 2023 Jun 6;rs.3.rs-2982838. http://dx.doi.org/10.21203/rs.3.rs-2982838/v1
- Kamphuis C, Barsom E, Schijven M, Christoph N. Augmented reality in medical education? Perspect Med Educ. 2014 Sep;3(4):300–11. http://dx.doi.org/10.1007/s40037-013-0107-7
- Thistlethwaite J. Interprofessional education: a review of context, learning and the research agenda. Med Educ. 2012 Jan;46(1):58–70. http://dx.doi.org/10.1111/j.1365-2923.2011.04143.x

