Abstracts
Abstract
Background: Medical school applications often require short written essays or personal statements, which are purportedly used to assess professional qualities related to the practice of medicine. With generative artificial intelligence (AI) tools capable of supplementing or replacing inputs by human applicants, concerns about how these tools impact written assessments are growing. This study explores how AI influences the ratings of essays used for medical school admissions
Methods: A within-subject experimental design was employed. Eight participants (academic clinicians, faculty researchers, medical students, and a community member) rated essays written by 24 undergraduate students and recent graduates from McMaster University. The students were divided into four groups: medical school aspirants with AI assistance (ASP-AI), aspirants without AI assistance (ASP), non-aspirants with AI assistance (NASP-AI), and essays generated solely by ChatGPT 3.5 (AI-ONLY). Participants were provided training in the application of single Likert scale tool before rating. Differences in ratings by writer group were determined via one-way between group ANOVA.
Results: Analyses revealed no statistically significant differences in ratings across the four writer groups (p = .358). The intraclass correlation coefficient was .147.
Conclusion: The proliferation of AI adds to prevailing questions about the value personal statements and essays have in supporting applicant selection. We speculate that these assessments hold less value than ever in providing authentic insight into applicant attributes. In this context, we suggest that medical schools move away from the use of essays in their admissions processes.
Résumé
Contexte : Les demandes d'admission dans les facultés de médecine exigent souvent de courtes lettres de motivation écrites ou des lettres de présentation, qui devraient être utilisées pour évaluer les qualités professionnelles liées à la pratique de la médecine. Les outils génératifs d'intelligence artificielle (IA) étant capables de compléter ou de remplacer les données fournies par les candidats humains, l'impact de ces outils sur les évaluations écrites suscite de plus en plus d'inquiétudes. Cette étude explore l'influence de l'IA sur l'évaluation des lettres de motivation utilisées pour les admissions dans les facultés de médecine.
Méthodes : Un plan expérimental à l'intérieur d'un sujet a été utilisé. Huit participants (cliniciens universitaires, chercheurs de la faculté, étudiants en médecine et un membre de la communauté) ont évalué des lettres de motivation rédigées par 24 étudiants de premier cycle et diplômés récents de l'Université McMaster. Les étudiants ont été répartis en quatre groupes : les aspirants à la faculté de médecine avec l'aide de l'IA (ASP-IA), les aspirants sans l'aide de l'IA (ASP), les non-aspirants avec l'aide de l'IA (NASP-IA), et les lettres de motivation générées uniquement par ChatGPT 3.5 (IA-UNIQUEMENT). Les participants ont reçu une formation pour l'application de l'échelle de Likert unique avant d'évaluer. Les différences d'évaluation selon le groupe de rédacteurs ont été déterminées au moyen d'une ANOVA à sens unique entre les groupes.
Résultats : Les analyses n'ont révélé aucune différence statistiquement significative dans les évaluations entre les quatre groupes de rédacteurs (p = 0,358). Le coefficient de corrélation intraclasse était de 0,147.
Conclusion : La prolifération de l'IA renforce les questions qui se posent présentement sur la valeur des lettres de présentation et des lettres de motivation dans la sélection des candidats. Nous supposons que ces évaluations ont moins de valeur que jamais pour ce qui est de fournir un aperçu authentique des attributs des candidats. Dans ce contexte, nous suggérons que les facultés de médecine abandonnent l'utilisation des lettres de motivation dans leurs processus d'admission.
Download the article in PDF to read it.
Download
Appendices
Bibliography
- Lee J, Wu AS, Li D, Kulasegaram K (Mahan). Artificial Intelligence in undergraduate medical education: a scoping review. Acad Med. 2021;96:S62-70. https://doi.org/10.1097/ACM.0000000000004291
- Darcy AM, Louie AK, Roberts LW. Machine learning and the profession of medicine. JAMA. 2016;315:551. https://doi.org/10.1001/jama.2015.18421
- Hashimoto DA, Johnson KB. The use of Artificial Intelligence tools to prepare medical school applications. Acad Med. 2023;98:978-82. https://doi.org/10.1097/ACM.0000000000005309
- Boscardin CK, Gin B, Golde PB, Hauer KE. ChatGPT and generative Artificial Intelligence for medical education: potential impact and opportunity. Acad Med. 2023. https://doi.org/10.1097/ACM.0000000000005439
- Salvatori P. Reliability and Validity of Admissions Tools Used to Select Students for the Health Professions. Adv Health Sci Educ. 2001;6:159-75. https://doi.org/10.1023/A:1011489618208
- Kulatunga Moruzi C, Norman GR. Validity of admissions measures in predicting performance outcomes: the contribution of cognitive and non-cognitive dimensions. Teach Learn Med. 2002;14:34-42. https://doi.org/10.1207/S15328015TLM1401_9
- Youdas JW, Hallman HO, Carey JR, Bogard CL, Garrett TR. Reliability and Validity of Judgments of Applicant Essays as a Predictor of Academic Success in an Entry-Level Physical Therapy Education Program. J Phys Ther Educ. 1992;6:15. https://doi.org/10.1097/00001416-199201000-00005
- Peeters MJ, Schmude KA, Steinmiller CL. Inter-rater reliability and false confidence in precision: Using standard error of measurement within PharmD admissions essay rubric development. Curr Pharm Teach Learn. 2014;6:298-303. https://doi.org/10.1016/j.cptl.2013.11.014
- MD Program University of Toronto. MD Program University of Toronto non-academic requirements. Available at: https://applymd.utoronto.ca/non-academic-requirements [Accessed on Nov 3, 2024].
- Schulich Medicine & Dentistry Admissions. Admission requirements. Available at: https://www.schulich.uwo.ca/med_dent_admissions/medicine/admission_requirements.html [Accessed on Nov 3, 2024].
- Toronto Metropolitan University (TMU) School of Medicine. Admissions requirements. Available at: https://www.torontomu.ca/school-of-medicine/programs/md/ [Accessed on Nov 3, 2024].
- Dalhousie University Medical School Admissions. Admission requirements. Available at: https://medicine.dal.ca/departments/core-units/admissions/admissions.html [Accessed on Nov 3, 2024].
- Queen's University School of Medicine. Queen's-Lakeridge Health MD Family Medicine program eligibility & application process. Available at: https://meds.queensu.ca/academics/mdprogram/queens-lakeridge-health-md-family-medicine-program/eligibility_application_process [Accessed on Nov 3, 2024].
- Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. Assessing personal qualities in medical school admissions: Acad Med. 2003;78:313-21. https://doi.org/10.1097/00001888-200303000-00016
- Marzuki, Widiati U, Rusdin D, Darwin, Indrawati I. The impact of AI writing tools on the content and organization of students' writing: EFL teachers' perspective. Cogent Educ. 2023;10:2236469. https://doi.org/10.1080/2331186X.2023.2236469
- Yeadon W, Inyang O-O, Mizouri A, Peach A, Testrow CP. The death of the short-form physics essay in the coming AI revolution. Phys Educ. 2023;58:035027. https://doi.org/10.1088/1361-6552/acc5cf
- Gao CA, Howard FM, Markov NS, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. Sci Comm Educ; 2022. https://doi.org/10.1101/2022.12.23.521610
- Herbold S, Hautli-Janisz A, Heuer U, Kikteva Z, Trautsch A. A large-scale comparison of human-written versus ChatGPT-generated essays. Sci Rep. 2023;13:18617. https://doi.org/10.1038/s41598-023-45644-9
- Ayers JW, Poliak A, Dredze M, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med. 2023;183:589. https://doi.org/10.1001/jamainternmed.2023.1838
- Waltzer T, Cox RL, Heyman GD. Testing the ability of teachers and students to differentiate between essays generated by ChatGPT and high school students. Hum Behav Emerg Technol. 2023;2023:1923981. https://doi.org/10.1155/2023/1923981
- Yan D, Fauss M, Hao J, Cui W. Detection of AI-generated essays in writing assessments. Psychol Test Assess Model. 2023;65:125-44.
- Fleckenstein J, Meyer J, Jansen T, Keller SD, Köller O, Möller J. Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays. Comput Educ Artif Intell. 2024;6:100209. https://doi.org/10.1016/j.caeai.2024.100209
- Chen D, Parsa R, Hope A, et al. Physician and artificial intelligence chatbot responses to cancer questions from social media. JAMA Oncol. 2024;10:956-60. https://doi.org/10.1001/jamaoncol.2024.0836
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)-a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377-81. https://doi.org/10.1016/j.jbi.2008.08.010
- Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019;95:103208. https://doi.org/10.1016/j.jbi.2019.103208
- Shaheen MY. Applications of Artificial Intelligence (AI) in healthcare: a review. Sci Prepr. 2021. https://doi.org/10.14293/S2199-1006.1.SOR-.PPVRY8K.v1
- Dias PP, Jayasinghe LB, Waldmann D. Investigation of mycelium-miscanthus composites as building insulation material. Results Mater. 2021;10:100189. https://doi.org/10.1016/j.rinma.2021.100189
- White J, Brownell K, Lemay J-F, Lockyer JM. "What do they want me to say?" The hidden curriculum at work in the medical school selection process: a qualitative study. BMC Med Educ. 2012;12:17. https://doi.org/10.1186/1472-6920-12-17
- Lin KY, Anspach RR, Crawford B, Parnami S, Fuhrel-Forbis A, Vries RGD. What must I do to succeed?: narratives from the US premedical experience. Soc Sci Med 1982. 2014;119:98. https://doi.org/10.1016/j.socscimed.2014.08.017
- Yang L, Chang I, Ritz S, Grierson L. Research experiences for Canadian aspiring physicians: a descriptive analysis of medical school admission policies concerning research involvement in Canada. BMC Med Educ. 2022;22:151. https://doi.org/10.1186/s12909-022-03207-y
- Chang I, Yang L, Elma A, Ritz SA, Grierson L. A brief report of aspiring medical student perceptions and behaviours concerning research experiences for selection into Canadian medical schools. Can Med Educ J. 2023;14:77-81. https://doi.org/10.36834/cmej.76255
- Brown B, Carpio B, Roberts J. The use of an autobiographical letter in the nursing admissions process: initial reliability and validity. Can J Nurs Res Rev Can Rech En Sci Infirm. 1991;23:9-20.