Investigating the threat of AI to undergraduate medical school admissions: a study of its potential impact on the rating of applicant essays
DOI:
https://doi.org/10.36834/cmej.79690Abstract
Background: Medical school applications often require short written essays or personal statements, which are purportedly used to assess professional qualities related to the practice of medicine. With generative artificial intelligence (AI) tools capable of supplementing or replacing inputs by human applicants, concerns about how these tools impact written assessments are growing. This study explores how AI influences the ratings of essays used for medical school admissions
Methods: A within-subject experimental design was employed. Eight participants (academic clinicians, faculty researchers, medical students, and a community member) rated essays written by 24 undergraduate students and recent graduates from McMaster University. The students were divided into four groups: medical school aspirants with AI assistance (ASP-AI), aspirants without AI assistance (ASP), non-aspirants with AI assistance (NASP-AI), and essays generated solely by ChatGPT 3.5 (AI-ONLY). Participants were provided training in the application of single Likert scale tool before rating. Differences in ratings by writer group were determined via one-way between group ANOVA.
Results: Analyses revealed no statistically significant differences in ratings across the four writer groups (p = .358). The intraclass correlation coefficient was .147.
Conclusion: The proliferation of AI adds to prevailing questions about the value personal statements and essays have in supporting applicant selection. We speculate that these assessments hold less value than ever in providing authentic insight into applicant attributes. In this context, we suggest that medical schools move away from the use of essays in their admissions processes.
References
1. Lee J, Wu AS, Li D, Kulasegaram K (Mahan). Artificial Intelligence in undergraduate medical education: a scoping review. Acad Med. 2021;96:S62-70. https://doi.org/10.1097/ACM.0000000000004291
2. Darcy AM, Louie AK, Roberts LW. Machine learning and the profession of medicine. JAMA. 2016;315:551. https://doi.org/10.1001/jama.2015.18421
3. Hashimoto DA, Johnson KB. The use of Artificial Intelligence tools to prepare medical school applications. Acad Med. 2023;98:978-82. https://doi.org/10.1097/ACM.0000000000005309
4. Boscardin CK, Gin B, Golde PB, Hauer KE. ChatGPT and generative Artificial Intelligence for medical education: potential impact and opportunity. Acad Med. 2023. https://doi.org/10.1097/ACM.0000000000005439
5. Salvatori P. Reliability and Validity of Admissions Tools Used to Select Students for the Health Professions. Adv Health Sci Educ. 2001;6:159-75. https://doi.org/10.1023/A:1011489618208
6. Kulatunga Moruzi C, Norman GR. Validity of admissions measures in predicting performance outcomes: the contribution of cognitive and non-cognitive dimensions. Teach Learn Med. 2002;14:34-42. https://doi.org/10.1207/S15328015TLM1401_9
7. Youdas JW, Hallman HO, Carey JR, Bogard CL, Garrett TR. Reliability and Validity of Judgments of Applicant Essays as a Predictor of Academic Success in an Entry-Level Physical Therapy Education Program. J Phys Ther Educ. 1992;6:15. https://doi.org/10.1097/00001416-199201000-00005
8. Peeters MJ, Schmude KA, Steinmiller CL. Inter-rater reliability and false confidence in precision: Using standard error of measurement within PharmD admissions essay rubric development. Curr Pharm Teach Learn. 2014;6:298-303. https://doi.org/10.1016/j.cptl.2013.11.014
9. MD Program University of Toronto. MD Program University of Toronto non-academic requirements. Available at: https://applymd.utoronto.ca/non-academic-requirements [Accessed on Nov 3, 2024].
10. Schulich Medicine & Dentistry Admissions. Admission requirements. Available at: https://www.schulich.uwo.ca/med_dent_admissions/medicine/admission_requirements.html [Accessed on Nov 3, 2024].
11. Toronto Metropolitan University (TMU) School of Medicine. Admissions requirements. Available at: https://www.torontomu.ca/school-of-medicine/programs/md/ [Accessed on Nov 3, 2024].
12. Dalhousie University Medical School Admissions. Admission requirements. Available at: https://medicine.dal.ca/departments/core-units/admissions/admissions.html [Accessed on Nov 3, 2024].
13. Queen's University School of Medicine. Queen's-Lakeridge Health MD Family Medicine program eligibility & application process. Available at: https://meds.queensu.ca/academics/mdprogram/queens-lakeridge-health-md-family-medicine-program/eligibility_application_process [Accessed on Nov 3, 2024].
14. Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. Assessing personal qualities in medical school admissions: Acad Med. 2003;78:313-21. https://doi.org/10.1097/00001888-200303000-00016
15. Marzuki, Widiati U, Rusdin D, Darwin, Indrawati I. The impact of AI writing tools on the content and organization of students' writing: EFL teachers' perspective. Cogent Educ. 2023;10:2236469. https://doi.org/10.1080/2331186X.2023.2236469
16. Yeadon W, Inyang O-O, Mizouri A, Peach A, Testrow CP. The death of the short-form physics essay in the coming AI revolution. Phys Educ. 2023;58:035027. https://doi.org/10.1088/1361-6552/acc5cf
17. Gao CA, Howard FM, Markov NS, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. Sci Comm Educ; 2022. https://doi.org/10.1101/2022.12.23.521610
18. Herbold S, Hautli-Janisz A, Heuer U, Kikteva Z, Trautsch A. A large-scale comparison of human-written versus ChatGPT-generated essays. Sci Rep. 2023;13:18617. https://doi.org/10.1038/s41598-023-45644-9
19. Ayers JW, Poliak A, Dredze M, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med. 2023;183:589. https://doi.org/10.1001/jamainternmed.2023.1838
20. Waltzer T, Cox RL, Heyman GD. Testing the ability of teachers and students to differentiate between essays generated by ChatGPT and high school students. Hum Behav Emerg Technol. 2023;2023:1923981. https://doi.org/10.1155/2023/1923981
21. Yan D, Fauss M, Hao J, Cui W. Detection of AI-generated essays in writing assessments. Psychol Test Assess Model. 2023;65:125-44.
22. Fleckenstein J, Meyer J, Jansen T, Keller SD, Köller O, Möller J. Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays. Comput Educ Artif Intell. 2024;6:100209. https://doi.org/10.1016/j.caeai.2024.100209
23. Chen D, Parsa R, Hope A, et al. Physician and artificial intelligence chatbot responses to cancer questions from social media. JAMA Oncol. 2024;10:956-60. https://doi.org/10.1001/jamaoncol.2024.0836
24. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)-a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377-81. https://doi.org/10.1016/j.jbi.2008.08.010
25. Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019;95:103208. https://doi.org/10.1016/j.jbi.2019.103208
26. Shaheen MY. Applications of Artificial Intelligence (AI) in healthcare: a review. Sci Prepr. 2021. https://doi.org/10.14293/S2199-1006.1.SOR-.PPVRY8K.v1
27. Dias PP, Jayasinghe LB, Waldmann D. Investigation of mycelium-miscanthus composites as building insulation material. Results Mater. 2021;10:100189. https://doi.org/10.1016/j.rinma.2021.100189
28. White J, Brownell K, Lemay J-F, Lockyer JM. "What do they want me to say?" The hidden curriculum at work in the medical school selection process: a qualitative study. BMC Med Educ. 2012;12:17. https://doi.org/10.1186/1472-6920-12-17
29. Lin KY, Anspach RR, Crawford B, Parnami S, Fuhrel-Forbis A, Vries RGD. What must I do to succeed?: narratives from the US premedical experience. Soc Sci Med 1982. 2014;119:98. https://doi.org/10.1016/j.socscimed.2014.08.017
30. Yang L, Chang I, Ritz S, Grierson L. Research experiences for Canadian aspiring physicians: a descriptive analysis of medical school admission policies concerning research involvement in Canada. BMC Med Educ. 2022;22:151. https://doi.org/10.1186/s12909-022-03207-y
31. Chang I, Yang L, Elma A, Ritz SA, Grierson L. A brief report of aspiring medical student perceptions and behaviours concerning research experiences for selection into Canadian medical schools. Can Med Educ J. 2023;14:77-81. https://doi.org/10.36834/cmej.76255
32. Brown B, Carpio B, Roberts J. The use of an autobiographical letter in the nursing admissions process: initial reliability and validity. Can J Nurs Res Rev Can Rech En Sci Infirm. 1991;23:9-20.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Joshua Choi, Jenny Zhao, Thuy-Anh Ngo, Lawrence Grierson
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.
