Re-evaluating the role of personal statements in pediatric residency admissions in the era of artificial intelligence: comparing faculty ratings of human and AI-generated statements
DOI:
https://doi.org/10.36834/cmej.81345Abstract
Background: Personal statements play a large role in pediatric residency applications, providing insights into candidates’ motivations, experiences, and fit for the program. With large language models (LLMs) such as Chat Generative Pre-trained Transformer (ChatGPT), concerns have arisen regarding how this may influence the authenticity of statements in evaluating candidates. This study investigates the efficacy and perceived authenticity of LLM-generated personal statements compared to human-generated statements in residency applications.
Methods: We conducted a blinded study comparing 30 ChatGPT-generated personal statements with 30 human-written statements. Four pediatric faculty raters assessed each statement using a standardized 10-point rubric. We analyzed the data using linear mixed-effects models, a chi-square sensitivity analysis, an evaluation of rater accuracy in identifying statement origin as well as consistency of scores amongst raters using intraclass correlation coefficients (ICC).
Results: There was no significant difference in mean scores between AI and human-written statements. Raters could only identify the source of a letter (AI or human) with 59% accuracy. There was considerable disagreement in scores between raters as indicated by negative ICCs.
Conclusions: AI-generated statements were rated similarly to human-authored statements and were indistinguishable by reviewers, highlighting the sophistication of these LLM models and the challenge in detecting their use. Furthermore, scores varied substantially between reviewers. As AI becomes increasingly used in application processes, it is imperative to examine its implications in the overall evaluation of applicants.
Downloads
References
1. Whalen A. CaRMS. 2024. Available from https://www.carms.ca/ [Accessed Oct 30, 2024].
2. Dirschl DR. MD. Scoring of orthopaedic residency applicants: Is a scoring system reliable? Clin Orthop Relat Res. 2002;399:260-264. https://doi.org/10.1097/00003086-200206000-00033
3. Hostetter L, Kelm D, Nelson D. Ethics of writing personal statements and letters of recommendations with large language models. ATS Sch. 2024;0038PS. https://doi.org/10.34197/ats-scholar.2024-0038PS
4. Zumsteg JM, Junn C. Will ChatGPT match to your program. Am J Phys Med Rehabil. 2023;102(6):545-547. https://doi.org/10.1097/PHM.0000000000002238
5. White BA, Sadoski M, Thomas S, Shabahang M. Is the evaluation of the personal statement a reliable component of the general surgery residency application? J Surg Educ. 2012;69(3):340-343. https://doi.org/10.1016/j.jsurg.2011.12.003
6. Burke H, Kazinka R, Gandhi R, et al. Artificial intelligence-generated writing in the ERAS personal statement: an emerging quandary for post-graduate medical education. Acad Psychiatry. 2025; 49:13-17. https://doi.org/10.1007/s40596-024-02080-9
7. Patel V, Deleonibus A, Wells MW, Bernard SL, Schwarz GS. Distinguishing authentic voices in the age of ChatGPT: comparing AI-generated and applicant-written personal statements for plastic surgery residency application. Ann Plast Surg. 2023;91(3):324-325. https://doi.org/10.1097/SAP.0000000000003653
8. Whitrock J, Pratt C, Carter M, et al. Does using artificial intelligence take the person out of personal statements? We can't tell. Surg. 2024;176(6):1610-1616. https://doi.org/10.1016/j.surg.2024.08.018
9. Johnstone RE, Neely G, Sizemore DC. Artificial intelligence software can generate residency application personal statements that program directors find acceptable and difficult to distinguish from applicant compositions. J Clin Anesth. 2023;89:111185. https://doi.org/10.1016/j.jclinane.2023.111185
10. Gao CA, Howard FM, Markov N.S., et al. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digit Med. 2023;6(75). https://doi.org/10.1038/s41746-023-00819-6
11. Chen J, Tao BK, Park S, Bovill E. Can ChatGPT fool the match? Artificial intelligence personal statements for plastic surgery residency applications: a comparative study. Plastic Surg. 2024;33(2):348-353. https://doi.org/10.1177/22925503241264832
12. Lum ZC, Guntupalli L, Saiz AM, et al. Can artificial intelligence fool residency selection committees? Analysis of personal statements by real applicants and generative AI, a randomized, single-blind multicenter study. JB JS Open Access. 2024;9(4):e24.00028. https://doi.org/10.2106/JBJS.OA.24.00028
13. Christophers B, Marr MC, Pendergrast TR. Medical school admission policies disadvantage low-income applicants. Perm J. 2022;26(2):172-176. https://doi.org/10.7812/TPP/21.181
14. Shadan M, Chhapra HU, Mashooq FN. Navigating challenges: Supporting non-native speaking medical students with AI and mentorship. Cogent Educ. 2024;12(1). https://doi.org/10.1080/2331186X.2025.2563991
15. Taylor C, Weinstein L, Mayhew H. The process of resident selection: A view from the residency director's desk. Obstet Gynecol. 1995;85(2):299-303. https://doi.org/10.1016/0029-7844(94)00388-T
16. Max BA, Gelfand B, Brooks MR, Beckerly R, Segal S. Have personal statements become impersonal? An evaluation of personal statements in anesthesiology residency applications. J Clin Anesth. 2010;22(5):346-351. https://doi.org/10.1016/j.jclinane.2009.10.007
17. Matsubara, S. Comment on "Artificial intelligence-generated writing in the ERAS personal statement: an emerging quandary for post-graduate medical education". Acad Psych. 2025;49,200-201. https://doi.org/10.1007/s40596-025-02123-9
18. Matsubara S, Matsubara D. Letter regarding: "Digital ink and surgical dreams: perceptions of artificial intelligence-generated essays in residency applications." J Surg Res. 2024;303:797-8. https://doi.org/10.1016/j.jss.2024.08.025
19. Subillaga O, Coulter AP, Tashjian D, Seymour N, Hubbs D. Artificial intelligence-assisted narratives: analysis of surgical residency personal statements. J Surg Educ. 2025;18:103566. https://doi.org/10.1016/j.jsurg.2025.103566
20. Montemayor C, Halpern J, Fairweather A. In principle obstacles for empathic AI: why we can't replace human empathy in healthcare. AI Soc. 2022;37(4):1353-1359. https://doi.org/10.1007/s00146-021-01230-z
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Brittany Curry, Amrit Kirpalani, Mia Remington, Tamara Van Hooren, Ye Shen, Erin Peebles
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.
