OSCEai: personalized interactive learning for undergraduate medical education
DOI:
https://doi.org/10.36834/cmej.79220Abstract
Background: This study aims to evaluate the effectiveness of the OSCEai, a large language model-based platform that simulates clinical encounters, in enhancing undergraduate medical education.
Methods: A web-based application, OSCEai, was developed to bridge theoretical and practical learning. Following use, medical students from the University of Calgary Class of 2026 completed an anonymized survey on the usability, utility, and overall experience of OSCEai.
Results: A total of 37 respondents answered the anonymized survey. The OSCEai platform was highly valued for its ability to provide data on demand (33/37), support self-paced learning (30/37), and offer realistic patient interactions (29/37). The ease of use and medical content quality were rated at 4.73 (95% CI: 4.58 to 4.88) and 4.70 (95% CI: 4.55 to 4.86) out of 5, respectively. Some participants (8/37) commented that few cases were not representative and needed clarification about app functionality. Despite these limitations, OSCEai was favorably compared to lecture-based teaching methods, with an overall reception rating of 4.62 (95% CI: 4.46 to 4.79) out of 5.
Interpretation: The OSCEai platform fills a gap in medical training through its scalable, interactive, and personalized design. The findings suggest that integrating technologies, like OSCEai, into medical curricula can enhance the quality and efficacy of medical education.
References
Servant-Miklos VFC. Fifty years on: a retrospective on the world’s first problem-based learning programme at McMaster University Medical School. Health Prof Educ. 2019 Mar;5(1):3–12. http://dx.doi.org/10.1016/j.hpe.2018.04.002 DOI: https://doi.org/10.1016/j.hpe.2018.04.002
Neville AJ, Norman GR. PBL in the undergraduate MD program at McMaster University: three iterations in three decades: Acad Med. 2007 Apr;82(4):370–4. http://dx.doi.org/10.1097/ACM.0b013e318033385d DOI: https://doi.org/10.1097/ACM.0b013e318033385d
Spaulding WB. The undergraduate medical curriculum (1969 model): McMaster university. Can Med Assoc J. 1969 Apr 12;100(14):659–64. https://www.ncbi.nlm.nih.gov/pubmed/5776441
Sanson-Fisher R, Hobden B, Carey M, Mackenzie L, Hyde L, Shepherd J. Interactional skills training in undergraduate medical education: ten principles for guiding future research. BMC Med Educ. 2019 May 15;19(1):144. http://dx.doi.org/10.1186/s12909-019-1566-2 DOI: https://doi.org/10.1186/s12909-019-1566-2
Kurtz SM, Silverman JD. The Calgary-Cambridge Referenced Observation Guides: an aid to defining the curriculum and organizing the teaching in communication training programmes. Med Educ. 1996 Mar;30(2):83–9. http://dx.doi.org/10.1111/j.1365-2923.1996.tb00724.x DOI: https://doi.org/10.1111/j.1365-2923.1996.tb00724.x
Gordon MS, Ewy GA, DeLeon AC Jr, et al. “Harvey,” the cardiology patient simulator: pilot studies on teaching effectiveness. Am J Cardiol. 1980 Apr;45(4):791–6. http://dx.doi.org/10.1016/0002-9149(80)90123-x DOI: https://doi.org/10.1016/0002-9149(80)90123-X
Wang S, Ren X, Ye J, Wang W, Huang H, Qin C. Exploration of simulation-based medical education for undergraduate students. Med. 2021 May 21;100(20):e25982. http://dx.doi.org/10.1097/MD.0000000000025982 DOI: https://doi.org/10.1097/MD.0000000000025982
Al-Elq A. Simulation-based medical teaching and learning. J Fam Community Med. 2010;17(1):35. http://dx.doi.org/10.4103/1319-1683.68787 DOI: https://doi.org/10.4103/1319-1683.68787
Kaplovitch E, Otremba M, Morgan M, Devine LA. Cost-efficient medical education: an innovative approach to creating educational products. J Grad Med Educ. 2019 Dec 1;11(6):713–6. http://dx.doi.org/10.4300/JGME-D-19-00155.1 DOI: https://doi.org/10.4300/JGME-D-19-00155.1
Wynter L, Burgess A, Kalman E, Heron JE, Bleasel J. Medical students: what educational resources are they using? BMC Med Educ. 2019 Dec;19(1):36. http://dx.doi.org/10.1186/s12909-019-1462-9 DOI: https://doi.org/10.1186/s12909-019-1462-9
Abd-Alrazaq A, AlSaad R, Alhuwail D, et al. large language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ. 2023 Jun 1;9:e48291. http://dx.doi.org/10.2196/48291 DOI: https://doi.org/10.2196/48291
Clusmann J, Kolbinger FR, Muti HS, et al. The future landscape of large language models in medicine. Commun Med (Lond). 2023 Oct 10;3(1):141. http://dx.doi.org/10.1038/s43856-023-00370-1 DOI: https://doi.org/10.1038/s43856-023-00370-1
Wu S, Irsoy O, Lu S, et al. BloombergGPT: a large language model for finance. arXiv [cs.LG]. 2023. http://arxiv.org/abs/2303.17564
Guo E, Gupta M, Deng J, Park Y-J, Paget M, Naugler C. Automated paper screening for clinical reviews using large language models: data analysis study. J Med Internet Res. 2024 Jan 12;26:e48996. http://dx.doi.org/10.2196/48996 DOI: https://doi.org/10.2196/48996
Unlu O, Shin J, Mailly CJ, et al. Retrieval-augmented generation–enabled GPT-4 for clinical trial screening. NEJM AI. 2024 Jun 17; http://dx.doi.org/10.1056/aioa2400181 DOI: https://doi.org/10.1056/AIoa2400181
Katz DM, Bommarito MJ, Gao S, Arredondo P. GPT-4 passes the bar exam. Philos Trans A Math Phys Eng Sci. 2024 Apr 15;382(2270):20230254. http://dx.doi.org/10.1098/rsta.2023.0254 DOI: https://doi.org/10.1098/rsta.2023.0254
Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya. Improving language understanding by generative pre-training. 2018.
Biri SK, Kumar S, Panigrahi M, Mondal S, Behera JK, Mondal H. Assessing the utilization of large language models in medical education: insights from undergraduate medical students. Cureus. 2023 Oct;15(10):e47468. http://dx.doi.org/10.7759/cureus.47468 DOI: https://doi.org/10.7759/cureus.47468
Safranek CW, Sidamon-Eristoff AE, Gilson A, Chartash D. The role of large language models in medical education: applications and implications. JMIR Med Educ. 2023 Aug 14;9:e50945. http://dx.doi.org/10.2196/50945 DOI: https://doi.org/10.2196/50945
Divito CB, Katchikian BM, Gruenwald JE, Burgoon JM. The tools of the future are the challenges of today: the use of ChatGPT in problem-based learning medical education. Med Teach. 2024 Mar;46(3):320–2. http://dx.doi.org/10.1080/0142159X.2023.2290997 DOI: https://doi.org/10.1080/0142159X.2023.2290997
Cumming School of Medicine. Re-Imagining Medical Education (RIME) 2024. Available from https://cumming.ucalgary.ca/mdprogram/current-students/pre-clerkship-year-1-2/re-imagining-medical-education-rime.
Egli A. ChatGPT, GPT-4, and other large language models: the next revolution for clinical microbiology? Clin Infect Dis. 2023 Nov 11;77(9):1322–8. http://dx.doi.org/10.1093/cid/ciad407 DOI: https://doi.org/10.1093/cid/ciad407
Zayyan M. Objective structured clinical examination: the assessment of choice. Oman Med J. 2011 Jul;26(4):219–22. http://dx.doi.org/10.5001/omj.2011.55 DOI: https://doi.org/10.5001/omj.2011.55
meta-llama/Meta-Llama-3-70B · Hugging Face. Available from https://huggingface.co/meta-llama/Meta-Llama-3-70B [Accessed on Jun 23, 2024].
Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004 Sep 29;6(3):e34. http://dx.doi.org/10.2196/jmir.6.3.e34 DOI: https://doi.org/10.2196/jmir.6.3.e34
Kulasegaram KM, Martimianakis MA, Mylopoulos M, Whitehead CR, Woods NN. Cognition before curriculum: rethinking the integration of basic science and clinical learning. Acad Med. 2013 Oct;88(10):1578–85. http://dx.doi.org/10.1097/ACM.0b013e3182a45def DOI: https://doi.org/10.1097/ACM.0b013e3182a45def
Bosse HM, Nickel M, Huwendiek S, Schultz JH, Nikendei C. Cost-effectiveness of peer role play and standardized patients in undergraduate communication training. BMC Med Educ. 2015 Oct 24;15:183. http://dx.doi.org/10.1186/s12909-015-0468-1 DOI: https://doi.org/10.1186/s12909-015-0468-1
Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011 Sep 7;306(9):978–88. http://dx.doi.org/10.1001/jama.2011.1234 DOI: https://doi.org/10.1001/jama.2011.1234
Walsh K, Levin H, Jaye P, Gazzard J. Cost analyses approaches in medical education: there are no simple solutions. Med Educ. 2013 Oct;47(10):962–8. http://dx.doi.org/10.1111/medu.12214 DOI: https://doi.org/10.1111/medu.12214
Emanuel EJ. The inevitable reimagining of medical education. JAMA 2020 Mar 24;323(12):1127. http://dx.doi.org/10.1001/jama.2020.1227 DOI: https://doi.org/10.1001/jama.2020.1227
McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011 Jun;86(6):706–11. http://dx.doi.org/10.1097/ACM.0b013e318217e119 DOI: https://doi.org/10.1097/ACM.0b013e318217e119
Kneebone RL, Scott W, Darzi A, Horrocks M. Simulation and clinical practice: strengthening the relationship. Med Educ. 2004 Oct;38(10):1095–102. http://dx.doi.org/10.1111/j.1365-2929.2004.01959.x DOI: https://doi.org/10.1111/j.1365-2929.2004.01959.x
Skelly K, Kim P, Rosenbaum M, Wilbur J. Goldilocks and entrustment: finding the amount of learner autonomy that’s just right. MedEdPORTAL. 2020 Oct 13;16:10987. http://dx.doi.org/10.15766/mep_2374-8265.10987 DOI: https://doi.org/10.15766/mep_2374-8265.10987
Conner SM, Choi N, Fuller J, et al. Trainee autonomy and supervision in the modern clinical learning environment: a mixed-methods study of faculty and trainee perspectives. Res Sq. 2023 Jun 6;rs.3.rs-2982838. http://dx.doi.org/10.21203/rs.3.rs-2982838/v1 DOI: https://doi.org/10.21203/rs.3.rs-2982838/v1
Kamphuis C, Barsom E, Schijven M, Christoph N. Augmented reality in medical education? Perspect Med Educ. 2014 Sep;3(4):300–11. http://dx.doi.org/10.1007/s40037-013-0107-7 DOI: https://doi.org/10.1007/S40037-013-0107-7
Thistlethwaite J. Interprofessional education: a review of context, learning and the research agenda. Med Educ. 2012 Jan;46(1):58–70. http://dx.doi.org/10.1111/j.1365-2923.2011.04143.x DOI: https://doi.org/10.1111/j.1365-2923.2011.04143.x
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Eddie Guo, Rashi Ramchandani, Ye-Jean Park, Mehul Gupta
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.
