Simulation vs workplace-based assessment in resuscitation: a cross-specialty descriptive analysis and comparison
DOI:
https://doi.org/10.36834/cmej.73692Abstract
Background: Simulation-based assessment can complement workplace-based assessment of rare or difficult to assess Entrustable Professional Activities (EPAs). We aimed to compare the use of simulation-based assessment for resuscitation-focused EPAs in three postgraduate medical training programs and describe faculty perceptions of simulation-based assessment.
Methods: EPA assessment scores and setting (simulation or workplace) were extracted from 2017-2020 for internal medicine, emergency medicine, and surgical foundations residents at the transition to discipline and foundations of discipline stages. A questionnaire was distributed to clinical competency committee members.
Results: Eleven percent of EPA assessments were simulation-based. The proportion of simulation-based assessment did not differ between programs but differed between transition (38%) and foundations (4%) stages within surgical foundations only. Entrustment scores differed between settings in emergency medicine at the transition level only (simulation: 4.82 ± 0.60 workplace: 3.74 ± 0.93). 70% of committee members (n=20) completed the questionnaire. Of those that use simulation-based assessment, 45% interpret them differently than workplace-based assessments. 73% and 100% trust simulation for high-stakes and low-stakes assessment, respectively.
Conclusions: The proportion of simulation-based assessment for resuscitation focused EPAs did not differ between three postgraduate medical training programs. Interpretation of simulation-based assessment data between committee members was inconsistent. All respondents trust simulation-based assessment for low-stakes, and the majority for high-stakes assessment. These findings have practical implications for the integration simulation into programs of assessment.
Metrics
References
Busing N, Harris K, MacLellan AM, et al. The future of postgraduate medical education in Canada. Acad Med J Assoc Am Med Coll. 2015;90(9):1258-1263. https://doi.org/10.1097/ACM.0000000000000815 DOI: https://doi.org/10.1097/ACM.0000000000000815
Frank JR, Snell LS, Sherbino J. CanMEDS 2015 Physician Competency Framework. Royal College of Physicians and Surgeons of Canada; 2015.
Harris P, Bhanji F, Topps M, et al. Evolving concepts of assessment in a competency-based world. Med Teach. 2017;39(6):603-608. https://doi.org/10.1080/0142159X.2017.1315071 DOI: https://doi.org/10.1080/0142159X.2017.1315071
Norcini J, Anderson MB, Bollela V, et al. 2018 Consensus framework for good assessment. Med Teach. 2018;40(11):1102-1109. https://doi.org/10.1080/0142159X.2018.1500016 DOI: https://doi.org/10.1080/0142159X.2018.1500016
Thoma B, Hall AK, Clark K, et al. Evaluation of a national competency-based assessment system in emergency medicine: a CanDREAM study. J Grad Med Educ. 2020;12(4):425-434. https://doi.org/10.4300/JGME-D-19-00803.1 DOI: https://doi.org/10.4300/JGME-D-19-00803.1
Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR, Collaborators for the IC. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682. https://doi.org/10.3109/0142159X.2010.500704 DOI: https://doi.org/10.3109/0142159X.2010.500704
Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality. Acad Med. 2013;88(6):872-883. https://doi.org/10.1097/ACM.0b013e31828ffdcf DOI: https://doi.org/10.1097/ACM.0b013e31828ffdcf
Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015;49(6):560-575. https://doi.org/10.1111/medu.12678 DOI: https://doi.org/10.1111/medu.12678
McMurray L, Hall AK, Rich J, Merchant S, Chaplin T. The nightmares course: a longitudinal, multidisciplinary, simulation-based curriculum to train and assess resident competence in resuscitation. J Grad Med Educ. 2017;9(4):503-508. https://doi.org/10.4300/JGME-D-16-00462.1 DOI: https://doi.org/10.4300/JGME-D-16-00462.1
Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med J Assoc Am Med Coll. 2012;87(10):1401-1407. https://doi.org/10.1097/ACM.0b013e3182677805 DOI: https://doi.org/10.1097/ACM.0b013e3182677805
Peters H, Holzhausen Y, Boscardin C, Cate O ten, Chen HC. Twelve tips for the implementation of EPAs for assessment and entrustment decisions. Med Teach. 2017;39(8):802-807. https://doi.org/10.1080/0142159X.2017.1331031 DOI: https://doi.org/10.1080/0142159X.2017.1331031
Russell E, Hall AK, Hagel C, Petrosoniak A, Dagnone JD, Howes D. Simulation in Canadian postgraduate emergency medicine training – a national survey. Can J Emerg Med. 2018;20(1):132-141. https://doi.org/10.1017/cem.2017.24 DOI: https://doi.org/10.1017/cem.2017.24
Prudhomme N, O’Brien M, McConnell MM, Dudek N, Cheung WJ. Relationship between ratings of performance in the simulated and workplace environments among emergency medicine residents. Can J Emerg Med. Published online 2020:1-8. https://doi.org/10.1017/cem.2020.388 DOI: https://doi.org/10.1017/cem.2020.388
Weersink K, Hall AK, Rich J, Szulewski A, Dagnone JD. Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring. Adv Simul. 2019;4. https://doi.org/10.1186/s41077-019-0099-4 DOI: https://doi.org/10.1186/s41077-019-0099-4
Hall AK, Damon Dagnone J, Moore S, et al. Comparison of simulation‐based resuscitation performance assessments with in‐training evaluation reports in emergency medicine residents: a canadian multicenter study. AEM Educ Train. 2017;1(4):293-300. https://doi.org/10.1002/aet2.10055 DOI: https://doi.org/10.1002/aet2.10055
Calhoun AW, Bhanji F, Sherbino J, Hatala R. Simulation for high-stakes assessment in pediatric emergency medicine. Clin Pediatr Emerg Med. 2016;17(3):212-223. https://doi.org/10.1016/j.cpem.2016.05.001 DOI: https://doi.org/10.1016/j.cpem.2016.05.001
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Jeremy D Seed, Stephen Gauthier, Boris Zevin, Andrew K Hall, Timothy Chaplin
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.
