Assessment burden by design: exploring the variability in competence by design assessment forms
DOI :
https://doi.org/10.36834/cmej.81428Résumé
Background: The Royal College of Physicians and Surgeons of Canada’s (RCPSC) Competence by Design (CBD) framework has been criticized for increasing assessment burden due to the high number of required Entrustable Professional Activity (EPA) assessments. Another contributing factor may be the inefficient design of assessment forms. We explored variability in form design to identify differences that could impact learners’ and assessors’ experience with CBD.
Methods: Annotated assessment forms for the ‘Core 1’ (C1) EPA were collected in March 2023 from all (n = 14, 100%) RCPSC emergency medicine (EM) residency programs in Canada that had implemented CBD. Forms were divided into six sections to analyze their design. The variability between form sections was described relative to RCPSC guidance on form design.
Results: EPA assessments were completed within six learning management systems. Variability was found throughout the form including: the number of context variables, included milestones, milestone rating criteria, and narrative feedback text boxes; phrasing of narrative feedback prompts, milestone descriptions, and entrustment score criterion; visual presentation of the entrustment score; arrangement of form components; and the components’ selection format. The mandatory completion of form components was inconsistent. Some forms could be partially completed by residents. One form added a global performance rating scale. The number of clicks required to complete a form ranged from 12 to 47.
Conclusion: We found considerable variability in the design of the EM C1 EPA assessment form. Variations that make completion more challenging could increase assessment burden. CBD programs should be aware of this and seek to optimize their form design.
Téléchargements
Références
1. Frank JR, Karpinski J, Sherbino J, et al. Competence by design: a transformational national model of time-variable competency-based postgraduate medical education. Perspect Med Educ. 2024;13(1):201–23. https://doi.org/10.5334/pme.1096
2. van Mil S. Intended and unintended consequences: the impact of competence by design implementation on the experiences of trainees in a pediatric residency program [Thesis]. 2024 Available from: https://macsphere.mcmaster.ca/items/478cf026-ea43-4387-bd86-f51670cc0768 [Accessed on Apr 8, 2025].
3. Ott MC, Pack R, Cristancho S, et al. “The most crushing thing”: understanding resident assessment burden in a competency-based curriculum. J Grad Med Educ. 2022 Oct 14;14(5):583–92. https://doi.org/10.4300/JGME-D-22-00050.1
4. Szulewski A, Braund H, Dagnone DJ, et al. The assessment burden in competency-based medical education: how programs are adapting. Acad Med. 2023 Nov;98(11):1261. https://doi.org/10.1097/ACM.0000000000005305
5. Royal College of Physicians and Surgeons of Canada. Competence by design: resident physician pulse check. Royal College of Physicians and Surgeons of Canada; Sept 2022. Available from: https://www.royalcollege.ca/content/dam/document/standards-and-accreditation/2022-resident-pulse-check-final-report-e.pdf [Accessed on Nov 17, 2025].
6. Tierney AA, Rosner BI. Clinical assessment of residents: a survey of clinician educators regarding resident assessment burden and modifiable factors. J Grad Med Educ. 2023 Feb 1;15(1):92–7. https://doi.org/10.4300/JGME-D-22-00188.1
7. Sweller J, van Merrienboer JJG, Paas FGWC. Cognitive architecture and instructional design. Educ Psychol Rev. Sept 1998;10(3):251–96. https://doi.org/10.1023/A:1022193728205
8. Royal College of Physicians and Surgeons of Canada. Competence by design observation template 1. Royal College of Physicians and Surgeons of Canada; 2017. Available from: https://www.royalcollege.ca/content/dam/documents/ibd/shared/cbd-observation-templates-e.pdf [Accessed on Nov 17, 2025].
9. Royal College of Physicians and Surgeons of Canada. Entrustable professional activity guide: emergency medicine. Royal College of Physicians and Surgeons of Canada; 2018. 2018. Available from: https://www.royalcollege.ca/content/dam/documents/ibd/shared/epa-guide-emergency-med-e.pdf [Accessed on Nov 17, 2025].
10. Royal College of Physicians and Surgeons of Canada. Competence by design technical guide 1: EPA observation forms. Royal College of Physicians and Surgeons of Canada; 2020. Available from: https://cumming.ucalgary.ca/sites/default/files/teams/6/CBD/cbd-technical-guide-1-epa-observation-forms-e%20(1).pdf. [Accessed on Nov 17, 2025].
11. Ismaeel R, Riggs J, Hall A, et al. Assessment burden by design: Exploring variability in assessment forms within competence by design. Int Conf Resid Educ ICRE. 2024;
12. Hall AK, Rich J, Dagnone JD, et al. It’s a marathon, not a sprint: rapid evaluation of competency-based medical education program implementation. Acad Med. 2020 May;95(5):786–93. https://doi.org/10.1097/ACM.0000000000003040
13. Hall AK, Oswald A, Frank JR, et al. Evaluating competence by design as a large system change initiative: readiness, fidelity, and outcomes. Perspect Med Educ. 2024 Feb 6;13(1):95–107. https://doi.org/10.5334/pme.962
14. Frank JR, Snell L, Sherbino J, editors. CanMEDS 2015 physician competency framework. Ottawa: Royal College of Physicians and Surgeons of Canada; 2015.
15. Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med. 2012 Oct;87(10):1401–7. https://doi.org/10.1097/ACM.0b013e3182677805
16. MacEwan MJ, Dudek NL, Wood TJ, Gofton WT. Continued validation of the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation): use on the simulated environment. Teach Learn Med. 2024;16. http://doi.org/10.1080/10401334.2015.1107483
17. Ott M, Apramian T, Cristancho S, Roth K. Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program. J Otolaryngol - Head Neck Surg. 2016 Jan;52(1):s40463-023-00649–2. http://doi.org/10.1186/s40463-023-00649-2
18. Woods R, Bandi V, Thoma B, et al. E-Mail nudges, faculty development workshops and template changes…what impacts the quality of supervisor narrative comments in competency based medical education? Int Conf Resid Educ ICRE Program Eval Summit 2024.
19. Thoma B, Bandi V, Carey R, et al. Developing a dashboard to meet competence committee needs: a design-based research project. Can Med Educ J. 2020 Jan 7 https://doi.org/10.36834/cmej.68903
20. Carey R, Wilson G, Bandi V, et al. Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project. Can Med Educ J. 2020 Sept 14 https://doi.org/10.36834/cmej.69682
21. Yilmaz Y, Carey R, Chan T, et al. Developing a dashboard for faculty development in competency-based training programs: a design-based research project. Can Med Educ J. 2021 June 30 https://doi.org/10.36834/cmej.72067
22. Yilmaz Y, Carey R, Chan T, et al. Developing a dashboard for program evaluation in competency-based training programs: a design-based research project. Can Med Educ J. 2022 Apr 27. https://doi.org/10.36834/cmej.73554
Téléchargements
Publié-e
Comment citer
Numéro
Rubrique
Licence
© Romaisa Ismaeel, Jamie Riggs, Mary Ott, Andrew K Hall, Catherine Patocka, Andrew Petrosoniak, Adam Szulewski, Teresa Chan, Brent Thoma 2025
Cette œuvre est sous licence Creative Commons Attribution - Pas d'Utilisation Commerciale - Pas de Modification 4.0 International.
La soumission d’un manuscrit original à la revue constitue une indication qu’il s’agit d’un travail original, qu’il n’a jamais été publié et qu’il n’est pas envisagé pour publication dans une autre revue. S’il est accepté, il sera publié en ligne et ne pourra l’être ailleurs sous la même forme, à des fins commerciales, dans quelque langue que ce soit, sans l’accord de l’éditeur.
La publication d’une recherche scientifique a pour but la diffusion de connaissances et, sous un régime sans but lucratif, ne profite financièrement ni à l’éditeur ni à l’auteur.
Les auteurs qui publient dans la Revue canadienne d’éducation médicale acceptent de publier leurs articles sous la licence Creative Commons Paternité - Pas d’utilisation commerciale, Pas de modification 4.0 Canada. Cette licence permet à quiconque de télécharger et de partager l’article à des fins non commerciales, à condition d’en attribuer le crédit aux auteurs. Pour plus de détails sur les droits que les auteurs accordent aux utilisateurs de leur travail, veuillez consulter le résumé de la licence et la licence complète.
