Assessment burden by design: exploring the variability in competence by design assessment forms
DOI:
https://doi.org/10.36834/cmej.81428Abstract
Background: The Royal College of Physicians and Surgeons of Canada’s (RCPSC) Competence by Design (CBD) framework has been criticized for increasing assessment burden due to the high number of required Entrustable Professional Activity (EPA) assessments. Another contributing factor may be the inefficient design of assessment forms. We explored variability in form design to identify differences that could impact learners’ and assessors’ experience with CBD.
Methods: Annotated assessment forms for the ‘Core 1’ (C1) EPA were collected in March 2023 from all (n = 14, 100%) RCPSC emergency medicine (EM) residency programs in Canada that had implemented CBD. Forms were divided into six sections to analyze their design. The variability between form sections was described relative to RCPSC guidance on form design.
Results: EPA assessments were completed within six learning management systems. Variability was found throughout the form including: the number of context variables, included milestones, milestone rating criteria, and narrative feedback text boxes; phrasing of narrative feedback prompts, milestone descriptions, and entrustment score criterion; visual presentation of the entrustment score; arrangement of form components; and the components’ selection format. The mandatory completion of form components was inconsistent. Some forms could be partially completed by residents. One form added a global performance rating scale. The number of clicks required to complete a form ranged from 12 to 47.
Conclusion: We found considerable variability in the design of the EM C1 EPA assessment form. Variations that make completion more challenging could increase assessment burden. CBD programs should be aware of this and seek to optimize their form design.
Downloads
References
1. Frank JR, Karpinski J, Sherbino J, et al. Competence by design: a transformational national model of time-variable competency-based postgraduate medical education. Perspect Med Educ. 2024;13(1):201–23. https://doi.org/10.5334/pme.1096
2. van Mil S. Intended and unintended consequences: the impact of competence by design implementation on the experiences of trainees in a pediatric residency program [Thesis]. 2024 Available from: https://macsphere.mcmaster.ca/items/478cf026-ea43-4387-bd86-f51670cc0768 [Accessed on Apr 8, 2025].
3. Ott MC, Pack R, Cristancho S, et al. “The most crushing thing”: understanding resident assessment burden in a competency-based curriculum. J Grad Med Educ. 2022 Oct 14;14(5):583–92. https://doi.org/10.4300/JGME-D-22-00050.1
4. Szulewski A, Braund H, Dagnone DJ, et al. The assessment burden in competency-based medical education: how programs are adapting. Acad Med. 2023 Nov;98(11):1261. https://doi.org/10.1097/ACM.0000000000005305
5. Royal College of Physicians and Surgeons of Canada. Competence by design: resident physician pulse check. Royal College of Physicians and Surgeons of Canada; Sept 2022. Available from: https://www.royalcollege.ca/content/dam/document/standards-and-accreditation/2022-resident-pulse-check-final-report-e.pdf [Accessed on Nov 17, 2025].
6. Tierney AA, Rosner BI. Clinical assessment of residents: a survey of clinician educators regarding resident assessment burden and modifiable factors. J Grad Med Educ. 2023 Feb 1;15(1):92–7. https://doi.org/10.4300/JGME-D-22-00188.1
7. Sweller J, van Merrienboer JJG, Paas FGWC. Cognitive architecture and instructional design. Educ Psychol Rev. Sept 1998;10(3):251–96. https://doi.org/10.1023/A:1022193728205
8. Royal College of Physicians and Surgeons of Canada. Competence by design observation template 1. Royal College of Physicians and Surgeons of Canada; 2017. Available from: https://www.royalcollege.ca/content/dam/documents/ibd/shared/cbd-observation-templates-e.pdf [Accessed on Nov 17, 2025].
9. Royal College of Physicians and Surgeons of Canada. Entrustable professional activity guide: emergency medicine. Royal College of Physicians and Surgeons of Canada; 2018. 2018. Available from: https://www.royalcollege.ca/content/dam/documents/ibd/shared/epa-guide-emergency-med-e.pdf [Accessed on Nov 17, 2025].
10. Royal College of Physicians and Surgeons of Canada. Competence by design technical guide 1: EPA observation forms. Royal College of Physicians and Surgeons of Canada; 2020. Available from: https://cumming.ucalgary.ca/sites/default/files/teams/6/CBD/cbd-technical-guide-1-epa-observation-forms-e%20(1).pdf. [Accessed on Nov 17, 2025].
11. Ismaeel R, Riggs J, Hall A, et al. Assessment burden by design: Exploring variability in assessment forms within competence by design. Int Conf Resid Educ ICRE. 2024;
12. Hall AK, Rich J, Dagnone JD, et al. It’s a marathon, not a sprint: rapid evaluation of competency-based medical education program implementation. Acad Med. 2020 May;95(5):786–93. https://doi.org/10.1097/ACM.0000000000003040
13. Hall AK, Oswald A, Frank JR, et al. Evaluating competence by design as a large system change initiative: readiness, fidelity, and outcomes. Perspect Med Educ. 2024 Feb 6;13(1):95–107. https://doi.org/10.5334/pme.962
14. Frank JR, Snell L, Sherbino J, editors. CanMEDS 2015 physician competency framework. Ottawa: Royal College of Physicians and Surgeons of Canada; 2015.
15. Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med. 2012 Oct;87(10):1401–7. https://doi.org/10.1097/ACM.0b013e3182677805
16. MacEwan MJ, Dudek NL, Wood TJ, Gofton WT. Continued validation of the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation): use on the simulated environment. Teach Learn Med. 2024;16. http://doi.org/10.1080/10401334.2015.1107483
17. Ott M, Apramian T, Cristancho S, Roth K. Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program. J Otolaryngol - Head Neck Surg. 2016 Jan;52(1):s40463-023-00649–2. http://doi.org/10.1186/s40463-023-00649-2
18. Woods R, Bandi V, Thoma B, et al. E-Mail nudges, faculty development workshops and template changes…what impacts the quality of supervisor narrative comments in competency based medical education? Int Conf Resid Educ ICRE Program Eval Summit 2024.
19. Thoma B, Bandi V, Carey R, et al. Developing a dashboard to meet competence committee needs: a design-based research project. Can Med Educ J. 2020 Jan 7 https://doi.org/10.36834/cmej.68903
20. Carey R, Wilson G, Bandi V, et al. Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project. Can Med Educ J. 2020 Sept 14 https://doi.org/10.36834/cmej.69682
21. Yilmaz Y, Carey R, Chan T, et al. Developing a dashboard for faculty development in competency-based training programs: a design-based research project. Can Med Educ J. 2021 June 30 https://doi.org/10.36834/cmej.72067
22. Yilmaz Y, Carey R, Chan T, et al. Developing a dashboard for program evaluation in competency-based training programs: a design-based research project. Can Med Educ J. 2022 Apr 27. https://doi.org/10.36834/cmej.73554
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Romaisa Ismaeel, Jamie Riggs, Mary Ott, Andrew K Hall, Catherine Patocka, Andrew Petrosoniak, Adam Szulewski, Teresa Chan, Brent Thoma
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.
