The Quality of Assessment for Learning score for evaluating written feedback in anesthesiology postgraduate medical education: a generalizability and decision study

Authors

DOI:

https://doi.org/10.36834/cmej.75876

Abstract

Background: Competency based residency programs depend on high quality feedback from the assessment of entrustable professional activities (EPA). The Quality of Assessment for Learning (QuAL) score is a tool developed to rate the quality of narrative comments in workplace-based assessments; it has validity evidence for scoring the quality of narrative feedback provided to emergency medicine residents, but it is unknown whether the QuAL score is reliable in the assessment of narrative feedback in other postgraduate programs.

Methods: Fifty sets of EPA narratives from a single academic year at our competency based medical education post-graduate anesthesia program were selected by stratified sampling within defined parameters [e.g. resident gender and stage of training, assessor gender, Competency By Design training level, and word count (≥17 or <17 words)]. Two competency committee members and two medical students rated the quality of narrative feedback using a utility score and QuAL score. We used Kendall’s tau-b co-efficient to compare the perceived utility of the written feedback to the quality assessed with the QuAL score. The authors used generalizability and decision studies to estimate the reliability and generalizability coefficients.

Results: Both the faculty’s utility scores and QuAL scores (r = 0.646, p < 0.001) and the trainees’ utility scores and QuAL scores (r = 0.667, p < 0.001) were moderately correlated. Results from the generalizability studies showed that utility scores were reliable with two raters for both faculty (Epsilon=0.87, Phi=0.86) and trainees (Epsilon=0.88, Phi=0.88).

Conclusions: The QuAL score is correlated with faculty- and trainee-rated utility of anesthesia EPA feedback. Both faculty and trainees can reliability apply the QuAL score to anesthesia EPA narrative feedback. This tool has the potential to be used for faculty development and program evaluation in Competency Based Medical Education. Other programs could consider replicating our study in their specialty.

Metrics

Metrics Loading ...

Author Biographies

Rob Woods, University of Saskatchewan

Department of Emergency Medicine, College of Medicine, University of Saskatchewan, Saskatoon, SK, Canada

Mary Ellen Walker, University of Saskatchewan

Department of Anesthesiology, College of Medicine, University of Saskatchewan, Saskatoon, SK, Canada

Jennifer M O'Brien, University of Saskatchewan

Department of Anesthesiology, College of Medicine, University of Saskatchewan, Saskatoon, SK, Canada

Teresa M Chan, McMaster University

Department of Medicine (Division of Emergency Medicine; Division of Education & Innovation), Michael G. DeGroote School of Medicine, Faculty of Health Sciences, McMaster University. Office of Continuing Professional Development & McMaster Education Research, Innovation, and Theory (MERIT) Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada.

References

Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–45. https://doi.org/10.3109/0142159x.2010.501190 DOI: https://doi.org/10.3109/0142159X.2010.501190

Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010 Aug 1;32(8):676–82. https://doi.org/10.3109/0142159x.2010.500704 DOI: https://doi.org/10.3109/0142159X.2010.500704

Caverzagie KJ, Nousiainen MT, Ferguson PC, et al. Overarching challenges to the implementation of competency-based medical education. Med Teach. 2017 Jun 3;39(6):588–93. https://doi.org/10.1080/0142159x.2017.1315075 DOI: https://doi.org/10.1080/0142159X.2017.1315075

Jensen AR, Wright AS, Kim S, Horvath KD, Calhoun KE. Educational feedback in the operating room: a gap between resident and faculty perceptions. Amer J Surg. 2012 Aug 1;204(2):248–55. https://doi.org/10.1016/j.amjsurg.2011.08.019 DOI: https://doi.org/10.1016/j.amjsurg.2011.08.019

Upadhyaya S, Rashid M, Davila-Cervantes A, Oswald A. Exploring resident perceptions of initial competency based medical education implementation. Can Med Educ J. 2021 Apr;12(2):e42–56. https://doi.org/10.36834/cmej.70943 DOI: https://doi.org/10.36834/cmej.70943

Weller JM, Naik VN, Diego RJS. Systematic review and narrative synthesis of competency-based medical education in anaesthesia. Brit J Anaesthesia. 2020 Jun 1;124(6):748–60. https://doi.org/10.1016/j.bja.2019.10.025 DOI: https://doi.org/10.1016/j.bja.2019.10.025

Yilmaz Y, Carey R, Chan T et al. Developing a dashboard for faculty development in competency-based training programs: a design-based research project. Can Med Ed J. 2021 Sep 15;12(4):48–64. https://doi.org/10.36834/cmej.72067

Chan TM, Sebok-Syer SS, Sampson C, Monteiro S. The Quality of Assessment of Learning (Qual) Score: validity evidence for a scoring system aimed at rating short, workplace-based comments on trainee performance. Teach Learn Med. 2020 Jul;32(3):319–29. https://doi.org/10.1080/10401334.2019.1708365 DOI: https://doi.org/10.1080/10401334.2019.1708365

Woods RA, Singh S, Thoma B, et al. Validity evidence for the QuAL (Quality of Assessment for Learning) score: a quality metric for supervisor comments in Competency Based Medical Education. Can Med Ed J. 2022; 13(6); 19-35. https://doi.org/10.36834/cmej.74860 DOI: https://doi.org/10.36834/cmej.74860

St-Onge C, Young M, Eva KW, Hodges B. Validity: one word with a plurality of meanings. Adv in Health Sci Educ. 2017 Oct;22(4):853–67. https://doi.org/10.1007%2Fs40037-018-0433-x DOI: https://doi.org/10.1007/s10459-016-9716-3

Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015 Jun;49(6):560–75. https://doi.org/10.1111/medu.12678 DOI: https://doi.org/10.1111/medu.12678

Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. 4th ed. Oxford: Oxford University Press; 2008. 452 p. Available from: https://oxford.universitypressscholarship.com/10.1093/acprof:oso/9780199231881.001.0001/acprof-9780199231881 [Accessed on Oct 3, 2021].

Field A. Discovering statistics using SPSS. 3rd ed., SAGE Publications.2009

Streiner DL. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess. 2003 Feb;80(1):99-103. https://doi.org/10.1207/S15327752JPA8001_18 DOI: https://doi.org/10.1207/S15327752JPA8001_18

Telio S, Regehr G, Ajjawi R. Feedback and the educational alliance: examining credibility judgements and their consequences. Med Educ. 2016 Sep;50(9):933–42. https://doi.org/10.1111/medu.13063 DOI: https://doi.org/10.1111/medu.13063

Acai A, Li SA, Sherbino J, Chan TM. Attending emergency physicians’ perceptions of a programmatic workplace-based assessment system: the McMaster Modular Assessment Program (McMAP). Teach Learn Med. 2019 Aug 8;31(4):434-44. https://doi.org/10.1080/10401334.2019.1574581 DOI: https://doi.org/10.1080/10401334.2019.1574581

Chan T, Oswald A, Hauer KE, et al. Diagnosing conflict: conflicting data, interpersonal conflict, and conflicts of interest in clinical competency committees. Med Teach. 2021 Jul 3;43(7):765-73. https://doi.org/10.1080/0142159x.2021.1925101 DOI: https://doi.org/10.1080/0142159X.2021.1925101

Dudek N, Dojeiji S. Twelve tips for completing quality in-training evaluation reports. Med Teach. 2014 Dec;36(12):1038–42. https://doi.org/10.3109/0142159x.2014.932897 DOI: https://doi.org/10.3109/0142159X.2014.932897

Gray JD. Global rating scales in residency education. Acad Med. 1996 Jan;71(1 Suppl):S55-63. https://doi.org/10.1097/00001888-199601000-00043 DOI: https://doi.org/10.1097/00001888-199601000-00043

Hatala R, Sawatsky AP, Dudek N, Ginsburg S, Cook DA. Using In-Training Evaluation Report (ITER) qualitative comments to assess medical students and residents: a systematic review. Acad Med. 2017 Jun 1;92(6):868–79. https://doi.org/10.1097/acm.0000000000001506 DOI: https://doi.org/10.1097/ACM.0000000000001506

Ginsburg S, van der Vleuten C, Eva KW, Lingard L. Hedging to save face: a linguistic analysis of written comments on in-training evaluation reports. Adv Health Sci Educ. 2016 Mar;21:175-88. https://doi.org/10.1007/s40037-021-00681-w DOI: https://doi.org/10.1007/s10459-015-9622-0

Ginsburg S, Regehr G, Lingard L, Eva KW. Reading between the lines: faculty interpretations of narrative evaluation comments. Med Ed. 2015 Mar;49(3):296-306. https://doi.org/10.1111/medu.12637 DOI: https://doi.org/10.1111/medu.12637

Ginsburg S, Eva K, Regehr G. Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Acad Med. 2013 Oct 1;88(10):1539-44. https://doi.org/10.1097/acm.0b013e3182a36c3d DOI: https://doi.org/10.1097/ACM.0b013e3182a36c3d

Ginsburg S, van der Vleuten CP, Eva KW, Lingard L. Cracking the code: residents’ interpretations of written assessment comments. Med Ed. 2017 Apr;51(4):401-10. https://doi.org/10.1111/medu.13158 DOI: https://doi.org/10.1111/medu.13158

Ginsburg S, Watling CJ, Schumacher DJ, Gingerich A, Hatala R. Numbers encapsulate, words elaborate: toward the best use of comments for assessment and feedback on entrustment ratings. Acad Med. 2021 Jul 1;96(7S):S81-6. https://doi.org/10.1097/acm.0000000000004089 DOI: https://doi.org/10.1097/ACM.0000000000004089

Li SA, Sherbino J, Chan TM. McMaster Modular Assessment Program (McMAP) through the years: residents' experience with an evolving feedback culture over a 3‐year period. AEM Educ Training. 2017 Jan;1(1):5-14. http://dx.doi.org/10.1002/aet2.10009 DOI: https://doi.org/10.1002/aet2.10009

Chan TM, Sherbino J, Mercuri M. Nuance and noise: lessons learned from longitudinal aggregated assessment data. J Grad Med Ed. 2017 Dec;9(6):724-9. https://doi.org/10.4300/jgme-d-17-00086.1 DOI: https://doi.org/10.4300/JGME-D-17-00086.1

Sebok-Syer SS, Klinger DA, Sherbino J, Chan TM. “It’s complicated”: understanding the relationships between checklists, rating scales, and written comments in workplace-based assessments. Acad Med. 2016 Nov 1;91(11):S10. https://doi.org/10.1097/ACM.000000000001373 DOI: https://doi.org/10.1097/ACM.0000000000001373

Yilmaz Y, Carey R, Chan TM et al. Developing a dashboard for faculty development in competency-based training programs: a design-based research project. Can Med Ed J. 2021 Oct 20;12(4):48-64. https://doi.org/10.36834/cmej.72067 DOI: https://doi.org/10.36834/cmej.72067

Miller J, Katz D. Gender differences in perception of workplace experience among anesthesiology residents. JEPM. 2018 Jan;20(1). DOI: https://doi.org/10.46374/volxx-issue1-miller

Pearce G, Sidhu N, Cavadino A, Shrivathsa A, Seglenieks R. Gender effects in anaesthesia training in Australia and New Zealand. Brit j anaesthesia. 2020 Mar 1;124(3):e70-6. https://doi.org/10.1016/j.bja.2019.12.020 DOI: https://doi.org/10.1016/j.bja.2019.12.020

Dayal A, O’Connor DM, Qadri U, Arora VM. Comparison of male vs female resident milestone evaluations by faculty during emergency medicine residency training. JAMA intern med. 2017 May 1;177(5):651-7. https://doi.org/10.1001/jamainternmed.2016.9616 DOI: https://doi.org/10.1001/jamainternmed.2016.9616

Mamtani M, Shofer F, Scott K, et al. Gender differences in emergency medicine attending physician comments to residents: a qualitative analysis. JAMA Network Open. 2022 Nov 1;5(11):e2243134-. https://doi.org/10.1001/jamanetworkopen.2022.43134 DOI: https://doi.org/10.1001/jamanetworkopen.2022.43134

Menchetti I, Eagles D, Ghanem D, Leppard J, Fournier K, Cheung WJ. Gender differences in emergency medicine resident assessment: a scoping review. AEM Educ Training. 2022 Oct;6(5):e10808. https://doi.org/10.1007/s11606-019-04884-0 DOI: https://doi.org/10.1002/aet2.10808

Santen SA, Yamazaki K, Holmboe ES, Yarris LM, Hamstra SJ. Comparison of male and female resident milestone assessments during emergency medicine residency training: a national study. Acad Med. 2020 Feb;95(2):263. https://doi.org/10.1097/acm.0000000000002988 DOI: https://doi.org/10.1097/ACM.0000000000002988

Downloads

Published

2023-12-30

How to Cite

1.
Choo EK, Woods R, Walker ME, O’Brien JM, Chan TM. The Quality of Assessment for Learning score for evaluating written feedback in anesthesiology postgraduate medical education: a generalizability and decision study . Can. Med. Ed. J [Internet]. 2023 Dec. 30 [cited 2024 Oct. 5];14(6):78-85. Available from: https://journalhosting.ucalgary.ca/index.php/cmej/article/view/75876

Issue

Section

Brief Reports

Most read articles by the same author(s)