Use the right words: evaluating the effect of word choice and word count on quality of narrative feedback in ophthalmology competency-based medical education assessments

Rachel Curtis; Christine C Moon; Tessa Hanmore; Wilma M Hopman; Stephanie Baxter

doi:10.36834/cmej.76671

Authors

Rachel Curtis Queen's University
Christine C Moon Queen's University
Tessa Hanmore Queen's University
Wilma M Hopman Queen's University
Stephanie Baxter Queen's University

DOI:

https://doi.org/10.36834/cmej.76671

Abstract

Background: The purpose of this study was to investigate the effect of word choice on the quality of narrative feedback in ophthalmology resident trainee assessments following the introduction of competency-based medical education at Queen’s University.

Methods: Assessment data from July 2017-December 2020 were retrieved from Elentra^TM (Integrated Teaching and Learning Platform) and anonymized. Written feedback was assigned a Quality of Assessment for Learning (QuAL) score out of five based on this previously validated rubric. The correlation between QuAL score and specific coaching words was determined using a Spearman’s Rho analysis. Independent samples t-tests were used to compare the QuAL score when a specific word was used, and when it was absent.

Results: A total of 1997 individual assessments were used in this analysis. The number of times the identified coaching words were used within a comment was significantly and positively associated with the total QuAL score, with the exception of “next time” (rho=0.039, p=0.082), “read” (rho = 0.036, p = 0.112), “read more” (rho = -0.025, p = 0.256) and “review” (rho = -0.017, p = 0.440). The strongest correlations were for “continue” (rho = 0.182, p < 0.001), “try(ing)” (rho = 0.113, p < 0.001) and “next step” (rho = 0.103, p < 0.001). The mean value of the QuAL score increased when coaching words were used vs. not used with the largest mean difference of 1.44 (p < 0.001) for “reflect”. A clear positive relationship was demonstrated between word count and QuAL score (rho = .556, p < 0.001).

Conclusions: The use of certain coaching words in written comments may improve the quality of feedback.

Metrics

Metrics Loading ...

References

Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR, International CBME Collaborators. The role of assessment in competency-based medical education. Med Teach. 2010 Aug 1;32(8):676-82. https://doi.org/10.3109/0142159X.2010.500704

Marcotte L, Egan R, Soleas E, Dalgarno N, Norris M, Smith C. Assessing the quality of feedback to general internal medicine residents in a competency-based environment. Can Med Ed J. 2019 Nov;10(4):e32. https://doi.org/10.36834/cmej.57323

Branfield Day L, Rassos J, Billick M, Ginsburg S. ‘Next steps are…’: An exploration of coaching and feedback language in EPA assessment comments. Med Teach. 2022 Aug 8:1-8. https://doi.org/10.1080/0142159X.2022.2098098

Tekian A, Borhani M, Tilton S, Abasolo E, Park YS. What do quantitative ratings and qualitative comments tell us about general surgery residents’ progress toward independent practice? Evidence from a 5-year longitudinal cohort. Amer J Surg. 2019 Feb 1;217(2):288-95. https://doi.org/10.1016/j.amjsurg.2018.09.031

Ginsburg S, van der Vleuten CP, Eva KW. The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data. Acad Med. 2017 Nov 1;92(11):1617-21. https://doi.org/10.1097/ACM.0000000000001669

Stodel EJ, Wyand A, Crooks S, Moffett S, Chiu M, Hudson CC. Designing and implementing a competency-based training program for anesthesiology residents at the University of Ottawa. Anesthesiol res pract. 2015 Dec 21;2015. https://doi.org/10.1155/2015/713038.

Caccia N, Nakajima A, Scheele F, Kent N. Competency-based medical education: developing a framework for obstetrics and gynaecology. J obstet gyn Can. 2015 Dec 1;37(12):1104-12. https://doi.org/10.1016/S1701-2163(16)30076-7.

Stockley D, Egan R, Van Wylick R, et al. A systems approach for institutional CBME adoption at Queen’s University. Med Teach. 2020 Aug 2;42(8):916-21. https://doi.org/10.1080/0142159X.2020.1767768.

Jurd S, de Beer W, Aimer M, Fletcher S, Halley E, Schapper C, Orkin M. Introducing a competency based fellowship programme for psychiatry in Australia and New Zealand. Austral Psych. 2015 Dec;23(6):699-705. https://doi.org/10.1177/1039856215600898.

Dudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisors’ completed clinical evaluation reports. Med Ed. 2008 Aug;42(8):816-22. https://doi.org/10.1111/j.1365-2923.2008.03105.x.

Cheung WJ, Dudek N, Wood TJ, Frank JR. Daily encounter cards—evaluating the quality of documented assessments. JGME. 2016 Oct;8(4):601-4. https://doi.org/10.4300/JGME-D-15-00505.1.

Roshan A, Wagner N, Acai A, et al. Comparing the quality of narrative comments by rotation setting. J Surg Educ. 2021 Nov 1;78(6):2070-7. https://doi.org/10.1016/j.jsurg.2021.06.012

Roshan A, Farooq A, Acai A, et al. The effect of gender dyads on the quality of narrative assessments of general surgery trainees. Amer J Surg. 2022 Jul 1;224(1), 179-184. https://doi.org/10.1016/j.amjsurg.2021.12.001

Ross S, Hamza D, Zulla R, Stasiuk S, Nichols D. Development of and preliminary validity evidence for the EFeCT feedback scoring tool. JGME. 2022 Feb;14(1):71-9. https://doi.org/10.4300/JGME-D-21-00602.1

Chan TM, Sebok-Syer SS, Sampson C, Monteiro S. The Quality of Assessment of Learning (Qual) score: validity evidence for a scoring system aimed at rating short, workplace-based comments on trainee performance. Teach Learn Med. 2020 May 26;32(3):319-29. https://doi.org/10.1080/10401334.2019.1708365.

Braund H, Dalgarno N, McEwen L, Egan R, Reid MA, Baxter S. Involving ophthalmology departmental stakeholders in developing workplace-based assessment tools. Can J Opthamol. 2019 Oct 1;54(5):590-600. https://doi.org/10.1016/j.jcjo.2019.01.013

Tomiak A, Braund H, Egan R, et al. Exploring how the new entrustable professional activity assessment tools affect the quality of feedback given to medical oncology residents. J Cancer Educ. 2020 Feb;35(1):165-77. https://doi.org/10.1007/s13187-018-1456-z

Ross S, Huie M, Schipper S. Use words that count: a content analysis to identify words and phrases that commonly appear in effective formative feedback. International Conference on Residency Education 2015 Oct 22.

Roberts A, Jellicoe M, Fox K. How does a move towards a coaching approach impact the delivery of written feedback in undergraduate clinical education?. Advances Health Sci Educ. 2022 March; 1:1-5. https://doi.org/10.1007/s10459-021-10066-7

Steinmann AF, Dy NM, Kane GC, et al. The modern teaching physician—responsibilities and challenges: an APDIM white paper. Am J Med. 2009 Jul 1;122(7):692-7. https://doi.org/10.1016/j.amjmed.2009.03.020

Hauer KE, Nishimura H, Dubon D, Teherani A, Boscardin C. Competency assessment form to improve feedback. Clin Teach 2018 Dec;15(6):472-7. https://doi.org/10.1111/tct.12726

Bismil R, Dudek NL, Wood TJ. In‐training evaluations: developing an automated screening tool to measure report quality. Med Ed. 2014 Jul;48(7):724-32. https://doi.org/10.1111/medu.12490

Chan TM, Sebok-Syer SS, Yilmaz Y, Monteiro S, Syer SS. The impact of electronic data to capture qualitative comments in a competency-based assessment system. Cureus. 2022. 14(3). https://doi.org/10.7759/cureus.23480

Curtis R, Moon CC, Hanmore T, Hopman W, Baxter S. Evaluating the effect of assessment form design on the quality of feedback in one Canadian ophthalmology residency program as an early adopter of CBME. Can J Ophthalmol. 2023 Jan; 58(4): E149-150. https://doi.org/10.1016/j.jcjo.2023.01.003 .

Zelenski AB, Tischendorf JS, Kessler M, et al. Beyond “read more”: an intervention to improve faculty written feedback to learners. JGME. 2019 Aug 1;11(4), 468-471. https://doi.org/10.4300/JGME-D-19-00058.1