If we assess, will they learn? Students’ perspectives on the complexities of assessment-for-learning
DOI:
https://doi.org/10.36834/cmej.73875Abstract
Introduction: Assessment can positively influence learning, however designing effective assessment-for-learning interventions has proved challenging. We implemented a mandatory assessment-for-learning system comprising a workplace-based assessment of non-medical expert competencies and a progress test in undergraduate medical education and evaluated its impact.
Methods: We conducted semi-structured interviews with year-3 and 4 medical students at McGill University to explore how the assessment system had influenced their learning in year 3. We conducted theory-informed thematic analysis of the data.
Results: Eleven students participated, revealing that the assessment influenced learning through several mechanisms. Some required little student engagement (i.e., feed-up, test-enhanced learning, looking things up after an exam). Others required substantial engagement (e.g., studying for tests, selecting raters for quality feedback, using feedback). Student engagement was moderated by the perceived credibility of the system and of the costs and benefits of engagement. Credibility was shaped by students’ goals-in-context: becoming a good doctor, contributing to the healthcare team, succeeding in assessments.
Discussion: Our assessment system failed to engage students enough to leverage its full potential. We discuss the inherent flaws and external factors that hindered student engagement. Assessment designers should leverage easy-to-control mechanisms to support assessment-for-learning and anticipate significant collaborative work to modify learning cultures.
References
Kreiter C, Green J, Lenoch S, Saiki T. The overall impact of testing on medical student learning: quantitative estimation of consequential validity. Adv Health Sci Educ. 2013;18(4):835-844. https://doi.org/10.1007/s10459-012-9395-7 DOI: https://doi.org/10.1007/s10459-012-9395-7
Norcini J, Anderson MB, Bollela V, et al. 2018 Consensus framework for good assessment. Med Teach. 2018;40(11):1102-1109. https://doi.org/10.1080/0142159X.2018.1500016 DOI: https://doi.org/10.1080/0142159X.2018.1500016
Cilliers FJ, Schuwirth LW, Herman N, Adendorff HJ, van der Vleuten CP. A model of the pre-assessment learning effects of summative assessment in medical education. Adv Health Sci Educ. 2012;17(1):39-53. https://doi.org/10.1007/s10459-011-9292-5 DOI: https://doi.org/10.1007/s10459-011-9292-5
Cilliers F, Schuwirth L, Adendorff H, Herman N, Van der Vleuten C. The mechanism of impact of summative assessment on medical students’ learning. Adv Health Sci Educ. 2010;15(5):695-715. https://doi.org/10.1007/s10459-010-9232-9 DOI: https://doi.org/10.1007/s10459-010-9232-9
Larsen DP, Butler AC, Roediger HL. Test-enhanced learning in medical education. Med Educ. 2008;42(10):959-966. https://doi.org/10.1111/j.1365-2923.2008.03124.x DOI: https://doi.org/10.1111/j.1365-2923.2008.03124.x
Larsen DP, Butler AC, Roediger Iii HL. Comparative effects of test-enhanced learning and self-explanation on long-term retention. Med Educ. 2013;47(7):674-682. https://doi.org/10.1111/medu.12141 DOI: https://doi.org/10.1111/medu.12141
Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81-112. https://doi.org/10.3102/003465430298487 DOI: https://doi.org/10.3102/003465430298487
Schuwirth LWT, Van der Vleuten CPM. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33(6):478-485. https://doi.org/10.3109/0142159X.2011.565828 DOI: https://doi.org/10.3109/0142159X.2011.565828
Schuwirth L. Making the horse drink: use of mini-CEX in an assessment for learning view. Adv Health Sci Educ. 2013;18:1–4. https://doi.org/10.1007/s10459-012-9423-7 DOI: https://doi.org/10.1007/s10459-012-9423-7
Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682. https://doi.org/10.3109/0142159X.2010.500704 DOI: https://doi.org/10.3109/0142159X.2010.500704
Harrison CJ, Könings KD, Molyneux A, Schuwirth LWT, Wass V, Van der Vleuten CPM. Web-based feedback after summative assessment: how do students engage? Med Educ. 2013;47(7):734-744. https://doi.org/10.1111/medu.12209 DOI: https://doi.org/10.1111/medu.12209
Harrison C, Könings K, Schuwirth L, Wass V, van der Vleuten C. Barriers to the uptake and use of feedback in the context of summative assessment. Adv Health Sci Educ. 2015;20(1):229-245. https://doi.org/10.1007/s10459-014-9524-6 DOI: https://doi.org/10.1007/s10459-014-9524-6
Driessen EW, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten CP. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 2012;34(3):226-231. https://doi.org/10.3109/0142159x.2012.652242 DOI: https://doi.org/10.3109/0142159X.2012.652242
Bok H, Teunissen P, Favier R, et al. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ. 2013;13(1):123. https://doi.org/10.1186/1472-6920-13-123 DOI: https://doi.org/10.1186/1472-6920-13-123
Heeneman S, Oudkerk Pool A, Schuwirth LW, van der Vleuten CP, Driessen EW. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 2015;49(5):487-498. https://doi.org/10.1111/medu.12645 DOI: https://doi.org/10.1111/medu.12645
Bindal T, Wall D, Goodyear HM. Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise? Med Teach. 2011;33(11):919-927. https://doi.org/10.3109/0142159X.2011.558140 DOI: https://doi.org/10.3109/0142159X.2011.558140
Rees CE, Cleland JA, Dennis A, Kelly N, Mattick K, Monrouxe LV. Supervised learning events in the Foundation Programme: a UK-wide narrative interview study. BMJ Open. 2014;4(10). https://doi.org/10.1136/bmjopen-2014-005980 DOI: https://doi.org/10.1136/bmjopen-2014-005980
Barrett A, Galvin R, Scherpbier AJJA, Teunissen PW, O’Shaughnessy A, Horgan M. Is the learning value of workplace-based assessment being realised? A qualitative study of trainer and trainee perceptions and experiences. Postgrad Med J. 2017;93(1097):138-142. https://doi.org/10.1136/postgradmedj-2015-133917 DOI: https://doi.org/10.1136/postgradmedj-2015-133917
Wiliam D, Thompson M. Integrating assessment with learning: What will it take to make it work? In: The Future of Assessment: Shaping Teaching and Learning. 1st ed. Lawrence Erlbaum Associates; 2008.
Freeman A, Van Der Vleuten C, Nouns Z, Ricketts C. Progress testing internationally. Med Teach. 2010;32(6):451-455. https://doi.org/10.3109/0142159X.2010.485231 DOI: https://doi.org/10.3109/0142159X.2010.485231
van der Vleuten CPM, Verwijnen GM, Wijnen WHFW. Fifteen years of experience with progress testing in a problem-based learning curriculum. Med Teach. 1996;18(2):103-109. https://doi.org/10.3109/01421599609034142 DOI: https://doi.org/10.3109/01421599609034142
Yielder J, Wearn A, Chen Y, et al. A qualitative exploration of student perceptions of the impact of progress tests on learning and emotional wellbeing. BMC Med Educ. 2017;17(1):148. https://doi.org/10.1186/s12909-017-0984-2 DOI: https://doi.org/10.1186/s12909-017-0984-2
Cruess R, McIlroy JH, Cruess S, Ginsburg S, Steinert Y. The Professionalism Mini-Evaluation Exercise: A Preliminary Investigation. Acad Med. 2006;81(10):S74-S78. https://doi.org/10.1097/00001888-200610001-00019 DOI: https://doi.org/10.1097/00001888-200610001-00019
Sargeant J, Mann K, Van der Vleuten C, Metsemakers J. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ. 2009;14(3):399-410. https://doi.org/0.1007/s10459-008-9124-4 DOI: https://doi.org/10.1007/s10459-008-9124-4
Saldaña J. The coding manual for qualitative researchers. 3rd ed. Sage; 2016.
Kiger ME, Varpio L. Thematic analysis of qualitative data: AMEE Guide No. 131. Med Teach. Published online May 1, 2020:1-9. https://doi.org/10.1080/0142159X.2020.1755030 DOI: https://doi.org/10.1080/0142159X.2020.1755030
Dory V, Gomez-Garibello C, Cruess R, Cruess S, Cummings BA, Young M. The challenges of detecting progress in generic competencies in the clinical setting. Med Educ. 2018;52(12):1259-1270. https://doi.org/10.1111/medu.13749 DOI: https://doi.org/10.1111/medu.13749
Ricci M, St-Onge C, Xiao J, Young M. Students as stakeholders in assessment: how students perceive the value of an assessment. Perspect Med Educ. 2018;7(6):352-361. https://doi.org/10.1007/s40037-018-0480-3 DOI: https://doi.org/10.1007/S40037-018-0480-3
Boudreau JD, Cassell E, Fuks A. A healing curriculum. Med Educ. 2007;41(12):1193-1201. https://doi.org/10.1111/j.1365-2923.2007.02905.x DOI: https://doi.org/10.1111/j.1365-2923.2007.02905.x
Young M, Ryan A. Postpositivism in health professions education scholarship. Acad Med. 2020;95(5):695-699. https://doi.org/10.1097/ACM.0000000000003089 DOI: https://doi.org/10.1097/ACM.0000000000003089
Molloy E, Ajjawi R, Bearman M, Noble C, Rudland J, Ryan A. Challenging feedback myths: values, learner involvement and promoting effects beyond the immediate task. Med Educ. 2020;54(1):33-39. https://doi.org/10.1111/medu.13802 DOI: https://doi.org/10.1111/medu.13802
Ajjawi R, Regehr G. When I say ... feedback. Med Educ. 2019;53(7):652-654. https://doi.org/10.1111/medu.13746 DOI: https://doi.org/10.1111/medu.13746
Tavares W, Eppich W, Cheng A, et al. Learning conversations: an analysis of their theoretical roots and their manifestations of feedback and debriefing in medical education. Acad Med. 2020;95(7):1020-1025. https://doi.org/10.1097/acm.0000000000002932 DOI: https://doi.org/10.1097/ACM.0000000000002932
Watling CJ, Kenyon CF, Zibrowski EM, et al. Rules of Engagement: residents’ perceptions of the in-training evaluation process. Acad Med. 2008;83(10):S97-S100. https://doi.org/10.1097/ACM.0b013e318183e78c DOI: https://doi.org/10.1097/ACM.0b013e318183e78c
Scarff CE, Bearman M, Chiavaroli N, Trumble S. Trainees’ perspectives of assessment messages: a narrative systematic review. Med Educ. 2019;53(3):221-233. https://doi.org/10.1111/medu.13775 DOI: https://doi.org/10.1111/medu.13775
Eva K, Armson H, Holmboe E, et al. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ. 2012;17(1):15-26. https://doi.org/10.1007/s10459-011-9290-7 DOI: https://doi.org/10.1007/s10459-011-9290-7
Gaunt A, Patel A, Rusius V, Royle TJ, Markham DH, Pawlikowska T. ‘Playing the game’: how do surgical trainees seek feedback using workplace-based assessment? Med Educ. 2017;51(9):953-962. https://doi.org/10.1111/medu.13380 DOI: https://doi.org/10.1111/medu.13380
Bowen L, Marshall M, Murdoch-Eaton D. Medical student perceptions of feedback and feedback behaviors within the context of the “educational alliance.” Acad Med. 2017;92(9):1303-1312. https://doi.org/10.1097/acm.0000000000001632 DOI: https://doi.org/10.1097/ACM.0000000000001632
Telio S, Regehr G, Ajjawi R. Feedback and the educational alliance: examining credibility judgements and their consequences. Med Educ. 2016;50(9):933-942. https://doi.org/10.1111/medu.13063 DOI: https://doi.org/10.1111/medu.13063
Moroz A, Horlick M, Mandalaywala N, T Stern D. Faculty feedback that begins with resident self-assessment: motivation is the key to success. Med Educ. 2018;52(3):314-323. https://doi.org/10.1111/medu.13484 DOI: https://doi.org/10.1111/medu.13484
Delva D, Sargeant J, Miller S, et al. Encouraging residents to seek feedback. Med Teach. 2013;35(12):e1625-31. https://doi.org/10.3109/0142159X.2013.806791 DOI: https://doi.org/10.3109/0142159X.2013.806791
Bok HGJ, Teunissen PW, Spruijt A, et al. Clarifying students’ feedback-seeking behaviour in clinical clerkships. Med Educ. 2013;47(3):282-291. https://doi.org/10.1111/medu.12054 DOI: https://doi.org/10.1111/medu.12054
Bing-You R, Hayes V, Palka T, Ford M, Trowbridge R. The art (and artifice) of seeking feedback: clerkship students’ approaches to asking for feedback. Acad Med. 2018;93(8):1218-1226. https://doi.org/10.1097/acm.0000000000002256 DOI: https://doi.org/10.1097/ACM.0000000000002256
Watling C, LaDonna KA, Lingard L, Voyer S, Hatala R. ‘Sometimes the work just needs to be done’: socio-cultural influences on direct observation in medical training. Med Educ. 2016;50(10):1054-1064. https://doi.org/10.1111/medu.13062 DOI: https://doi.org/10.1111/medu.13062
Ramani S, Könings KD, Mann KV, Pisarski EE, van der Vleuten CPM. About politeness, face, and feedback: exploring resident and faculty perceptions of how institutional feedback culture influences feedback practices. Acad Med. 2018;93(9):1348-1358. https://doi.org/10.1097/acm.0000000000002193 DOI: https://doi.org/10.1097/ACM.0000000000002193
Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 2016;50(9):943-954. https://doi.org/10.1111/medu.13067 DOI: https://doi.org/10.1111/medu.13067
Molloy E, Boud D, Henderson M. Developing a learning-centred framework for feedback literacy. Assess Eval High Educ. 2020;45(4):527-540. https://doi.org/10.1080/02602938.2019.1667955 DOI: https://doi.org/10.1080/02602938.2019.1667955
Winstone NE, Nash RA, Parker M, Rowntree J. Supporting learners’ agentic engagement with feedback: a systematic review and a taxonomy of recipience processes. Educ Psychol. 2017;52(1):17-37. https://doi.org/10.1080/00461520.2016.1207538 DOI: https://doi.org/10.1080/00461520.2016.1207538
Royal College of Physicians and Surgeons of Canada. CanMEDS 2015: physician competency framework. (Frank JR, Snell L, Sherbino J, eds.).; 2015.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Valérie Dory, Maryam Wagner, Richard Cruess, Sylvia Cruess, Meredith Young
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.
