Construct-Aligned Rating Scales Improve the Reliability of Program Evaluation Data

  • Shayna Rusticus University of British Columbia
  • Kevin Eva University of British Columbia
  • Linda Peterson University of British Columbia
Keywords: construct alignment, construct-aligned ratings scales, medical education, program evaluation, Readiness for Clerkship survey, reliability

Abstract

In workplace-based assessment, research has suggested that aligning rating scales with how clinical supervisors naturally conceptualize trainee performance improves reliability and makes assessment more efficient. This study examined the generalizability of those findings by determining if construct alignment improves the reliability of competency-focused program evaluation ratings provided by 135 Year 3 medical students and 185 faculty supervisors. Participants were randomly assigned to complete the Readiness for Clerkship survey using one of two construct-aligned rating scales or a competence-focused rating scale. Mean differences, Spearman correlations, and Generalizability analyses compared the scales’ utility. Relative to the competence-oriented scale, the construct-aligned scales showed greater consistency in the rank order of items between students and faculty. Eight student raters were needed to achieve G=0.8 using the independence scale, 12 were needed using the behavioral/independence scale, and 22 were needed using the competence-oriented scale. Faculty were less consistent in their ratings and required more raters (k=25-32), regardless of the scale used.  These results extend previous research into the benefits of construct-aligned scales by suggesting that students offer better ratings when the scales used to collect their judgments are aligned with the constructs of independence and sophistication rather than being phrased in terms of competence.

Author Biographies

Shayna Rusticus, University of British Columbia
Statistical Analyst, Evaluation Studies Unit, Faculty of Medicine
Kevin Eva, University of British Columbia
Senior Scientist, Centre for Health Education Scholarship, Faculty of Medicine.
Linda Peterson, University of British Columbia
Senior Evaluator, Evaluation Studies Unit, Faculty of Medicine
Published
2018-06-27