Construct-Aligned Rating Scales Improve the Reliability of Program Evaluation Data
In workplace-based assessment, research has suggested that aligning rating scales with how clinical supervisors naturally conceptualize trainee performance improves reliability and makes assessment more efficient. This study examined the generalizability of those findings by determining if construct alignment improves the reliability of competency-focused program evaluation ratings provided by 135 Year 3 medical students and 185 faculty supervisors. Participants were randomly assigned to complete the Readiness for Clerkship survey using one of two construct-aligned rating scales or a competence-focused rating scale. Mean differences, Spearman correlations, and Generalizability analyses compared the scales’ utility. Relative to the competence-oriented scale, the construct-aligned scales showed greater consistency in the rank order of items between students and faculty. Eight student raters were needed to achieve G=0.8 using the independence scale, 12 were needed using the behavioral/independence scale, and 22 were needed using the competence-oriented scale. Faculty were less consistent in their ratings and required more raters (k=25-32), regardless of the scale used. These results extend previous research into the benefits of construct-aligned scales by suggesting that students offer better ratings when the scales used to collect their judgments are aligned with the constructs of independence and sophistication rather than being phrased in terms of competence.
The Canadian Evaluation Society (CES) owns the copyright to all material published in the Journal. Authors are informed of this policy prior to submission of the final copy for publication. Requests for permission to reprint, post or distribute copies of articles (electronic or hardcopy) come to the Editor-in-chief, who has the authority to decide on behalf of CES.