Using Item Analysis to Assess Objectively the Quality of the Calgary-Cambridge OSCE Checklist

Abstract

Background:  The purpose of this study was to investigate the use of item analysis to assess objectively the quality of items on the Calgary-Cambridge Communications OSCE checklist.

Methods:  A total of 150 first year medical students were provided with extensive teaching on the use of the Calgary-Cambridge Guidelines for interviewing patients and participated in a final year end 20 minute communication OSCE station.  Grouped into either the upper half (50%) or lower half (50%) communication skills performance groups, discrimination, difficulty and point biserial values were calculated for each checklist item.

Results:  The mean score on the 33 item communication checklist was 24.09 (SD = 4.46) and the internal reliability coefficient was ? = 0.77. Although most of the items were found to have moderate (k = 12, 36%) or excellent (k = 10, 30%) discrimination values, there were 6 (18%) identified as ‘fair’ and 3 (9%) as ‘poor’. A post-examination review focused on item analysis findings resulted in an increase in checklist reliability (? = 0.80).

Conclusions:  Item analysis has been used with MCQ exams extensively. In this study, it was also found to be an objective and practical approach to use in evaluating the quality of a standardized OSCE checklist.