The critical role of direct observation in entrustment decisions

Matthew Sibbald; Muqtasid Mansoor; Michael Tsang; Sarah Blissett; Geoffrey Norman

doi:10.36834/cmej.72040

The critical role of direct observation in entrustment decisions

Authors

Matthew Sibbald McMaster University
Muqtasid Mansoor McMaster University
Michael Tsang McMaster University
Sarah Blissett Western University
Geoffrey Norman McMaster University

DOI:

https://doi.org/10.36834/cmej.72040

Abstract

Background: Entrustment decisions may be retrospective (based on past experiences with a trainee) or real-time (based on direct observation). We investigated judgments of entrustment based on assessor prior knowledge of candidates and based on systematic direct observation, conducted in an objective structured clinical exam (OSCE).

Methods: Sixteen faculty examiners provided 287 retrospective and real-time entrustment ratings of 16 cardiology trainees during OSCE stations in 2019 and 2020. Reliability and validity of these ratings were assessed by comparing correlations across stations as a measure of reliability, differences across postgraduate years as an index of construct validity, correlation to standardized in-training exam (ITE) as a measure of criterion validity, and reclassification of entrustment as a measure of consequential validity.

Results: Both retrospective and real-time assessments were highly reliable (all intra-class correlations >0.86). Both increased with year of postgraduate training. Real-time entrustment ratings were significantly correlated with standardized ITE scores; retrospective ratings were not. Real-time ratings explained 37% (2019) and 46% (2020) of variance in examination scores vs. 21% (2019) and 7% (2020) for retrospective ratings. Direct observation resulted in a different level of entrustment compared with retrospective ratings in 44% of cases (p = <0.001).

Conclusions: Ratings based on direct observation made unique contributions to entrustment decisions.

Author Biography

Matthew Sibbald, McMaster University

Matt Sibbald, MD MHPE PhD FRCPC CHSE

Director, Center for Simulation-Based Learning Cardiology Residency Program Director Assistant Professor, Department of Medicine McMaster University

Interventional Cardiologist

Hamilton Health Sciences

Niagara Health Systems

Downloads

Published

2021-11-02 — Updated on 2021-11-02

Versions

2021-11-02 (2)
2021-11-02 (1)

How to Cite

Sibbald M, Mansoor M, Tsang M, Blissett S, Norman G. The critical role of direct observation in entrustment decisions . Can. Med. Ed. J [Internet]. 2021 Nov. 2 [cited 2024 Dec. 24];12(5):18-23. Available from: https://journalhosting.ucalgary.ca/index.php/cmej/article/view/72040

Download Citation

Issue

Vol. 12 No. 5 (2021)

Section

Original Research

License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.

Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.