Assessment of laparoscopic skills: comparing the reliability of global rating and entrustability tools




Background: Competence by design (CBD) residency programs increasingly depend on tools that provide reliable assessments, require minimal rater training, and measure progression through the CBD milestones. To assess intraoperative skills, global rating scales and entrustability ratings are commonly used but may require extensive training.  The Competency Continuum (CC) is a CBD framework that may be used as an assessment tool to assess laparoscopic skills. The study aimed to compare the CC to two other assessment tools: the Global Operative Assessment of Laparoscopic Skills (GOALS) and the Zwisch scale.

Methods: Four expert surgeons rated thirty laparoscopic cholecystectomy videos. Two raters used the GOALS scale while the remaining two raters used both the Zwisch scale and CC. Each rater received scale-specific training. Descriptive statistics, inter-rater reliabilities (IRR), and Pearson’s correlations were calculated for each scale.

Results: Significant positive correlations between GOALS and Zwisch (r = 0.75, p < 0.001), CC and GOALS (r = 0.79, p < 0.001), and CC and Zwisch (r = 0.90, p < 0.001) were found. The CC had an inter-rater reliability of 0.74 whereas the GOALS and Zwisch scales had inter-rater reliabilities of 0.44 and 0.43, respectively. Compared to GOALS and Zwisch scales, the CC had the highest inter-rater reliability and required minimal rater training to achieve reliable scores.

Conclusion: The CC may be a reliable tool to assess intraoperative laparoscopic skills and provide trainees with formative feedback relevant to the CBD milestones. Further research should collect further validity evidence for the use of the CC as an independent assessment tool.


Metrics Loading ...


Royal College of Physicians and Surgeons of Canada. CBD start, launch and exam schedule. [Accessed Oct 3, 2020].

University of Toronto Department of Medicine. Competency based medical education. . [Accessed Oct 3, 2020].

de Montbrun S, Satterthwaite L, Grantcharov TP. Setting pass scores for assessment of technical performance by surgical trainees. Br J Surg. 2016;103(3):300-306.

Epstein, Ronald M., Cox, Malcolm, Irby DM. Assessment in medical education. NEJM. 2007;100(2):387-396.

Rudolph JW, Simon R, Raemer DB, Eppich WJ. Debriefing as formative assessment: Closing performance gaps in medical education. Acad Emerg Med. 2008;15(11):1010-1016.

Middleton RM, Baldwin MJ, Akhtar K, Alvand A, Rees JL. Which Global Rating Scale? J Bone Jt Surg. 2016;98(1):75-81.

Kramp KH, Van Det MJ, Hoff C, Lamme B, Veeger NJGM, Pierie JPEN. Validity and reliability of Global Operative Assessment of Laparoscopic Skills (GOALS) in novice trainees performing a laparoscopic cholecystectomy. J Surg Educ. 2015;72(2):351-358.

Gumbs AA, Hogle NJ, Fowler DL. Evaluation of resident laparoscopic performance using global operative assessment of laparoscopic skills. J Am Coll Surg. 2007;204(2):308-313.

Vassiliou MC, Feldman LS, Andrew CG, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005;190(1):107-113.

Williams RG, Sanfey H, Chen XP, Dunnington GL. A controlled study to determine measurement conditions necessary for a reliable and valid operative performance assessment: a controlled prospective observational study. Ann Surg. 2012;256(1):177-187.

YouTube. March 27 2015 CBD Webinar EPA Milestones. [Accessed Oct 3, 2020].

The Royal College of Physicians and Surgeons of Canada. EPAs and CanMEDS milestones. [Accessed Oct 3, 2020].

Darosa DA, Zwischenberger JB, Meyerson SL, et al. A theory-based model for teaching and assessing residents in the operating room. J Surg Educ. 2013;70(1):24-30.

George BC, Teitelbaum EN, Meyerson SL, et al. Reliability, validity, and feasibility of the zwisch scale for the assessment of intraoperative performance. J Surg Educ.. Vol 71. Elsevier Inc.; 2014:e90-e96.

Royal College of Physicians and Surgeons of Canada. CBD competence continuum diagram. 2015;(June):2015. [Accessed on May 17, 2022].

Al-Moteri M. Entrustable professional activities in nursing: a concept analysis. Int J Nurs Sci. 2020;7(3):277-284.

Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682.

Research Guides at University of Southern California. Types of research designs - organizing your social sciences research paper [Accessed Mar 11, 2022].

Gawad N, Fowler A, Mimeault R, Raiche I. The inter-rater reliability of technical skills assessment and retention of rater training. J Surg Educ. 2019;76(4):1088-1093.

Ilgen JS, Ma IWY, Hatala R, Cook DA. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ. 2015;49(2):161-173.

Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. Entrustability scales: outlining their usefulness for competency-based clinical assessment. Acad Med. 2016;91(2):186-190.

Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155-163.

Touchie C, Ten Cate O. The promise, perils, problems and progress of competency-based medical education. Med Educ. 2016;50(1):93-100.

Crossley J, Johnson G, Booth J, Wade W. Good questions, good answers: Construct alignment improves the performance of workplace-based assessment scales. Med Educ. 2011;45(6):560-569.

Weller JM, Castanelli DJ, Chen Y, Jolly B. Making robust assessments of specialist trainees’ workplace performance. Br J Anaesth. 2017;118(2):207-214.

Gawad N, Fowler A, Mimeault R, Raiche I. The inter-rater reliability of technical skills assessment and retention of rater training. J Surg Educ. 2019;76(4):1088-1093.

Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg Endosc Other Interv Tech. 2003;17(10):1525-1529.

Bilgic E, Watanabe Y, McKendy K, et al. Reliable assessment of operative performance. Am J Surg. 2016;211(2):426-430.

Tekian A, Watling CJ, Roberts TE, Steinert Y, Norcini J. Qualitative and quantitative feedback in the context of competency-based education. Med Teach. 2017;39(12):1245-1249.

Silber CG, Nasca TJ, Paskin DL, Eiger G, Robeson M, Veloski JJ. Do global rating forms enable program directors to assess the ACGME competencies? Acad Med. 2004;79(6):549-556.

Anderson PAM. Giving feedback on clinical skills: are we starving our young?




How to Cite

Alibhai K, Fowler A, Gawad N, Wood TJ, Raîche I. Assessment of laparoscopic skills: comparing the reliability of global rating and entrustability tools . Can. Med. Ed. J [Internet]. 2022 Aug. 2 [cited 2024 Jul. 21];13(6):36-45. Available from:



Original Research