Medical Council of Canada Qualifying Examinations and performance in future practice

Elizabeth Wenghofer; John Boulet

doi:10.36834/cmej.73770

Authors

Elizabeth Wenghofer Laurentian University https://orcid.org/0000-0003-3144-0343
John Boulet Uniformed Services University of the Health Sciences

DOI:

https://doi.org/10.36834/cmej.73770

Abstract

The purpose of medical licensing examinations is to protect the public from practitioners who do not have adequate knowledge, skills, and abilities to provide acceptable patient care, and therefore evaluating the validity of these examinations is a matter of accountability. Our objective was to discuss the Medical Council of Canada's Qualifying Examinations (MCCQEs) Part I (QE1) and Part II (QE2) in terms of how well they reflect future performance in practice.

We examined the supposition that satisfactory performance on the MCCQEs are important determinants of practice performance and, ultimately, patient outcomes. We examined the literature before the implementation of the QE2 (pre-1992), post QE2 but prior to the implementation of the new Blueprint (1992-2018), and post Blueprint (2018-present).

The literature suggests that MCCQE performance is predictive of future physician behaviours, that the relationship between examination performance and outcomes did not attenuate with practice experience, and that associations between examination performance and outcomes made sense clinically.

While the evidence suggests the MCC qualifying examinations measure the intended constructs and are predictive of future performance, the validity argument is never complete. As new competency requirements emerge, we will need to develop valid and reliable mechanisms for determining practice readiness in these areas.

References

Wenghofer EF. Research in medical regulation: an active demonstration of accountability. J Med Reg. 2015;101(3):13-7. https://doi.org/10.30770/2572-1852-101.3.13

Swanson DB, Roberts TE. Trends in national licensing examinations in medicine. Med ed. 2016;50(1):101-14. https://doi.org/10.1111/medu.12810

Bobos P, Pouliopoulou DV, Harriss A, Sadi J, Rushton A, MacDermid JC. A systematic review and meta-analysis of measurement properties of objective structured clinical examinations used in physical therapy licensure and a structured review of licensure practices in countries with well-developed regulation systems. PloS one. 2021;16(8):e0255696. https://doi.org/10.1371/journal.pone.0255696

Archer J, Lynn N, Coombes L, Roberts M, Gale T, Regan de Bere S. The medical licensing examination debate. Regul Gov. 2017;11(3):315-22. https://doi.org/10.1111/rego.12118

Archer J, Lynn N, Roberts M, Coombes L, Gale T, de Regand Bere S. A systematic review on the impact of licensing examinations for doctors in countries comparable to the UK. Plymouth: Plymouth University. 2015.

Schuwirth L. National licensing examinations, not without dilemmas. Med ed. 2016;50(1):15-7. https://doi.org/10.1111/medu.12891

Tsichlis JT, Del Re AM, Carmody JB. The past, present, and future of the united states medical licensing examination step 2 clinical skills examination. Cureus. 2021;13(8). https://doi.org/10.7759/cureus.17157

Wormald BW, Schoeman S, Somasunderam A, Penn M. Assessment drives learning: an unavoidable truth? Anat sci educ. 2009;2(5):199-204. https://doi.org/10.1002/ase.102

Murray DJ, Boulet JR. Anesthesiology Board Certification Changes: A Real-time Example of "Assessment Drives Learning." Anesthesiology. 2018;128(4):704-6. https://doi.org/10.1097/ALN.0000000000002086

Norman G, Neville A, Blake JM, Mueller B. Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Med teach. 2010;32(6):496-9. https://doi.org/10.3109/0142159X.2010.486063

Boulet J, van Zanten M. Ensuring high-quality patient care: the role of accreditation, licensure, specialty certification and revalidation in medicine. Med ed 2014: 48: 75-86 https://doi.org/10.1111/medu.12286

Schuwirth L. The need for national licensing examinations. Med ed. 2007;41(11):1022-3. https://doi.org/10.1111/j.1365-2923.2007.02856.x

Boulet JR, Durning SJ. What we measure… and what we should measure in Med ed. 2019;53(1):86-94. https://doi.org/10.1111/medu.13652

Archer J, Lynn N, Coombes L, et al. The impact of large scale licensing examinations in highly developed countries: a systematic review. BMC med ed. 2016;16(1):212-212. https://doi.org/10.1186/s12909-016-0729-7

Roberts WL, Gross GA, Gimpel JR, et al. An investigation of the relationship between COMLEX-USA licensure examination performance and state licensing board disciplinary actions. Acad Med. 2020;95(6):925-30. https://doi.org/10.1097/ACM.0000000000003046

Cuddy MM, Young A, Gelman A, et al. Exploring the relationships between USMLE performance and disciplinary action in practice: a validity study of score inferences from a licensure examination. Acad Med. 2017;92(12):1780-5. https://doi.org/10.1097/ACM.0000000000001747

Norcini JJ, Boulet JR, Opalek A, Dauphinee WD. The relationship between licensing examination performance and the outcomes of care by international medical school graduates. Acad Med. 2014;89(8):1157-62. https://doi.org/10.1097/ACM.0000000000000310

Hudson KM, Feinberg G, Hempstead L, Zipp C, Gimpel JR, Wang Y. Association between performance on COMLEX-USA and the American College of Osteopathic Family Physicians in-service examination. J Grad Med Ed. 2018;10(5):543-7. https://doi.org/10.4300/JGME-D-17-00997.1

Arnhart KL, Cuddy MM, Johnson D, Barone MA, Young A. Multiple United States medical licensing examination attempts and the estimated risk of disciplinary actions among graduates of US and Canadian medical schools. Acad Med. 2021;96(9):1319-23. https://doi.org/10.1097/ACM.0000000000004210

Patel MD, Heitkamp DE, Jordan SG. Re: The relationship between US medical licensing examination step scores and ABR core examination outcome and performance: a multi-institutional study. J Am Coll Radiol. 2021;18(1):8-9. https://doi.org/10.1016/j.jacr.2020.10.018

Medical Council of Canada. The MCC suspends the delivery of the MCCQE Part II [Internet]. Available from: https://mcc.ca/news/mcc-suspends-delivery-of-mccqe-part-ii/ [Accessed on Nov 7, 2021].

Medical Council of Canada. Medical Council of Canada Qualifying Examination (MCCQE) Part 1 Annual Technical Report 2019 [Internet]. 2019 Apr. Available from: https://mcc.ca/media/MCCQE-Part-I-Annual-Technical-Report-2019.pdf [Accessed on Apr 1, 2022].

Medical Council of Canada. Medical Council of Canada Qualifying Examination (MCCQE) Part II Annual Technical Report 2018 [Internet]. 2018 Oct. Available from: https://www.mcc.ca/media/MCCQE-Part-II-Technical-Report-Oct-2018.pdf [Accessed on Apr 1, 2022].

Medical Council of Canada. Blueprint Project: Examinations Blueprint and Content Specifications - Final Version [Internet]. 2014 Sep. Available from: https://mcc.ca/media/Blueprint-Report.pdf [Accessed on Nov 7, 2021].

Pugh DM, Wood TJ, Boulet JR. Assessing procedural competence: validity considerations. Simul Healthc 2015;10(5):288-94. https://doi.org/10.1097/SIH.0000000000000101

Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane's framework. Med ed 2015: 49: 560-575 https://doi.org/10.1111/medu.12678

Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.

Kane MT. Validating interpretive arguments for licensure and certification examinations. Eval Health Prof. 1994;17(2):133-59. https://doi.org/10.1177/016327879401700202

Kane M. Validating high-stakes testing programs. Educ Meas: Issues Pract. 2002;21(1):31-41. https://doi.org/10.1111/j.1745-3992.2002.tb00083.x

Clauser B, Margolis M, Swanson D. Issues of validity and reliability for assessments in medical education. In: Holmboe ES, Durning SJ, Hawkins RE, editors. Practical guide to the evaluation of clinical competence. Second edition. Philadelphia, PA: Elsevier; 2018. p. 22-36.

Kane M. The argument-based approach to validation. School Psych Rev. 2013;42(4):448-57. https://doi.org/10.1080/02796015.2013.12087465

Tamblyn R. Is the public being protected? prevention of suboptimalmedical practice through training programs and credentialing examinations. Eval Health Prof. 1994;17(2):198-221. https://doi.org/10.1177/016327879401700205

Tamblyn R, Abrahamowicz M, Bartlett G, et al. Executive summary. Medical Council of Canada: The Quebec-Ontario Follow-up Study of the Association between Scores Achieved on the MCCQE Part II Examination and Performance in Clinical Practice [Internet]. 2011. Available from: https://mcc.ca/media/Tamblyn_Score-Association_MCCQE-Part-II_Clinical-Practice-Performance_2009.pdf [Accessed on Jun 5, 2021].

Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007 Sep 5;298(9):993-1001. https://doi.org/10.1001/JAMA.298.9.993

Wenghofer E, Klass D, Abrahamowicz M, et al. Doctor scores on national qualifying examinations predict quality of care in future practice. Med ed. 2009;43(12):1166-73. https://doi.org/10.1111/j.1365-2923.2009.03534.x

Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Influence of physicians' management and communication ability on patients' persistence with antihypertensive medication. Arch Intern Med. 2010;170(12):1064-72. https://doi.org/10.1001/archinternmed.2010.167

Kawasumi Y, Ernst P, Abrahamowicz M, Tamblyn R. Association between physician competence at licensure and the quality of asthma management among patients with out-of-control asthma. Arch Intern Med. 2011;171(14):1292-4. https://doi.org/10.1001/archinternmed.2011.319

Cadieux G, Abrahamowicz M, Dauphinee D, Tamblyn R. Are physicians with better clinical skills on licensing examinations less likely to prescribe antibiotics for viral respiratory infections in ambulatory care settings? Med care. 2011;156-65. https://doi.org/10.1097/MLR.0b013e3182028c1a

Meguerditchian A-N, Dauphinee D, Girard N, et al. Do physician communication skills influence screening mammography utilization? BMC. 2012;12(1):1-8. https://doi.org/10.1186/1472-6963-12-219

Sherbino J, Dore KL, Wood TJ, et al. The relationship between response time and diagnostic accuracy. Acad Med. 2012;87(6):785-91. https://doi.org/10.1097/ACM.0b013e318253acbd

De Champlain AF, Streefkerk C, Roy M, Tian F, Qin S, Brailovsky C. Predicting family medicine specialty certification status using standardized measures: for a sample of international medical graduates engaged in a practice-ready assessment pathway to provisional licensure. J Med Reg. 2014;100(4):8-16. https://doi.org/10.30770/2572-1852-100.4.8

Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288(23):3019-26. https://doi.org/10.1001/JAMA.288.23.3019

Cadieux G, Tamblyn R, Dauphinee D, Libman M. Predictors of inappropriate antibiotic prescribing among primary care physicians. CMAJ. 2007;177(8):877-83. https://doi.org/10.1503/CMAJ.070151

De Champlain AF, Ashworth N, Kain N, Qin S, Wiebe D, Tian F. Does pass/fail on medical licensing exams predict future physician performance in practice? a longitudinal cohort study of Alberta physicians J Med Reg. 2020;106(4):17-26. https://doi.org/10.30770/2572-1852-106.4.17

Kane MT, Crooks TJ, Cohen AS. Designing and evaluating standard-setting procedures for licensure and certification tests. Adv Health Sci Educ. 1999;4(3):195-207. https://doi.org/10.1023/A:1009849528247

Price T, Lynn N, Coombes L, Roberts M, Gale T, de Bere SR, et al. The international landscape of medical licensing examinations: a typology derived from a systematic review. International J health policy and management. 2018;7(9):782. https://doi.org/10.15171/ijhpm.2018.32

Breithaupt K. Medical Licensure Testing White Paper for the Assessment Review Task Force of the Medical Council of Canada. 2011.

McConnell M, Gu A, Arshad A, Mokhtari A, Azzam K. An innovative approach to identifying learning needs for intrinsic CanMEDS roles in continuing professional development. Med ed online. 2018;23(1):1497374. https://doi.org/10.1080/10872981.2018.1497374