Medical Council of Canada Qualifying Examinations and performance in future practice
DOI:
https://doi.org/10.36834/cmej.73770Abstract
The purpose of medical licensing examinations is to protect the public from practitioners who do not have adequate knowledge, skills, and abilities to provide acceptable patient care, and therefore evaluating the validity of these examinations is a matter of accountability. Our objective was to discuss the Medical Council of Canada's Qualifying Examinations (MCCQEs) Part I (QE1) and Part II (QE2) in terms of how well they reflect future performance in practice.
We examined the supposition that satisfactory performance on the MCCQEs are important determinants of practice performance and, ultimately, patient outcomes. We examined the literature before the implementation of the QE2 (pre-1992), post QE2 but prior to the implementation of the new Blueprint (1992-2018), and post Blueprint (2018-present).
The literature suggests that MCCQE performance is predictive of future physician behaviours, that the relationship between examination performance and outcomes did not attenuate with practice experience, and that associations between examination performance and outcomes made sense clinically.
While the evidence suggests the MCC qualifying examinations measure the intended constructs and are predictive of future performance, the validity argument is never complete. As new competency requirements emerge, we will need to develop valid and reliable mechanisms for determining practice readiness in these areas.
Metrics
References
Wenghofer EF. Research in medical regulation: an active demonstration of accountability. J Med Reg. 2015;101(3):13-7. https://doi.org/10.30770/2572-1852-101.3.13
Swanson DB, Roberts TE. Trends in national licensing examinations in medicine. Med ed. 2016;50(1):101-14. https://doi.org/10.1111/medu.12810
Bobos P, Pouliopoulou DV, Harriss A, Sadi J, Rushton A, MacDermid JC. A systematic review and meta-analysis of measurement properties of objective structured clinical examinations used in physical therapy licensure and a structured review of licensure practices in countries with well-developed regulation systems. PloS one. 2021;16(8):e0255696. https://doi.org/10.1371/journal.pone.0255696
Archer J, Lynn N, Coombes L, Roberts M, Gale T, Regan de Bere S. The medical licensing examination debate. Regul Gov. 2017;11(3):315-22. https://doi.org/10.1111/rego.12118
Archer J, Lynn N, Roberts M, Coombes L, Gale T, de Regand Bere S. A systematic review on the impact of licensing examinations for doctors in countries comparable to the UK. Plymouth: Plymouth University. 2015.
Schuwirth L. National licensing examinations, not without dilemmas. Med ed. 2016;50(1):15-7. https://doi.org/10.1111/medu.12891
Tsichlis JT, Del Re AM, Carmody JB. The past, present, and future of the united states medical licensing examination step 2 clinical skills examination. Cureus. 2021;13(8). https://doi.org/10.7759/cureus.17157
Wormald BW, Schoeman S, Somasunderam A, Penn M. Assessment drives learning: an unavoidable truth? Anat sci educ. 2009;2(5):199-204. https://doi.org/10.1002/ase.102
Murray DJ, Boulet JR. Anesthesiology Board Certification Changes: A Real-time Example of "Assessment Drives Learning." Anesthesiology. 2018;128(4):704-6. https://doi.org/10.1097/ALN.0000000000002086
Norman G, Neville A, Blake JM, Mueller B. Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Med teach. 2010;32(6):496-9. https://doi.org/10.3109/0142159X.2010.486063
Boulet J, van Zanten M. Ensuring high-quality patient care: the role of accreditation, licensure, specialty certification and revalidation in medicine. Med ed 2014: 48: 75-86 https://doi.org/10.1111/medu.12286
Schuwirth L. The need for national licensing examinations. Med ed. 2007;41(11):1022-3. https://doi.org/10.1111/j.1365-2923.2007.02856.x
Boulet JR, Durning SJ. What we measure… and what we should measure in Med ed. 2019;53(1):86-94. https://doi.org/10.1111/medu.13652
Archer J, Lynn N, Coombes L, et al. The impact of large scale licensing examinations in highly developed countries: a systematic review. BMC med ed. 2016;16(1):212-212. https://doi.org/10.1186/s12909-016-0729-7
Roberts WL, Gross GA, Gimpel JR, et al. An investigation of the relationship between COMLEX-USA licensure examination performance and state licensing board disciplinary actions. Acad Med. 2020;95(6):925-30. https://doi.org/10.1097/ACM.0000000000003046
Cuddy MM, Young A, Gelman A, et al. Exploring the relationships between USMLE performance and disciplinary action in practice: a validity study of score inferences from a licensure examination. Acad Med. 2017;92(12):1780-5. https://doi.org/10.1097/ACM.0000000000001747
Norcini JJ, Boulet JR, Opalek A, Dauphinee WD. The relationship between licensing examination performance and the outcomes of care by international medical school graduates. Acad Med. 2014;89(8):1157-62. https://doi.org/10.1097/ACM.0000000000000310
Hudson KM, Feinberg G, Hempstead L, Zipp C, Gimpel JR, Wang Y. Association between performance on COMLEX-USA and the American College of Osteopathic Family Physicians in-service examination. J Grad Med Ed. 2018;10(5):543-7. https://doi.org/10.4300/JGME-D-17-00997.1
Arnhart KL, Cuddy MM, Johnson D, Barone MA, Young A. Multiple United States medical licensing examination attempts and the estimated risk of disciplinary actions among graduates of US and Canadian medical schools. Acad Med. 2021;96(9):1319-23. https://doi.org/10.1097/ACM.0000000000004210
Patel MD, Heitkamp DE, Jordan SG. Re: The relationship between US medical licensing examination step scores and ABR core examination outcome and performance: a multi-institutional study. J Am Coll Radiol. 2021;18(1):8-9. https://doi.org/10.1016/j.jacr.2020.10.018
Medical Council of Canada. The MCC suspends the delivery of the MCCQE Part II [Internet]. Available from: https://mcc.ca/news/mcc-suspends-delivery-of-mccqe-part-ii/ [Accessed on Nov 7, 2021].
Medical Council of Canada. Medical Council of Canada Qualifying Examination (MCCQE) Part 1 Annual Technical Report 2019 [Internet]. 2019 Apr. Available from: https://mcc.ca/media/MCCQE-Part-I-Annual-Technical-Report-2019.pdf [Accessed on Apr 1, 2022].
Medical Council of Canada. Medical Council of Canada Qualifying Examination (MCCQE) Part II Annual Technical Report 2018 [Internet]. 2018 Oct. Available from: https://www.mcc.ca/media/MCCQE-Part-II-Technical-Report-Oct-2018.pdf [Accessed on Apr 1, 2022].
Medical Council of Canada. Blueprint Project: Examinations Blueprint and Content Specifications - Final Version [Internet]. 2014 Sep. Available from: https://mcc.ca/media/Blueprint-Report.pdf [Accessed on Nov 7, 2021].
Pugh DM, Wood TJ, Boulet JR. Assessing procedural competence: validity considerations. Simul Healthc 2015;10(5):288-94. https://doi.org/10.1097/SIH.0000000000000101
Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane's framework. Med ed 2015: 49: 560-575 https://doi.org/10.1111/medu.12678
Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.
Kane MT. Validating interpretive arguments for licensure and certification examinations. Eval Health Prof. 1994;17(2):133-59. https://doi.org/10.1177/016327879401700202
Kane M. Validating high-stakes testing programs. Educ Meas: Issues Pract. 2002;21(1):31-41. https://doi.org/10.1111/j.1745-3992.2002.tb00083.x
Clauser B, Margolis M, Swanson D. Issues of validity and reliability for assessments in medical education. In: Holmboe ES, Durning SJ, Hawkins RE, editors. Practical guide to the evaluation of clinical competence. Second edition. Philadelphia, PA: Elsevier; 2018. p. 22-36.
Kane M. The argument-based approach to validation. School Psych Rev. 2013;42(4):448-57. https://doi.org/10.1080/02796015.2013.12087465
Tamblyn R. Is the public being protected? prevention of suboptimalmedical practice through training programs and credentialing examinations. Eval Health Prof. 1994;17(2):198-221. https://doi.org/10.1177/016327879401700205
Tamblyn R, Abrahamowicz M, Bartlett G, et al. Executive summary. Medical Council of Canada: The Quebec-Ontario Follow-up Study of the Association between Scores Achieved on the MCCQE Part II Examination and Performance in Clinical Practice [Internet]. 2011. Available from: https://mcc.ca/media/Tamblyn_Score-Association_MCCQE-Part-II_Clinical-Practice-Performance_2009.pdf [Accessed on Jun 5, 2021].
Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007 Sep 5;298(9):993-1001. https://doi.org/10.1001/JAMA.298.9.993
Wenghofer E, Klass D, Abrahamowicz M, et al. Doctor scores on national qualifying examinations predict quality of care in future practice. Med ed. 2009;43(12):1166-73. https://doi.org/10.1111/j.1365-2923.2009.03534.x
Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Influence of physicians' management and communication ability on patients' persistence with antihypertensive medication. Arch Intern Med. 2010;170(12):1064-72. https://doi.org/10.1001/archinternmed.2010.167
Kawasumi Y, Ernst P, Abrahamowicz M, Tamblyn R. Association between physician competence at licensure and the quality of asthma management among patients with out-of-control asthma. Arch Intern Med. 2011;171(14):1292-4. https://doi.org/10.1001/archinternmed.2011.319
Cadieux G, Abrahamowicz M, Dauphinee D, Tamblyn R. Are physicians with better clinical skills on licensing examinations less likely to prescribe antibiotics for viral respiratory infections in ambulatory care settings? Med care. 2011;156-65. https://doi.org/10.1097/MLR.0b013e3182028c1a
Meguerditchian A-N, Dauphinee D, Girard N, et al. Do physician communication skills influence screening mammography utilization? BMC. 2012;12(1):1-8. https://doi.org/10.1186/1472-6963-12-219
Sherbino J, Dore KL, Wood TJ, et al. The relationship between response time and diagnostic accuracy. Acad Med. 2012;87(6):785-91. https://doi.org/10.1097/ACM.0b013e318253acbd
De Champlain AF, Streefkerk C, Roy M, Tian F, Qin S, Brailovsky C. Predicting family medicine specialty certification status using standardized measures: for a sample of international medical graduates engaged in a practice-ready assessment pathway to provisional licensure. J Med Reg. 2014;100(4):8-16. https://doi.org/10.30770/2572-1852-100.4.8
Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288(23):3019-26. https://doi.org/10.1001/JAMA.288.23.3019
Cadieux G, Tamblyn R, Dauphinee D, Libman M. Predictors of inappropriate antibiotic prescribing among primary care physicians. CMAJ. 2007;177(8):877-83. https://doi.org/10.1503/CMAJ.070151
De Champlain AF, Ashworth N, Kain N, Qin S, Wiebe D, Tian F. Does pass/fail on medical licensing exams predict future physician performance in practice? a longitudinal cohort study of Alberta physicians J Med Reg. 2020;106(4):17-26. https://doi.org/10.30770/2572-1852-106.4.17
Kane MT, Crooks TJ, Cohen AS. Designing and evaluating standard-setting procedures for licensure and certification tests. Adv Health Sci Educ. 1999;4(3):195-207. https://doi.org/10.1023/A:1009849528247
Price T, Lynn N, Coombes L, Roberts M, Gale T, de Bere SR, et al. The international landscape of medical licensing examinations: a typology derived from a systematic review. International J health policy and management. 2018;7(9):782. https://doi.org/10.15171/ijhpm.2018.32
Breithaupt K. Medical Licensure Testing White Paper for the Assessment Review Task Force of the Medical Council of Canada. 2011.
McConnell M, Gu A, Arshad A, Mokhtari A, Azzam K. An innovative approach to identifying learning needs for intrinsic CanMEDS roles in continuing professional development. Med ed online. 2018;23(1):1497374. https://doi.org/10.1080/10872981.2018.1497374
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Elizabeth Wenghofer, John Boulet
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.