Prescribing competency assessment for Canadian medical students: a pilot evaluation.

Background
The knowledge and ability to prescribe safely and effectively is a core competency for every graduating medical student. Our previous research suggested concerns about medical student prescribing abilities, and interest in a standardized assessment process.


Methods
A multi-year cross-sectional study evaluating the feasibility, acceptability, and discriminative ability of an online prescribing competency assessment for final year Canadian medical students was conducted. Students at nine sites of four Ontario medical schools were invited to participate in an online one-hour exam of eight domains related to prescribing safely. Student feedback on perceived fairness, clarity, and ease of use formed the primary outcome. Exam performance and parity between schools were the secondary outcome.


Results
A total of 714 students completed the assessment during spring final review courses between 2016 and 2018. Student feedback was more favourable than not for appropriateness of content (53.5% agreement vs 18.3% disagreement), clarity of questions (65.5% agreement vs 11.6% disagreement), question layout and presentation (70.8% agreement vs 12.2% disagreement), and ease of use of online interface (67.1% agreement vs 13.6% disagreement). Few (23.6% believed their course work had prepared them for the assessment. Mean total exam score was 70.0% overall (SD 10.4%), with 47.6% scoring at or above the pass threshold of 70%.Conclusion: Our prescribing competency assessment proved feasible, acceptable, and discriminative, and indicated a need for better medical school training to improve prescribing competency. Further evaluation in a larger sample of medical schools is warranted.


Introduction
Every physician, regardless of specialty, must learn clinical pharmacology, therapeutics, and competent prescribing skills. In Canada, doctors write more than 500 million prescriptions annually for more than 1100 prescription drug families. 1 However, prescribed medications are a common source of adverse events. Medication errors are estimated to be the fifth or sixth leading cause of death. 2,3 Prescribing safely involves a combination of complex tasks: 1) gathering an appropriate history from the patient and their medical records; 2) performing a relevant physical exam; 3) applying knowledge to understand whether any medication is indicated, which medication would be most cost-effective for the individual patient's benefit-harm profile; and 4) creating and communicating a coherent, legible prescription with a practical plan to monitor the patient's progress. 4 Both the literature and observations of senior physicians suggest that newly licenced physicians are having difficulties keeping up with knowledge and skills of clinical pharmacology. 5,6 This is understandable since this specialty is arguably the most rapidly advancing area of medicine. [7][8][9] In addition, poor performance on tests of clinical decision-making or communication has been shown to predict patient complaints and disciplinary action over subsequent years of practice. 10 All of this is of sufficient concern that medical educators in Canada and the United States are creating a required Entrustable Professional Activity based on the ability to write a high quality prescription and counsel accordingly, and Canadian prescribing competencies have been developed in collaboration with the Royal College of Physicians and Surgeons of Canada. 11 These concerns are not isolated to North American medical graduates. The EQUIP study in England reviewed 124,260 medication orders, finding an error rate as high as 10.3% among junior housestaff. 9 In response to this study and a series of serious medication safety incidents, education leaders in clinical pharmacology and the British Pharmacological Society instituted the Prescribing Safety Assessment (PSA), an online examination that final year medical students must pass prior to their licensing exams. 12 Drawing on a large database of validated questions, the full PSA consists of 60 questions and lasts for two hours. 13 Students participate in the PSA at the end of medical school a few months before their medical licensure examinations, and are encouraged to prepare using practice resources. 14 Approximately 53,000 students have completed the exam in the last five years, with very positive feedback on fairness and the helpfulness of the exam to their competence to practice. 9,15,16 These students expressed appreciation for the initiative taken to address the prescription competency concerns, for increased confidence when prescribing as a first year resident, and for making them aware of key prescribing resources such as formularies. 13,15,16 Clinical Pharmacology and Toxicology (CPT) is a much younger, smaller, and less standardized specialty in Canada compared to its UK counterpart of Clinical Pharmacology and Therapeutics. Within the CanMEDS framework of medical education, CPT knowledge and prescribing skills competencies require skills within each and every CanMEDS domain (Scholar, Collaborator, Communicator, Advocate, Leader, Professional) including the more general Medical Expert umbrella. 17 Indeed, small studies at McMaster University have suggested that medical students perceive CPT training to be important but inadequate, and find an online assessment acceptable. 18,19 A subsequent survey of medical education leaders in all 17 of the Canadian medical schools showed that faculty had substantial concerns about the prescribing abilities of a sizable minority of their medical student graduates and junior residents, and expressed strong interest in the incorporation of a standardized prescribing assessment into the medical curricula and licensing process. 20 e105 Our objectives were to test the technical feasibility, acceptability, and discriminative ability of a Canadian version of the Prescribing Safety Assessment (C-PSA) and use it to gauge the prescribing competence of final year Canadian medical students.

Methods
We designed a cross-sectional study. In collaboration with the British Pharmacological Society, we developed a one-hour, 30 question PSA and adapted it to common Canadian scenarios and medical terminology. The C-PSA covered eight essential domains of prescribing competence (see Table 1). 21 Each question had been previously analyzed using a modified Angoff technique for level of difficulty, a method similar to the Bookmark method of validation employed by the Medical Council of Canada for their qualifying examinations. 22,23

Prescription Review
Deciding which components of the current prescription list are inappropriate, unsafe, or ineffective based on clinical circumstances.

Planning Management
Deciding which combination of therapies would be the most appropriate to manage a particular clinical situation.

Providing Information
Deciding which are the important pieces of information that should be provided to patients to allow them to choose whether to take the medication, or to enhance its safety and effectiveness.

Calculation Skills
Making an accurate drug dosage calculation based on numerical information.

Adverse Drug Reactions
Identifying likely adverse reactions of specific drugs, drugs that are likely to be causing specific adverse drug reactions, potentially dangerous drug interactions and deciding on the best approach to managing a clinical presentation that results from the adverse effects of a drug.

Drug Monitoring
Deciding on how to monitor the beneficial and harmful effects of medicines.

Data Interpretation
Deciding on the meaning of the results of investigations as they relate to decisions about on-going drug therapy.
All nine sites at four medical schools agreed to pilot the assessment as part of their Medical Council of Canada Qualifying Examination Part I (MCCQE) review courses for final year medical students in the springs of 2016, 2017, and 2018. All final year medical students at the four schools were contacted by email and invited to participate, with assurances that scores on the C-PSA were not part of their final transcripts. Individual registration was required to assign an anonymous study ID number and to send individuals their own confidential scores. We provided information on the scope of the assessment, question domains, and a mock exam to students on the PSA website (https://prescribingsafetyassessment.ac.uk/resource s) prior to the assessment.
Each participating university's research ethics board reviewed and approved the project.
At the time of the exam for each school, each student signed on to their pre-registered account simultaneously in a classroom setting using wireless connections. Students were provided and encouraged to utilize a direct link to a standardized medical and drug information database, RxTx, as part of the exam interface during the exam (i.e., an "open book" exam). 24,25 At the end of the assessment, each participant completed feedback questions on the test's appropriateness, difficulty level, fairness, time frame, layout, ease of use, and clarity.
Feedback obtained from the students on the assessment, particularly acceptability based on perceived fairness, clarity, and ease of use (details below), was set as the primary outcome of this study. Student total scores, scores on each domain, and whether student performance was comparable between the different medical schools were defined as the secondary outcomes. Based on prior use of the questions with British medical students but allowing for the abbreviated preparation period and shorter exam, we set the exam pass threshold at 70%. Only 134 (23.6%) students agreed that their medical school coursework had prepared them for the assessment. Based on 568 (79.6%) responses, 202 students (35.6%) reported writing ten or fewer prescriptions during their entire training. Students were generally not in favour of making a C-PSA a mandatory requirement to pass prior to licensing exams. Table A1 (Appendix A) also displays exam scores overall, by domain and by school. Mean total score was 70.0% (SD 10.5%; range 5% to 96%.) with 47.6% of students meeting the suggested overall pass score of 70%. There were significant differences between some of the score domains and between schools (p < 0.05).

Discussion
This assessment of prescribing competency among final year medical students is the first of its kind in Canada, and the first time that the PSA was hosted wirelessly anywhere. The lack of technical difficulty and the rapid turnaround of marks support the feasibility of this formative assessment of prescribing competency internationally. However, there was a significant difference in overall performance scores between question domains. The lack of preparation time at all schools, no doubt, affected ratings and scores, but this study was meant to be a partial validation and anchoring exercise. The clarity of the assessment, the time provided, and the interface were all acceptable. However, only a few students felt that their medical school training had adequately prepared them for the content in the assessment, and the perceived training gap was reflected in the test scores. The overall pass rate of 47.6% suggests that current training for prescribing competency may be inadequate. Estimates of mandatory teaching time for clinical pharmacology in Ontario medical schools range from 15 to 55 hours in total. Students from the school with the highest number of mandatory teaching hours were the most likely to feel their coursework prepared them for the assessment, and scored the highest overall.
We acknowledge several limitations to our study. First, only four Canadian medical schools were included, with low recruitment success at one of the schools. Second, the pass threshold of 70% was determined indirectly using the PSA experience with British medical students, although we had good reasons to expect that prescribing competency thresholds would be similar. Third, we provided our participants no specific preparation for their C-PSA assessment, and we were unable to provide a good, general prescribing resource online during the exam since there is no Canadian national formulary. Lastly, the exam platform (ASP.Net MVC and PostgreSQL) anonymized the feedback questionnaire responses, therefore we could not measure the correlation between the number of prescriptions and assessment performance. We plan to remedy this particular limitation for future assessments.

Conclusion
In conclusion, we have demonstrated that a prescribing competency assessment format used regularly in the UK can be adapted and successfully applied in Canada. Our results suggest that graduating medical students are not practice-ready and have not achieved prescribing competency. Further, we recommend testing the assessment across Canada to first determine the scalability of e107 implementation and second to determine medical student prescribing competence more broadly.