The development of national entrustable professional activities to inform the training and assessment of public health and preventative medicine residents

Background Entrustable Professional Activities (EPAs) have emerged to bridge the gap between the learning of individual competencies and competence in real world practice. EPAs capture the critical core work of a discipline integrating competencies from multiple domains. This report describes the development of a set of EPAs for specialty training in Public Health and Preventive Medicine (PHPM) in Canada. Methods The PHPM EPAs were developed using multiple existing sources. A combination of workshops and a national online survey was used to consult with PHPM program directors, the national specialty committee, and competency-based education experts. Results A national survey of PHPM program directors had a 71% response rate with 80% or more of respondents agreeing with all of the 20 EPA titles and all but one of their descriptions. Competency developmental stage-specific milestones were identified for each EPA. Conclusion The identification of the EPAs and their milestones will increase emphasis on the demonstrated performance of the specialty’s core work. Simulations applicable to several EPAs have been developed. The EPAs have also been incorporated into a PHPM National Review Course and will be used to develop a national PHPM curriculum, as well as a national written practice examination.

Milestone: "the expected ability of a health care professional at a stage of expertise." 5 Entrustable Professional Activities are units of work that together "constitute the mass of critical elements that operationally define a profession." 6 Typically numbering between 20-30 EPAs per discipline, 7 EPAs identify what a professional does in practice and how to know a graduate is competent to independently and safely practice these activities. 3 Milestones complement EPAs by identifying the expected ability of a health professional for achieving an EPA by a particular stage of development. 5 Fulfilling a milestone requires the application of multiple competencies.
A growing number of medical disciplines in Canada and elsewhere have begun developing and applying EPAs in their training programs including family medicine, geriatrics, and internal medicine. [8][9][10][11] Entrustable Professional Activities have not previously been applied to the training of public health physicians, whose focus is on protecting and promoting the health of populations.
In Canada, public health and preventive medicine (PHPM) specialists train in a 5-year, post-MD Royal College of Physicians and Surgeons (RCPSC) residency program comprised of: 1-2 years of clinical training; at least two (4-month) semesters of academic public health training; and, at least 18 months of supervised placements in public healthrelated practice settings. 12 Graduates work at local (e.g., Medical Officer of Health), provincialterritorial, federal and international system levels, as well as academic, clinical, and private sector settings.
With a recently released set of detailed PHPM objectives of training, 1 EPAs offer to translate these into a coherent set of practice-based outcomes and provide a consistent focus on residents' development throughout the various stages of PHPM training. This is particularly relevant since PHPM residents have had one of the lowest final examination success rates of the RCPSC specialties. 13 With Queen's University School of Medicine setting the goal for its postgraduate specialty programs to adopt CBME for all incoming residents, an opportunity to pursue EPA development for PHPM was created. As a small specialty, collaboration was sought with other interested PHPM programs, initially in Ontario and then nationally. Representatives from two other PHPM programs joined the Ontario programs to lead this project with the remaining programs providing feedback on the developed EPAs.
While the development of PHPM EPAs is programdriven, this work aligns with the Royal College's Competence by Design (CBD) initiative, which is transforming resident education to be more outcome-focussed with, among other features, the identification of training stage-specific EPAs and milestones. 14 With PHPM's participation in this initiative scheduled to commence in 2019, 15 the Royal College indicated to the National Specialty Committee that the experience defining and applying EPAs would be useful for informing the specialty's future participation in the CBD initiative. While EPAs are conceptually attractive, their application poses a number of challenges to programs, residents and faculty including e73 mechanisms and tools to assess the achievement of EPAs. 7 The purpose of this report is to describe the development of a set of end-of-training EPAs for PHPM. This specialty's approach to, and experience in, developing EPAs will be informative to other disciplines considering a similar undertaking.

Methods
The EPA development process is illustrated in Figure  1. An online survey was conducted of all PHPM program directors across Canada. Their level of agreement with each of the EPA titles and descriptions was assessed using a five-point scale with the anchors "strongly agree," "agree," "neither agree nor disagree," "disagree," and "strongly disagree." Additional comments or suggestions for improvement were also sought for each item.
The survey results were discussed by the group of program directors and external experts that had attended the workshop and minor revisions were made to the EPA titles and descriptions. This was deemed sufficient in order for the EPAs to be piloted while awaiting the Royal College's CBD process.
To address the milestones that would accompany the EPAs, the CanMEDS 2015 competency developmental stages were aligned with the four stages of PHPM training (Table 1). While the original 31 Calgary PHPM EPAs had accompanying milestones, these needed to be resorted to align with the smaller number of 20 final EPAs. For newly identified EPAs, workshop participants began development of milestone statements, which were completed by the project consultant. An additional gap in milestones was created when many of Calgary's existing "transition to discipline" milestones were shifted to later developmental stages where they would be more realistic to achieve for most programs. Multiple sources were utilized to identify public health-related competencies that could be reasonably acquired within clinical training settings during the "transition to discipline" developmental stage. [16][17][18]

Results
Half of the final 20 EPA titles were easily identified from the comparison of the two pre-existing EPA sets from the University of Calgary and the Ontariobased PHPM programs. To resolve the remaining practice areas, a key decision was made to focus on EPAs reflecting the core tasks achieved by the endof-training. "Overarching"-type EPAs were utilized to encompass multiple more granular, "nested EPAs," 2 which would all need to be demonstrated in order to successfully complete the overarching EPA (Table 2). While termed "end-of-training," it was explicitly recognized that a resident could demonstrate earlier fulfilment of an EPA since the demonstration of competence is not always dependent upon a specific amount of training time.
19 The survey of program directors achieved a 71% response rate with a total of 11 respondents from 10 of the 14 PHPM. One of the respondents did not complete the survey, but had attended the two-day workshop and indicated agreement with the EPA titles and descriptions. Combining results of "strongly agree" and "agree" categories from the 5point scale, 80% or more of survey respondents agreed with all of the EPA titles and all but one of their descriptions ( Table 3).
The survey results were used to make minor improvements to the EPA titles and descriptions with particular attention to the distinctions and inter-dependencies between EPAs (see Appendix A for an example). e75 The milestones were reviewed to assess their alignment with the finalized EPA titles and descriptions with improvements made where necessary; Table 4 provides an example of the final EPA for managing a communicable disease outbreak (see Table 4 in Appendix B).

Discussion
The purpose of this report is to describe the development of a set of end-of-training EPAs to capture the core, critical tasks of PHPM practice. This specialty's approach to, and experience in, developing EPAs will be informative to other disciplines considering a similar undertaking. Consistent with guidelines for EPA development, 7 a combination of expert meetings and surveys were utilized to consolidate and refine previously developed sets of EPA titles. The final EPA titles and descriptions received high levels of agreement among PHPM program directors across the country, otherwise a Delphi approach would have been required.
The EPAs are anticipated to be informative for residents, residency programs and faculty members. For residents, the EPAs provide an explicit description of the core tasks of the specialty by identifying, through milestones, the expected ability for achieving an EPA at each stage of development (see Table 4 in Appendix B for example). This will support residents to be more proactively engaged in their training to ensure that they are getting the necessary experiences and are being observed and assessed. 7 Greater stage-specific clarity will benefit PHPM programs since they are reliant on clinical training programs and graduate academic programs to provide their residents with public health-related competencies associated with the care of individual patients, as well as graduate-level public healthrelated knowledge and skills, respectively. These are Conduct an environmental health risk assessment 100% 91%

Domain: Public Health Leadership & Management
Act as spokesperson to communicate about public health issues to the public, their elected representatives, inter-sectoral partners, and the health system.

100% 80%
Lead and manage strategic planning 82% 73% Lead and manage the operational planning and evaluation of a public health program 100% 90% Lead and manage a team and organization. 90% 100% Lead and manage change within an organization 100% 89% Lead and manage a quality improvement initiative 80% 100%

Domain: Protect the Public's Health
Manage communicable diseases of public health importance 90% 100% Manage a communicable disease outbreak 89% 100% Manage environmental health risks, exposures and incidents of public health importance 90% 100% Manage a cluster of cancer or other adverse health outcome 90% 100% Prepare for and manage public health incidents and emergencies 90% 80%

Domain: Promote Health and Prevent Diseases and Injuries
Conduct a health policy analysis 90% 100% Lead and manage strategies and programming to promote health and health equity 90% 100% Advocate for the adoption and implementation of healthy public policies 89% 100% Design and manage a population-based screening program 100% 100% *Note: the EPAs shown are the post-survey version, which have minor wording changes in selected items e76 foundational to the PHPM-specific training rotations that occur in the final years of the residency program. In these later training stages, the EPAs will support and encourage a greater emphasis on observing residents demonstrating their competence for specific, relevant tasks. This is a further step from historical medical training perspectives that assumed a defined time period within a placement would result in sufficient acquisition of needed competencies. 19 Instead, while time is viewed as a resource for acquiring competencies, the focus of training is outcome-oriented with the demonstrated achievement of the milestones and EPAs. 20 Implementation of EPAs requires committed, experienced and highly-trained faculty. 2 While providing clearer guidance on the continuum of learning and milestones for assessing development, EPAs place greater demands for making formal assessment and promotion decisions, establishing portfolio systems to capture and track assessments, and to train and mentor faculty. 7 For example, a family medicine program created electronic EPA field notes to structure day-to-day assessment and feedback, and to serve as a summative tool to ground competency declarations about residents. 8 In the absence of mechanisms and tools to assess the achievement of EPAs, they risk being another competency-related conceptual framework collecting dust. Some EPAs such as response to a public health emergency may not be universally observable for all residents. Nevertheless, residents need to be prepared for these scenarios, which is why they are included in the EPAs. The implication is that programs require a variety of tools (simulations among them) which can be used to provide residents with exposure and practice opportunities.
The Queen's PHPM program has developed simulations that are directly applicable to several EPAs including public health emergencies, outbreak management, and cancer cluster investigation, which will be made available to residents in all programs, as well as a refresher to existing PHPM physicians. Additional case-based learning tools and simulations will be needed to support and assess residents' development. Considering the small size of each PHPM program, a national approach to development should be pursued if possible.
The EPAs have already been incorporated into the first annual PHPM National Review Course. 21 There has also been agreement among program directors to use the EPAs in developing a national PHPM curriculum and a national written practice examination. Several PHPM programs have started or are looking to pilot the EPAs on a voluntary basis and their experience will be informative when the specialty formally enters the RCPSC's CBD process to integrate EPAs and milestones across the curriculum. 22 Overall, it is anticipated that the development of EPAs will strengthen the recruitment, training and assessment of PHPM residents, which will collectively lead to stronger PHPM practice. Greater standardization of competence for agreed-upon core tasks will support greater employer confidence for what can be expected of a PHPM graduate. In addition to better tracking of residents' training progress, monitoring the success rate of PHPM residents in their final RCPSC examination are among potential impact measures.

Conclusion
Entrustable Professional Activities are a vital bridge between detailed lists of learning objectives and the training and assessment of the core, critical tasks of a specialist. A set of 20 end-of-training EPAs have been identified for PHPM, which are anticipated to strengthen the recruitment, training and assessment of future PHPM specialists. Implementation of the EPAs will have wide-spread implications for residents, residency programs and teaching faculty. Appendix A

EPA 5 Conduct an Environmental Health Risk Assessment Relation to Other EPAs
This EPA includes the conduct of an environmental health risk assessment to characterize the nature and magnitude of health risks to humans from chemical contaminants and other stressors that may be present in the environment. This includes: • identifying a hazard and assessing exposure • characterizing and managing the risk • communicating the risk.
Example: assess and report on the health impacts of wind turbines EPA 4 -Conduct a health impact assessment This EPA focuses on assessing health impacts of a broader range of policies and scenarios such as a municipal redevelopment plan, the introduction of a casino in a city/neighbourhood, etc. EPA 14 -Manage environmental health risks, exposures, and incidents of public health importance This EPA focuses on managing a scenario involving a potential exposure to a known health risk with particular emphasis on air quality, water safety, and food safety (e.g., a chemical contamination incident of an aquifer supplying community drinking water).
e80 Appendix B