Developing Academic Advisors and Competence Committees members: A community approach to developing CBME faculty leaders

Introduction Implementing competency-based medical education (CBME) at the institutional level poses many challenges including having to rapidly enable faculty to be facilitators and champions of a new curriculum which utilizes feedback, coaching, and models of programmatic assessment. This study presents the necessary competencies required for Academic Advisors (AA) and Competence Committee (CC) members, as identified in the literature and as perceived by faculty members at Queen’s University. Methods This study integrated a review of available literature (n=26) yielding competencies that were reviewed by the authors followed by an external review consisting of CBME experts (n=5). These approved competencies were used in a cross-sectional community consultation survey distributed one year before (n=83) and one year after transitioning to CBME (n=144). Findings Our newly identified competencies are a useful template for other institutions. Academic Advisor competencies focused on mentoring and coaching, whereas Competence Committee member’s competencies focused on integrating assessments and institutional policies. Competency discrepancies between stakeholder groups existing before the transition had disappeared in the post-implementation sample. Conclusions We found value in taking an active community-based approach to developing and validating faculty leader competencies sooner rather than later when transitioning to CBME. The evolution of Competence Committees members and Academic Advisors requires the investment of specialized professional development and the sustained engagement of a collaborative community with shared concerns.


Introduction
Implementing competency-based medical education (CBME) at the institutional level poses many challenges. [1][2][3] One of these is having to rapidly enable faculty to be facilitators and champions of a new curriculum which utilizes feedback, coaching, and new models of programmatic assessment. [4][5][6] The transition to CBME brings with it opportunities for new positions. 2,3 Two notable examples are the voluntary and emergent service roles of the Academic Advisor (AA) and Competence Committees member (CC). Academic Advisors, sometimes referred to as Coaches, 7,8 synthesize the assessment data that trainees accrue and provide a holistic summary of what the trainee should focus on next. CC members adjudicate portfolios of assessment data and make determinations on which trainees are competent at the current stage of training and should be promoted to greater responsibilities at the next level. 7 These roles are evolving in medical education and faculty leaders need to develop their own competencies in order to support trainee development in CBME. 3,9,10 With the transition to CBME occurring worldwide, it is crucial that the physicians choosing to enter these CBME support roles have the skills and competencies to thrive in their new roles and to support trainees more effectively.
While faculty in medicine come into their physician roles having received extensive medical training, many receive little training on how to become effective coaches and teachers. 3,5 Indeed, many physician assessors have been trained under different paradigms, where the focus was on summative and time-based assessment, rather than formative assessments and other forms of coaching. 2,4,5 To ensure success, we believed that Queen's faculty such as academic physicians and their community and distributed clinical preceptors entering the AA and CC members' positions must be responsive to the specific needs of individual residents. As well, they need to be accepting of the new competencies they themselves need when taking on these new positions.
In 2014, Queen's University set an ambitious goal to implement CBME across all 29 of its specialty training programs starting on July 1, 2017. In our view, the successful launch of CBME at Queen's, and the continued support of its ongoing implementation, would require the development of teaching and coaching competencies that are aligned with the roles e48 of AAs and CCs and with the larger Queen's community. As a first step, this study's purpose was to identify the competencies that would guide our institutions' faculty development to ease a successful roll-out of CBME. In keeping with the community approach to transitioning the Queen's Postgraduate Medical Education (PGME) to CBME, our goal was to create evidence-based faculty development initiatives such as workshops, small group teaching sessions, and modules) that would assist in developing these competencies. This community approach encompassed faculty, clinical preceptors and resident physicians in Royal College of Physician and Surgeons of Canada programs across our academic and distributed medical education networks including preceptors and adjunct faculty. As a second study outcome, the two-phase survey design would be able to show if a consensus on the perceived necessary competencies could be found among the stakeholders at Queen's. We conducted a literature review to ascertain existing competencies for AAs and CCs. PGME stakeholders completed a survey to refine and achieve relative consensus on what the competencies in the faculty development initiatives would need to be going forward. The approval of the stakeholders in the surveys would then be described and compared to show the extent to which the different stakeholders came to agree after the second iteration of the survey. This project began with a literature review which informed a community-based stakeholder consultation approach comprised of two surveys delivered two years apart, which included a review of competencies by international experts involved in CBME (See Figure 1). 11,12 Survey respondents were a varied group consisting of program directors, CBME program leads, frontline physicians, and residents involved in the transition to CBME at Queen's University.

Research
Phase One: As a first step, we conducted a literature review of competencies for medical education using an all-database search of EBSCOhost in April 2016 resulting in 340 hits using the search terms 'competencies' AND 'academic advisor' OR 'competence committee'. 13 In addition, author consultations yielded another 26 articles for consideration (See Figure 2). As an end result, the competencies were gleaned from 26 articles that met the eligibility criteria of providing peer-reviewed, English language in-text competencies for medical educators. [3][4][5] Phase Two: We collated competencies into itemized lists for AA and CC members and then circulated them by email to five invited members of the ICBME 37 group (an international consortium of CBME leaders external to the author team) who proposed slight language changes and updating of terms which were reconciled by the author team before recirculating. The external reviewers stated their approval of the list as comprehensive within three iterations.
Phase Three: Once we finalized the list of competencies for AAs and CCs, we sought to ascertain the level of approval of the Queen's PGME community stakeholders of the 29 Royal College Specialty programs. To this end, we administered a survey of these competencies by email listservs using Qualtrics software to stakeholder groups including residents, attending physicians, CBME leads, and program directors totalling 316 stakeholders throughout the Royal College Specialty PGME community at Queen's one year before the transition to CBME (June 2016; n= 83; response rate = 26.2%), and one year after the transition to CBME (July 2018; n=144; response rate= 45.6%). The two survey samples were therefore gathered two years apart and were independently gathered from one another with anonymous responses. For both survey occasions, stakeholders ranked the competencies they thought were most important using a 5-point Likert-type scale (1= strongly disagree, 5= strongly agree). Respondents were asked how important a given competency was from their perspective for the given role. This survey was designed to delineate how well received our developed competencies for AA and CC members matched with expectations of stakeholders "on the ground", and what changes would need to be made to further align these competencies with the postgraduate community of AAs and CCs. Participants also proposed additional competencies for AAs and CCs from their own perspectives. These stakeholderidentified additional competencies which were then triangulated with those included in the survey to form the learning goals of modules aimed to support the transition for physicians entering the new positions of AA and CC members. The responses were datacleaned and then analysed as independent samples before and after implementation of CBME in SPSS v.23 yielding descriptive and inferential statistics including analysis of variance (ANOVA). This allowed us to identify the degree of acceptance or rejection of the competencies among the school as a whole and amongst each of the various groups.
This survey was designed to delineate how well received our developed competencies for AA and CC members matched with expectations of stakeholders "on the ground", and what changes would need to be made to further align these competencies with the postgraduate community of AAs and CCs. Participants also proposed additional competencies for AAs and CCs from their own perspectives. These stakeholderidentified additional competencies which were then triangulated with those included in the survey to form the learning goals of modules aimed to support the transition for physicians entering the new positions of AA and CC members. The responses were datacleaned and then analysed as independent samples before and after implementation of CBME in SPSS v.23 yielding descriptive and inferential statistics including analysis of variance (ANOVA). This allowed us to identify the degree of acceptance or rejection of the competencies among the school as a whole and amongst each of the various groups.

Results
The review and consolidation of literature findings resulted in twenty-one competencies for academic advisors and ten for competence committee members. The review of competencies by five members of the ICBME group, who are recognized as influencers within the CBME movement made primarily language revisions and gave their approval to the competencies displayed in Tables 2 and 4.
The newly defined competencies for AAs and CCs were well received with an overwhelmingly positive reception from all groups of stakeholders although there were varied levels of approval (See Table 1 for stakeholder demographics). Assessment, as well as mentoring competencies were the most positively rated The highest rated competencies for AAs centred upon mentoring, such as "recognize learners in distress and provide appropriate resources within the educational structure to assist" (at 4.77 out of 5), followed by "facilitates learner to take ownership of developing and updating learning plans" (at 4.73 out of 5). While still highly rated skills, assessment competencies were rated as less important for AAs than mentoring competencies. The lowest rated competency was "assists colleagues to develop lifelong learning skills in their learners, which was rated at 3.59 out of 5 (with a variance of 1.11) indicating polarized views on the importance of this competency.
In the case of AAs (See Table 2) the level of agreement with competencies, although tentative prior to the implementation of CBME, was significantly increased at the 99% confidence level in the ANOVA (F= 26.187, p= <0.001, d= 1.22; large effect size) when surveyed after implementation. There were significant differences in approval of AA competencies between residents and groups composed of attending physicians in the pre-implementation sample (F= 4.886, p = <0.01, d= 0.83; large effect size).
Residents reported much less approval. However, at the time of the post-implementation sample, there were no significant differences and approval was consistent indicating the formation of a relatively strong consensus.
e51 In addition to rating the competencies from the literature, raters also proposed their own competencies (See Table 3). Seventy-three additional competencies were proposed which were categorized into eight groupings Similar to the competencies identified in the literature, the most common competencies for AA involved mentoring and coaching skills.
e52 Approachable Non-intimidating -Possess the qualities that would allow a resident to express their concerns or insecurities freely without fear of reprimand Active listener CBME Expertise and CanMeds competencies 9 Knowledgeable -About both the program requirements and the processes of competency assessments. For example, many of our staff are under the impression that residency will be strictly competency (vs time-based), which is a common misperception that the Royal College has repeatedly denied.
Knowledge of the CBME stages / EPAs and how the residents progress, in order to offer appropriate assessment of resident in their current stage. Effective Feedback and Assessment

11
Ability to give specific feedback Analyzes and integrates diverse assessment data to generate comprehensive feedback Ability to synthesize various forms of assessment Clinical Teaching and Learning

11
Understanding of CBME stages and evaluations Specialty knowledge -ie. it should be an emergency doctor for emergency resident Royal college certified physician in the same speciality of the trainee Objectivity 3 Objective -Use objective, rather than subjective, measures to assess progress Impartial Reliability and Organizational Skills 5 Reliable in timeliness of feedback and meeting Time management For CC members, the highest rated competencies (See Table 4) were centred around enforcing policy and triangulating and utilizing assessment data including: "understand their role, policies, and the process regarding resident assessment and progress" (rated at 4.72 out of 5). Another very highly rated competency was "collates and interprets evidence of learning and provides meaningful insight based on multiple sources, including direct observation" (at 4.50 out of 5). Similar to the AA group, "assists colleagues to develop lifelong learning skills in their learners" was the lowest rated competency (at 3.72 out of 5). In the case of Competence Committees members (See Table 4), the level of agreement with competencies although tentative prior to CBME implementation, when surveyed after implementation levels of agreement consistently significantly increased at 99% confidence (F= 9.336, p= 0.003, d= 0.91; large effect size).
There were significant differences in approval of CC competencies between residents and groups composed of attending physicians in the preimplementation sample (F= 3.944, p = 0.01, d= 0.60; medium effect size), however, at the time of the postimplementation sample, there were no significant differences which supports the idea that a consensus emerged among all the stakeholders e53 Respondents rated the competencies from the literature and offered their own competencies for Competence Committee members. 36 additional competencies were proposed which were categorized into six groupings (See Table 5). The most common of these additional proposed competencies were focused on developing a deep knowledge of CBME.

Discussion
This study identified 21 AA competencies and 10 CC competencies from the literature and refined them to the Queen's context enabling the process of tailored faculty development that molded consensus while building faculty capacity. Although AAs are expected to help monitor trainee development to help adjudicate and deliver feedback to the residents in their care, the AA' competencies were more highly rated when they focused on mentoring and coaching skills, rather than assessment skills. In comparison, CC members' competencies were rated more highly when focused on assessment, integrating multiple sources of formative and summative assessments, and abiding by the new policies governing CBME such as when to promote a resident (in addition to the actions of past resident promotion committees which operated with less frequent assessments typically of a summative nature). As a result of the competencies most highly rated by the Queen's raters, modules were developed by CBME content experts with the competencies as learning goals. The modules are a central part of the induction process for faculty new to CBME, constituting a key part of the portfolio of new faculty resources at our institution.
The community-based approach of refining, proposing, and rating competencies was important as it promoted engagement from all stakeholders including faculty, residents, and CBME specialists across PGME. It also provided a comprehensive view on what the faculty in these AA & CC positions needed to be able to do, which was informed by the literature, molded by expert consensus, and uniquely aligned to current practices of medical education delivery at Queen's University. Following analysis of survey results, the final modules were assigned to experts who began development on online modules available to all PGME faculty. These modules will take the form of slideshows that will be narrated and presented using an online learning management software to be accessible publicly as a part of the portfolio of efforts to facilitate the development of skilled AAs, CC members, and preceptors in general.
Raters had varying expectations for what successful AA and CC members should be able to do before implementation. After implementation, the gap had closed for both the AA and CC competencies. The events of implementation, chiefly the mandated faculty development for program leaders and consistent outreach efforts with our larger medicine community convincingly paid dividends in terms of increased approval of the competencies and molding consensus on the expectations for AAs and CCs at our institution. The lack of differences after implementation points towards a greater degree of shared understanding among stakeholders and lends support to the notion that taking an accelerated path together as an institution, as was done at Queen's, can result in a culture shift towards shared priorities. 10

Limitations
This study has limitations in generalizability and methodology. This was a single centre study, which means that it is as much a reflection as a product of Queen's University. Although, other institutions would likely face similar challenges in infrastructure, capacity, stakeholder wariness, and structural change when faced with the same paradigm shift, other contextual factors are necessary to consider when generalizing these findings to other contexts. Methodologically, due to the ethical concerns of potentially identifying stakeholders, the survey asked respondents to answer limited demographic questions, which prevented comparisons by some demographic factors. The anonymous nature of responses also made it impossible to note which respondents answered both surveys. This study focused on institutional perspectives rather than intergroup comparisons, making the gathered demographic information sufficient.

Conclusion
We found value in taking an active community-based approach to identifying the competencies that would guide our institutions faculty development. The creation of CC members and AA roles requires a sustained investment of specialized professional development. To this end, our evidence-informed approach was an effective way to develop shared competencies for teaching faculty that enriched our community of practice and developed a better understanding of each program's needs. e55

Practice points
• AA's competencies were most aligned with mentoring and teaching rather than assessing • The lack of consensus very much present one year before implementation in terms of priorities has largely subsided one year after implementation as groups work together in the new reality of CBME as opposed to being driven by their initial conceptions • CC members' competencies were most aligned with assessment knowledge and less with a teaching role • Communities transitioning to CBME can greatly benefit from engaging their stakeholders in the design of professional development to best deliver content that suits the community's needs.
Conflicts of interest: The authors have no conflicts of interest to report