Patient involvement in resident assessment within the Competence by Design context: a mixed-methods study.

Background
Patients can contribute to resident assessment in Competence by Design (CBD). This study explored the extent, nature, as well as the facilitators and hindrances of patient involvement in resident assessment within and across Canadian specialty/sub-specialty/special programs that are transitioning or have transitioned to CBD.


Methods
We used a two-phase sequential explanatory mixed-methods design. In Phase 1, we surveyed program directors (PDs). In Phase 2, we interviewed PDs from Phase 1.


Results
In Phase 1, 63 (62.4%) respondents in the CBD preparation stage, do not know if patients will be involved in resident assessment, 21 (20.8%) will involve patients, and 17 (16.8%) will not involve patients. Of those in the field-testing or implementation stages, 24 (72.7%) do not involve patients in resident assessment, five (15.2%) do involve patients, and four (12.1%) do not know if they involve patients. In Phase 2, 12 interviewees raised nine factors that facilitate or hinder patient involvement including, patients' interests/abilities, guidelines/processes for patient involvement, type of Entrustable Professional Activities, type of patient interactions in programs, and support from healthcare organizations.


Conclusion
Patient involvement in resident assessment is limited. We need to engage in discussions on how to support such involvement within CBD.


Introduction
Competency-based medical education (CBME) requires multiple assessors of residents, 1,2 authentic assessment, and multi-faceted assessment programs. 1,3 Given the emphasis that CBME places on assessment for learning as well as the range of competencies included in, for example, the CanMEDS framework, it is essential to develop assessment programs that encompass a range of assessment strategies and assessors. 1 Patients and family members or caregivers (herein referred to as patients), if provided with the opportunity, have the ability to contribute to these assessment programs. Residency programs can include assessments by patients as part of multisource feedback (MSF). 4 Used for the assessment of competencies, MSF can include the collection of data on residents' performance from peers, supervising physicians, allied health professionals, and patients. By including patient assessments in MSF, residency programs can ensure that patients' first-hand experiences with residents as well as their perceptions of residents' skills and abilities are included as part of the learning process.
Researchers suggest that residents need assessments from patients to improve their performance, 5 to develop and maintain patient-centred professionalism, 6 to develop positive physicianpatient relationships, 7 and to understand how their interactions with patients impact health outcomes. 8 Residents have reported high satisfaction with and utilization of patient assessments. 9,10 Patient involvement in resident assessment can also empower patients to improve the care that they and others receive. 11,12 In recognition of these mutual benefits, the Royal College of Physicians and Surgeons of Canada (RCPSC) recommends the involvement of patients in resident assessment in their CBME model, entitled Competence by Design (CBD). However, there is limited research on patient involvement in resident assessment. 13 Towle and Godolphin's bibliography on patient involvement in health professions education noted that a mere 29 of 657 (4.4%) studies focused on patient involvement in assessment, seven of which are from medicine. 14 These studies found that patients can assess residents ' professionalism, collaboration, interpersonal abilities, and communication skills. 7,[15][16][17] Authors also showed that patients can provide reliable assessments of residents' skills. 12,18,19 Nonetheless, these studies did not explore the extent and nature of patient involvement in resident assessment within the emerging Canadian CBD context. Thus, the purpose of this two-phase sequential explanatory mixed-methods study was to explore and document the current state of and plans for patient involvement in resident assessment within these CBD programs to identify resources and activities needed to advance and sustain this important component of resident assessment. The following research questions, as they related to CBD, guided our study: To what extent are patients or will patients be involved in resident assessment?
2. Why are patients or will patients be involved or not involved in resident assessment?
3. What skills and abilities can patients assess?
Phase 2: Interviews 4. What factor(s) facilitate and hinder patient involvement in resident assessment?

Study design
We used a two-phase sequential explanatory mixedmethods design. 20 Phase 1 encompassed quantitative survey data from program directors (PDs) whose programs were in the CBD preparation, field-testing, or implementation stages at the time of the study. We then used the findings from Phase 1 to inform participant-level questions and identify participants for Phase 2, which comprised qualitative interview data with selected PDs. Our rationale for using interviews was pragmatic. When designing the study, we recognized that it would be challenging to schedule focus groups with busy PDs but that one-onone interviews would allow the PDs to participate at times that were most convenient for them. Nevertheless, the quantitative and qualitative study portions were equally important in terms of addressing the research topic. 20 We opted for this mixed methods design because we anticipated that the qualitative interview findings (Phase 2) would build on the initial quantitative survey results (Phase 1) and thus, provide a more comprehensive understanding of patient involvement in resident e86 assessment than either quantitative or qualitative designs alone. 20 This design also enabled us to create and ask specific interview questions to PDs who responded "yes," "no," or "don't know" to questions of patient involvement in resident assessment on their Phase 1 survey and thereby elicit and illustrate accurate descriptions and understandings of the factor(s) that facilitate and hinder patient involvement in resident assessment. We obtained ethics approval for both Phases 1 and 2 from the University of Ottawa (File number 03-17-08).

Phase 1: Survey
Sample. Based on the RCPSC's website, we identified 17 primary specialties, 12 sub-specialties, and one special program (i.e., Surgical Foundations) that were in the CBD preparation, field-testing, or implementation stages. We distributed the online survey to 462 program directors (PDs).

Instrument development.
We developed survey dimensions and items using literature on strategies for investigating patient involvement in medical education 9,13,21,22 and assessment practices. 16,[23][24][25] Survey items pertained to one of the following dimensions: (a) level and type of patient involvement; (b) purpose of assessment; (c) reasons for patient involvement; and (d) skills and abilities. Six reviewers knowledgeable in patient involvement in medical education, assessment, and CBD reviewed the survey. We also piloted it with five former PDs and five physician-educators ineligible for the study. We then translated the 23-item survey to French and piloted it with three former French PDs. Data analysis. Two of us (KM & KE) analyzed the data ourselves and recognized that our interpretations of the data were mediated through us as medical education researchers and proponents of patient involvement in medical education and CBD. Miles' and Huberman's three-step iterative process (i.e., data condensation, data displays, drawing and verifying conclusions) 28 informed our inductive analyses. The goal of the analyses was to identify the major themes articulated by the PDs representing factors that facilitate or hinder patient involvement in resident assessment. Throughout the analyses, we identified factors that were present in more than one interview.

Data collection procedures. A Research
Following each interview, we independently listened to the audio-recordings to identify factors that interviewees highlighted as facilitators or hindrances to patient involvement in resident assessment. We documented these factors to establish an audit trail. Following transcription, the RA compared the transcripts to the audio-recordings. We then independently reviewed the transcripts and summaries to create our own coding systems. We read each transcript, annotated phrases, and coded the data. We incorporated codes not previously identified in our listening to the audio-recordings. We then met to discuss our independent analyses. Together, we merged our analyses, created data e87 display tables, and generated thematic conclusions, including exemplar quotations.

Phase 1: Survey
Demographics. We obtained 134 completed surveys (response rate of 29.0%). When asked about their programs' CBD stage, 101 (75.4%) indicated CBD preparation (i.e., the specialty committee is working with the RCPSC to prepare the program for CBD), 21 (15.7%) selected CBD field-testing (i.e., the program is field testing aspects of CBD), and 12 (9.0%) stated CBD implementation (i.e., residents are in a CBDbased program). Tables 1 and 2 provide demographic details.  To what extent are patients or will patients be involved in resident assessment? Of the 101 respondents in the CBD preparation stage, 63 (62.4%) stated that they did not know if patients will be involved in resident assessment, 21 (20.8%) said that they will involve patients, and 17 (16.8%) indicated they will not involve patients. Among the 33 respondents in the field-testing or implementation stages, 24 (72.7%) said that they do not involve patients in resident assessment, five (15.2%) stated that they do involve patients, and four (12.1%) stated that they do not know if they involve patients.
e88 Preparation stage. Ten (47.6%) of the 21 respondents who are in the preparation stage and who indicated that they will be involving patients in resident assessment stated that they did not know how often patients will be involved over a four-week period. The remaining respondents stated that patients will never (n=2, 9.5%), rarely (n=4; 19.0%), sometimes (n=4; 19.0%), or frequently (n=1; 4.8%) be involved over a four-week period. Twelve (57.1%) of these 21 respondents stated that patients will be involved in formative assessment, one (4.8%) stated that patients will be involved in summative assessment, and eight (38.1%) responded that they did not know if patients will be involved in formative or summative assessment. When asked if patients will provide written assessments, ten of the 21 respondents (47.6%) noted that they did not know, six (28.6%) indicated that patients will provide written assessments, and five (19.0%) said that patients will not. In regard to verbal assessments, over half (n=11; 52.4%) of the 21 respondents said "yes" patients will provide verbal assessments, seven (33.3%) said they "did not know," and three (14.3%) said "no." Four (19.0%) of the 21 answered that patients will assess residents in Objective Structured Clinical Examinations (OSCEs) and all confirmed that patients will not assess residents in simulation. All 21 respondents indicated that patients will not be involved in assessment tool development.

Field-testing or implementation stage.
Of the five respondents in the field-testing or implementation stages who stated that they involve patients in resident assessment, four (80.0%) stated that patients are rarely involved in the assessment of residents over a four-week period and one (20%) indicated that they are frequently involved. All five stated that patients are involved in formative rather than summative assessment. In terms of providing written assessments, three (60.0%) indicated that patients provide written assessments. When asked about verbal assessments, two (40.0%) said patients provide them. All respondents confirmed that patients are not involved in assessments for OSCEs or simulation. They also noted that patients are not involved in assessment tool development.

Why are patients or will patients be involved or not involved in resident assessment?
The main reason why respondents in the preparation and field-testing or implementation stages will be involving or do involve patients in resident assessment is to provide and gain firsthand information on the care provided by residents. This reason was cited by 16 (76.2%) of the 21 programs in the preparation stage and four (80.0%) of the five programs in the field-testing or implementation stages that will be involving patients in assessment. Other reasons indicated by respondents included: (a) to improve the quality of care (57.1% preparation stage, 40.0% field-testing or implementation stages); (b) to empower patients (28.6% preparation stage, 20.0% field-testing or implementation stages); and (c) to satisfy program accreditation requirements (19.0% preparation stage, 40.0% field-testing or implementation stages). Appendix A provides a list of reasons why patients might not be or are not involved in resident assessment. Table 3 presents respondents' views on which skills and abilities patients can (and cannot) assess. Across the CBD stages and irrespective of patient involvement in resident assessment, the majority of respondents thought that patients could assess residents' communication and respectfulness.

Phase 2: Interviews
Demographics. We interviewed 12 PDs; eight (66.7%) in the CBD preparation stage and four (33.3%) in the field-testing or implementation stages. Of the interviewees in the preparation stage, four (50.0%) reported that patients will be involved in resident assessment and four (50.0%) said that they did not know if patients will be involved. All interviewees in the field-testing or implementation stages noted that patients are not involved in resident assessment. The interviewees represented Anesthesiology, General Surgery, Medical Oncology, Neonatal-Perinatal Medicine, Nephrology, Obstetrics and Gynecology, Otolaryngology, Pediatrics, Respirology, and Surgical Foundations from various geographical locations. e89

What factor(s) facilitate or hinder patient involvement in resident assessment?
Factors that some interviewees viewed as facilitators others instead viewed as hindrances (see Table 4). Appendix B provides exemplar participant quotations.

Patients' interests and abilities.
All participants in the preparation and field-testing or implementation stages, irrespective of their desires to involve patients in assessment, thought that patients' interests and abilities to be involved will be/is a hindrance. They commented on how they believe that patients will/do not want to be involved in education or that they will/do not have the time to participate in assessment. Interviewees also expressed concern about the expertise of patients to assess residents. They noted that patients would need training in resident assessment and medical education requirements. Interviewees thought that providing such training is unfeasible given current program resources.
Funding. Regardless of the participants' CBD stage and their interest in involving patients in resident assessment, they all said that insufficient funding hinders them from involving patients. Participants commented on how they would require additional funding to implement and ensure ongoing patient involvement. They discussed how patient involvement is costly and impossible within existing budgets.
-indicates that the interviewees perceived the factor to hinder patient involvement in resident assessment; + -indicates that the interviewees perceived the factor to both hinder and facilitate patient involvement in resident assessment e91

Guidelines and processes for patient involvement in assessment.
Participants across all CBD stages viewed the lack of guidelines and processes as a hindrance to patient involvement in resident assessment. They noted the non-existence of guidelines and processes on how to collect resident assessment data from patients, which patients they should involve, how many patient assessments they should collect, what patient assessment reports should include, and if these reports should only include anonymized data. Several interviewees also commented on how the online assessment system will be/is restricted to faculty members and that there are no guidelines or processes on how patients will/can access it. They expressed that guidelines and processes would facilitate and standardize patient assessment activities but that without them, patient assessments will be/are haphazard or not occur. Moreover, most participants noted that they would not involve patients in resident assessment unless it was mandatory.

Faculty members' and residents' perceptions.
Some interviewees in the preparation stage described how their positive perceptions towards patient involvement in resident assessment would facilitate their efforts to involve patients. These participants believe that patients will bring unique insights to wellrounded assessment programs and provide residents with important information that they can use to improve their competencies. However, others in the preparation, field-testing, or implementing stages flagged that residents' and faculty members' prior preconceived negative perceptions towards patient assessments might hinder involvement. They commented on how they may not take it seriously. Participants also discussed how they perceive patient assessments to be biased (i.e., overly positive or negative) and thus, hinders them from collecting and using them. They noted that if they, as the PDs, are not supportive of patient assessments then they will not occur, as they champion the various assessment strategies. Furthermore, several of these interviewees disclosed that they do not see patient assessments as a priority.

Staffing and time.
Participants discussed staffing and time as factors that both facilitate and hinder patient involvement in resident assessment. They focused on the amount of time available for staff to collect patient assessments and the willingness of staff and residents to assume such responsibilities. Some interviewees in the preparation stage explained how they would ask clerks to distribute assessment forms to patients. They also suggested that nurses could assist or residents could collect the patient assessments. However, others within the preparation stage detailed how they will not have the staff or time to collect patient assessments. They discussed how the time commitments of teaching and clinical responsibilities would hinder them from involving patients in resident assessment. Those in the fieldtesting or implementation stages echoed these thoughts and added that they do not have anyone in their programs who can assume responsibility for overseeing and collecting patient assessments. They also explained that residents would forget to collect these assessments and thus, it would be impractical to assign the task to them.
Existence and availability of patient assessment tools. Participants whose programs are in the preparation stage thought that the existence and availability of patient assessment tools will be both a facilitator and hindrance to patient involvement in assessment. As a facilitator, some explicated how they plan to involve patients in resident assessment using existing tools within their 360° assessment systems. However, others in the preparation stage noted that they do not have access to or are unaware of existing patient assessment tools, especially those with validity and reliability evidence. Interviewees in the field-testing or implementation stages also reiterated their lack of access to and knowledge of patient assessment tools as a hindrance. They noted the need to develop patient-specific assessment tools that align with their existing Entrustable Professional Activities (EPAs), but they explained that the development of such tools is beyond their expertise.

Type of Entrustable Professional Activities.
Some participants in the preparation stage highlighted how the EPAs that they are developing for their programs will require or are amenable to patient involvement in resident assessment. In some cases, they described how it would be essential to obtain patient assessments to know if residents are progressing appropriately. Conversely, others in the preparation, field-testing, or implementation stages stressed how their programs' EPAs will hinder their abilities to involve patients because the EPAs are not patientoriented. They stated that the EPAs for their e92 programs do not require patient assessments. They explained how the EPAs focus solely on skills/abilities that only require faculty members' assessments. These interviewees also noted that they will/have not included patients as assessment sources and thus, patients will/are not involved.

Type of patient interactions in program.
Several participants in the preparation stage discussed longitudinal patient interactions in their programs. They believed that these interactions would facilitate their abilities to incorporate patient assessments and enable patients to comment on residents' competencies over time. However, other interviewees in the preparation, field-testing, or implementation stages commented on how their types of patient interactions are not conducive to patient involvement in resident assessment. They explained how interactions are episodic or short and therefore, patients will/do not have enough time with residents to provide reliable and constructive assessments.
Support from healthcare organizations. Participants in the preparation stage who plan to involve patients explained how support from the academic hospitals in which their residents are training would be essential for facilitating patient involvement in resident assessment. They noted that they are involving administrators as well as patient advisory groups in preparation efforts to understand how they can best involve patients. These interviewees detailed how these stakeholders will facilitate their programs' efforts to contact patients for assessment activities. They discussed how these stakeholders are also providing them with advice on how to respect patients' privacy concerns when involved in resident assessment. Participants mentioned that the patientcentred philosophies of their academic hospitals would support patient involvement in resident assessment. Nonetheless, those in the field-testing and implementation stages said regardless of their hospitals' philosophies, they do not have staff or mechanisms to assist and guide them in involving patients in resident assessment.

Discussion
This study provides an understanding of the current extent, nature, as well as the facilitators and hindrances of patient involvement in resident assessment. The majority of Phase 1 respondents in the CBD preparation stage indicated that they did not know if patients would be involved in resident assessment. This uncertainty is not surprising as these programs are still in early CBD stages and navigating the transition process. However, the majority of respondents in the field-testing or implementation stages indicated that they do not involve patients in resident assessment. This finding does not bode well for patient involvement in resident assessment, as many of the programs in the preparation stage may look to these early CBD adopters, who are piloting the model to ensure that it is appropriate and feasible before implementing it across all programs, to inform their programs' assessment activities. 29 This lack of patient involvement in assessment also differs from the methods used to assess residents in Accreditation Council for Graduate Medical Education (ACGME) programs in the United States. 24 Holt et al. in their review of methods used to assess residents in ACGME programs found that 61% of programs involved patients as assessors and that 34% included family members as assessors. 24 All respondents in the present study also noted a lack of patient involvement in the development of assessment tools. This finding is unfortunate as research shows that patients can contribute to tool creation. [30][31][32] Moreover, participants in Phase 2 commented on having no access to or awareness of patient assessment tools. However, there are several existing patient assessment tools with validity and reliability evidence that programs could use as part of their MSF activities. 31,33-37 Nevertheless, researchers and other RCPSC stakeholders could better promote these tools to PDs to facilitate use. In addition, prior to using any existing tools, programs need to ensure that the tools' items align with their programs' EPAs, as these EPAs should be used as the blueprints that guide tool selection. 38,39 On another note, participants in the current study who will or do involve patients in resident assessment noted how patient involvement provides patients' perspectives to residents, improves care provision, and empowers patients. These benefits are consistent with the literature on patient involvement in assessment. 7,40-42 That said, some participants discussed how they will or are only involving patients in resident assessment to satisfy program accreditation requirements. Unfortunately, this form e93 of involvement often leads to token patient involvement. 8 Phase 1 participants identified that patients could best assess residents' communication and respectfulness, which is congruent with previous studies. 7,16,24 In relation to the assessment of these skills, interviewees also noted that opportunities for longitudinal patient-resident interactions would facilitate patient involvement. Researchers have shown that a series of longitudinal patient assessments can illustrate learners' evolution of patient-oriented behaviours and communication abilities. 43 Conversely, other interviewees detailed how resident-patient interactions in their programs are episodic or short and thus not conducive to patient involvement in resident assessment. However, as we have shown in a previous study, parent assessments of residents is feasible in pediatric emergency departments, where interactions are brief and isolated. 31 In both Phases 1 and 2, regardless of CBD stage or intentions of involving patients, participants indicated several reasons why they might not or do not involve patients in resident assessment, including lack of funding and time. Holmboe et al. 38 and Pinsk et al. 44 confirm that CBME and CBD, respectively, lead to increased demands on faculty and resources and that assessment efforts are time-intensive. Thus, adding patient assessments is undoubtedly challenging. Participants also suggested that their lack of knowledge on how to involve patients in resident assessment is a reflection of a dearth of guidelines and processes on how to do this. Such guidelines and processes can support a culture of patient inclusion 8,12 and include information on remuneration for patient assessors, 17 strategies for increasing the diversity of patients involved in assessment, 45 and data collection techniques. 32 Of interest, Phase 2 interviewees highlighted hindrances and facilitators to involving patients in resident assessment that were not strongly represented in Phase 1. For example, while a small number of Phase 1 respondents indicated that their programs do not believe patients can assess residents, all Phase 2 participants thought that patients' interests and abilities will be or are a hindrance to patient involvement in resident assessment. Interviewees focused heavily on patients' lack of expertise to assess residents. Other researchers have commonly cited concerns about overburdening patients or about patients' abilities to provide meaningful and reliable assessments because of their emotional, physical, or mental health. 46 Few have commented specifically on patients' expertise. Those that have alluded to patients' expertise suggested that faculty members may feel threatened by the transfer of some assessment power from themselves to patients and thus, resist it, 47 or commented that programs have provided minimal or inappropriate training to patients to enable effective assessment involvement. 48 Interviewees also expanded on staffing as a factor that can both hinder and facilitate patient involvement in resident assessment. Some noted that their programs will not or do not have the staff to collect patient assessments. Naylor et al. 17 echoed these participants' concerns noting that the Canadian context has deficiencies in infrastructure for supporting authentic patient involvement. However, other interviewees also commented on the potential of having other health professionals and residents assist with patient assessments. Previous research studies 35,36, 49 have successfully used such processes, but it is unclear how these processes would work in Canadian healthcare systems where health professionals' job descriptions do not include the collection of patient assessments.
Lastly, interviewees raised the importance of having leadership and support to facilitate patient involvement from both the healthcare organizations in which the residents provide care and the PDs themselves. A lack of sustained leadership and support is one of the main reasons why patient involvement in medical education is not mainstream. 45 In order to facilitate such involvement, Towle et al. 45 endorse the promotion of patient involvement "through directives such as accreditation standards, external and internal policies, pronouncements from professional bodies and best practice statements."

Limitations & Future Directions for Research
This study has four limitations that future research can mitigate. First, some may view the survey response rate of 29.0% as a limitation. While it is plausible that the data would have yielded different results if the response rate was higher, the response e94 rate is consistent with other surveys targeted at PDs 50 and those involving healthcare professionals. 51 However, future research in this area could work on increasing this response rate. Second, since we limited the survey to the perspectives of PDs for specialty/sub-specialty/special programs associated with the RCPSC, in future, it would be interesting to survey PDs for Family Medicine programs. Third, many respondents from Phase 1 elected to not participate in Phase 2. We do not know if there were any differences between those who participated in Phase 2 and those who did not. It is possible that PDs who participated where more interested in the topic and thus, expressed different views from those who did not participate. Lastly, although Phase 2 included a small group of PDs, it included the views of PDs from several specialty/sub-specialty/special programs and various geographical locations. Nevertheless, it would be beneficial to undertake future studies to gather additional PDs' perspectives on the topic. Moreover, patients and residents are key stakeholders on this topic, and future studies should also explore their views. Such studies could use focus groups, rather than one-on-one interviews, in order to facilitate the obtainment of high-quality data, as the interactional, synergistic nature would encourage these stakeholders to clarify or expand upon their discussion points in relation to those raised by others.
Patient involvement in resident assessment appears limited and sporadic across Canadian specialty, subspecialty, and special programs that are transitioning or have transitioned to CBD. Unfortunately, the majority of respondents whose programs are in the CBD field-testing or implementation stages indicated that they do not involve patients in resident assessment. This lack of involvement will inevitably have a deleterious impact on the extent of patient involvement in resident assessment, since other programs may follow the examples and activities of these early CBD adopters. The PDs also identified factors that facilitate and hinder such involvement. Overall, by highlighting the current state of patient involvement in resident assessment as well as these factors, we are optimistic that those leading CBD transitions will be motivated to engage in critical discussions about the extent to which, how, and why we need to enact measures to better support patient involvement in resident assessment. After all, since patients are central to healthcare and medical education, residents may benefit from and appreciate the involvement of patients in their assessment.
Conflicts of interest: There are no conflicts of interest for any of the authors.  -Don't even qualify to assess a doctor or resident (N01). -In a clinical encounter often we're running late whenever you have residents in your clinic and the patient tries to leave as soon as they can when they're done right….won't want to stay to assess (N02). -Should we ask our patients admitted on the ward? But they are some of the sickest patients. Are they really going to feel like completing an assessment…? So…I just don't know (N03).

Funding
As a facilitator -NA -NA As a hindrance -It's very expensive…it would be a massive undertaking…think about how that's going to financially pan out…they're not supplying any of the money that's going to get my faculty interested….I suspect that unless there is some significant investment in some way this is going to kind of fall off the map (DK01). -Funding is something that is required to be able to do this…I don't necessarily think we have it (Y03). -Funding is a big thing…big part of the problem (Y04).
-The funding is going to be the same, all the onus is on…is on me [Program Director] to reorganize everything without any additional resources (N01). -It's too hard to figure out the logistics and the funding (N02). -It's resources! They give us about 25 cents during the year to run the program… (N03).

Type of Entrustable Professional Activities (EPAs)
As a facilitator -In each EPA there's room for it….the direct patient interaction type of EPAs (DK04).
-NA e101 -In certain EPAs, you know, you would want to have so many assessments from the family…in order to be deemed competent in that EPA (Y01). -I think some of the EPAs would definitely lend themselves to that…you know there's a whole ton, I think that would lend themselves to patient involvement (Y03). As a hindrance -It's going to be a bit of a challenge to make sure that we are directly dealing with the specific questions asked in each EPA (DK04). -EPAs not patient-oriented at this point…It doesn't really fall nicely into EPAs….the way the EPAs have been kind of developed and written out there just potentially is no place for it (Y04).
-The EPAs that we've been given do not include patients in the assessment (N01). -There's nothing in our EPAs that required patient involvement (N02). -When we are mapping out the EPAs and we are tying them to learner experiences, there is no place for patients right now. I don't see there is much place for patient-specific feedback (N04).

Type of patient interactions in program
As a facilitator -If I'm going to have a patient fill out a form to assess my residents, I'd rather have that longitudinal relationship (DK02). -There's a longitudinal clinic…I mean they [patients] could easily do that in that situation (DK04). -We try to match up the residents with a family that they follow over time…. This would be good for families assessing (Y01). -NA As a hindrance -They may see the person [patient] in the clinic but not see the person on the floor or they be involved in the patient's care through the emergency department while they're on-call and never see the patient again…problematic to get reliable feedback (DK01).
-We provide indirect or short patient care like initial consults (N01). -The question is how are we going to do this?
Because currently we have nurses in the operating room, on the ward, and in the outpatient clinic that assess the residents, in addition to the clerks of the outpatient [program specialty] clinic. But the interaction [of the resident] with the patient in the outpatient [program specialty] clinic isn't very long. Will the resident choose to give their assessment just to patients they clicked with and it went very well (N03)? -If we look at most [program specialists], it's mostly day surgeries. You know, the patients they don't stay that long at the hospital (N03). -The relationship we get with the patient is brief and intense and most the time they don't remember who we are. We don't have an opportunity to build a trust opportunity and there is really no way we could get our faculty to get onboard with patient assessment because of this (N04). -They [patients] don't have a face or name relationship with us (N04).

Support from healthcare organizations
As a facilitator -Involving our [hospital] administration....so they feel we are respecting the patient's privacy…and that we're not asking for anything that is going to violate privacy (DK04). -It's the family-centred care in the hospital…we cannot just separate the parents from we do (DK03).
-NA e102 -In pediatric hospitals whereby we're probably more likely to buy in to the concept of involving patients and families because of family-centred care (Y03). -I know over here, you know, patient advisory groups and family advisory groups and things like that it would help in terms of recruiting patients (Y03). -I'm told there's a pool of patients ready to this work. They [hospital] have an infrastructure for contacting and indexing them according to what they have….They have all the contact information (Y02). As a hindrance -So you know, I don't know how the hospital would react or feel about this. They have no resources to help us…this is especially an issue for them on how the data will be anonymized (DK01).
-Some institutions themselves just happen to have some resources available to them, say for example there may already be sort of patient advisory groups and things like that so maybe pre-existing pools of patient…to engage them in different ways in assessment….we don't have that (N02).
Note. Y beside the participant identification number indicates that the interviewee's program will be involving patients in resident assessment. DK beside the participant identification number indicates that the interviewee's program does not know if it will be involving patients in resident assessment. N beside the participant identification number indicates that the interviewee's program does not involve patients in resident assessment.