Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project

Background Canadian specialty programs are implementing Competence By Design, a competency-based medical education (CBME) program which requires frequent assessments of entrustable professional activities. To be used for learning, the large amount of assessment data needs to be interpreted by residents, but little work has been done to determine how visualizing and interacting with this data can be supported. Within the University of Saskatchewan emergency medicine residency program, we sought to determine how our residents’ CBME assessment data should be presented to support their learning and to develop a dashboard that meets our residents’ needs. Methods We utilized a design-based research process to identify and address resident needs surrounding the presentation of their assessment data. Data was collected within the emergency medicine residency program at the University of Saskatchewan via four resident focus groups held over 10 months. Focus group discussions were analyzed using a grounded theory approach to identify resident needs. This guided the development of a dashboard which contained elements (data, analytics, and visualizations) that support their interpretation of the data. The identified needs are described using quotes from the focus groups as well as visualizations of the dashboard elements. Results Resident needs were classified under three themes: (1) Provide guidance through the assessment program, (2) Present workplace-based assessment data, and (3) Present other assessment data. Seventeen dashboard elements were designed to address these needs. Conclusions Our design-based research process identified resident needs and developed dashboard elements to meet them. This work will inform the creation and evolution of CBME assessment dashboards designed to support resident learning.


Introduction
The Royal College of Physicians and Surgeons of Canada has implemented competency-based medical education (CBME) through the Competence by Design (CBD) program. 1 Programmatic assessment is a core element of CBD. 2 It requires faculty to provide frequent, low-stakes assessments of entrustable professional activities EPAs) which include both a numerical rating of resident entrustability and narrative feedback. [3][4][5] Within emergency medicine (EM), 28 EPAs are mapped across four stages of training. 6 Since CBD's implementation, the volume of assessments has increased substantially relative to the traditional assessment program. 6-8 As a result, the importance of effective presentation of assessment data has been characterized as critical within the early CBME reviews, commentaries and program evaluation literature. [9][10][11][12][13] EPAs can be used both for the assessment of learning and for learning. 14 Assessment for learning occurs through reflective practice and self-regulated learning. 6,15-17 which requires a clear and accessible e33 presentation of assessment data. Fortunately, sophisticated analytical and visualization techniques have been developed in other professional fields (e.g. business and sport) that effectively present large amounts of data. 18 Dashboards are often used to visually display important information and allow its monitoring. 19,20 They have been used in medical education to present learning analytics data to learners to enhance their experience. 21,22 We sought to develop a dashboard to support resident learning using an iterative design-based research process 23,24 that incorporates collaboration with and feedback from information technology experts, assessment experts, data managers, and dashboard users. 11 Within the University of Saskatchewan emergency medicine residency program, we investigated how our residents' CBME assessment data should be presented to support their learning and developed a dashboard meeting their needs.

Methods
To meet this objective, we employed a design-based research process 25-27 and followed best practices in dashboard design 11,28 including the collection and analysis of iterative feedback from our resident stakeholders. We report the results of our qualitative analysis in compliance with reporting standards for qualitative research. 29,30. Our research methodology was deemed exempt from ethical review by the University of Saskatchewan Research Ethics Board (BEH ID 463).

Settings and participants
This project was situated within the Royal College of Physicians and Surgeons of Canada EM residency program at the University of Saskatchewan between July 1, 2018 and September 30, 2019. All residents in our Royal College EM program transitioned to the EPA-based Competence By Design program in July of 2018. There were 14 full time residents enrolled within the program from then until June 30, 2019 and 18 enrolled from July 1-September 30, 2019.
All resident participants were asked via email by their colleagues (RC and GW) to participate in focus groups that were held in our EM Resident Library. Participation in focus groups was voluntary and food was provided at each session.

Research team
Our research team included two residents (RC and GW), an established medical education researcher (BT), our program director (RW), our competence committee chair (LM), an external expert in medical education research and assessment (TMC), a computer science research assistant (VB), and a computer science professor (DM).

Design based research process
The Design-Based Research methodology (11,24) that we employed aligned with the approach outlined in our previous work on competency committee dashboards. 31 Design-based research is an "authentic, contextually aware, collaborative, theoretically focused, methodologically diverse, practical, iterative, and operation-oriented" process 11,25 which aims to bridge practice and research in education through the integration of investigation and intervention. 24,25,27 Our process consisted of the four phases of design-based research. 25

Phase 1. Analysis and exploration
The senior author (BT) reviewed the literature on reflective practice and self-regulated learning, 15-17 learning analytics, 18,32 and data visualization 11,20,23,32 to generate ideas for the initial iteration of the resident dashboard. In November 2018, our resident investigators (RC and GW) facilitated simultaneous focus groups lasting 64 minutes. Data collection included field notes taken during the meeting, transcribed audio recording, and visuals drawn by the participants. The guiding questions asked of the focus group were: What assessment data does our program collect? Is this assessment data used to guide your learning? If so, how? How could this assessment data be presented to support your learning?

Phase 2. Design and construction
The initial focus group data were transcribed and qualitatively analyzed to inform the creation of a dashboard prototype. Two authors (VB and BT) then met between two and four times monthly to discuss the results and design the prototype dashboard. Given the overlap between the needs of the competence committee 31 and the residents, the initial dashboard was very similar for each of the groups. The first dashboard prototype was released to residents in December 2018.

Phase 3. Evaluation and reflection
Phases 2 and 3 alternated into the next year with each of the additional three focus groups spurring the creation and evolution of dashboard elements. The subsequent three focus groups were held in March, June, and September of 2019 and lasted 48, 30, and 38 minutes, respectively. During each, the EM residents reviewed the dashboard and were asked: How are you using the dashboard? What information needs to be added? How could the assessment data be presented to better support your learning? Following each of the focus groups, the narrative data was transcribed and qualitatively analyzed to inform the development of the thematic framework and the evolution of the dashboard (Phase 2).

Phase 4. Implementation and spread
The final phase describes the implementation and spread of the dashboard at and beyond our institution. Uptake in other contexts demonstrates the generalizability of our findings and dashboard to broader contexts. As this phase does not contribute directly to the determination of resident needs, we did not include it with our results.

Qualitative analysis
Narrative data from the focus groups included drawings, field notes taken by the facilitators, and transcribed audio data. The data was analyzed through a constructivist grounded theory approach to identify the core needs for resident assessment data. Dashboard elements (data, analytics and visualizations) were designed to meet these needs and spurred discussion at subsequent focus groups regarding the optimal presentation of the data they included. Comments related to resident perspectives on their progress and general comments regarding CBD were excluded from the analysis.
The qualitative analysis was conducted using a constructivist grounded theory approach and constant comparative method. 33 Following the first focus group, two authors (RC and GW) independently developed codebooks with representative quotes for each code. They then met and amalgamated their codebooks by adding, modifying, and removing codes on a consensus basis. One author (RC) compiled the codes into a preliminary framework of resident needs. Following each subsequent focus groups, the same authors coded the data and refined the thematic framework while selecting representative quotes for each need. BT reviewed all the transcripts, codes, and the framework intermittently to ensure that it was comprehensive and representative of the data. He provided additional suggestions to refine the thematic framework throughout the analysis process. He also liaised directly with the programming team (DM and VB) to prioritize updates to the resident dashboard. The resulting thematic framework was described using representative quotes as well as images of the dashboard elements mapped to each theme.
The investigators considered their own positionality and its potential impact on their data interpretation throughout the coding process. RC and GW were both emergency medicine residents within the program who regularly utilized the dashboard during most of the coding period (GW transferred to another residency program on July 1, 2019 but continued to contribute to the data analysis until the project was complete). BT is an emergency physician who was a Residency Program Committee member during the period of study. He previously served as the Program Director, CBD Lead, and Competence Committee Chair of the residency program. We acknowledge that the close involvement of the three coding investigators with the residency program was likely to impact their interpretations of the data, however, their involvement in this capacity was a pragmatic decision that allowed rapid and contextualized coding and reduced delays in the iterative dashboard design process.
Participant checks occurred in two ways. First, each of the residents were asked to review the thematic analysis and provide feedback on anything that was out of keeping with their perspective. Second, the residents were consulted throughout the dashboard development process and had the opportunity to provide feedback when the dashboard elements did not meet their needs.

Data management and dashboard programming
All EPA assessment data for our residency program was entered by faculty into the Royal College of Physicians and Surgeons Mainport ePortfolio (Ottawa, ON). The data was then exported and uploaded to the dashboard each Monday by the emergency medicine Program Administrator. During the upload process, EPA data was reformatted and e35 tagged with the rotation each resident was in when each EPA was completed. Contextual and non-EPA information (e.g. resident name, program start date, phase of training, rotation schedule, exam scores) was entered into the dashboard by the Program Administrator. All dashboard data was stored on a secure server in the Department of Computer Science at the University of Saskatchewan.
The dashboard was developed on a distributed web architecture with three components: a database server to securely hold the data, a web server for hosting the website, and a back end server to authenticate users and perform CRUD (create, read, update, and delete) operations. 31 This allowed each of these parts to be updated independently, which facilitated rapid prototyping based on user feedback. This also allowed the dashboard to be easily adopted by additional programs. Dashboard visualizations are rendered in a scale and transform invariant Scalable Vector Graphics (SVG) format that make the user experience consistent across various screen sizes and orientations. Logging into the dashboard required authentication through the University of Saskatchewan's Central Authentication Service. Access to data was restricted based on pre-assigned user roles with residents restricted to viewing only their own data. The dashboard source code was published on GitHub 34 under an open access license to allow its rapid replication by other institutions. There are no plans to commercialize the dashboard.

Results
Seven to 10 residents from a variety of postgraduate years participated in each of the four focus groups ( Table 1). The preponderance of male residents reflected the gender balance of the residents in the residency program during the period of study (12 male and two female at the time of the first three focus groups; 12 male and five female at the time of the final focus group).
The qualitative analysis identified three resident needs ( Table 2). Each need has been described with representative quotes and linked to their corresponding dashboard elements in Table 2. Given the limitations of tables and figures, a video was developed to provide a dynamic demonstration of the dashboard and how it addressed each resident need (Video 1).

Provide guidance through the assessment program
In all focus groups, the residents were asked how they used the dashboard and what they used it for. Reinforcing the importance of this need, most residents indicated that they used it to obtain a sense of their overall progress throughout the residency program.
One simple element which was incorporated to provide guidance was a calendar with an incorporated rotation schedule that was always kept up to date ( Figure 1). By clicking on the 'View History' button just above the calendar of the current year, rotations from prior years can be viewed. This provided a quick reference for which rotations have been completed each year. Additional features were also incorporated into the rotation schedule to provide the residents with further guidance on the assessment program.  Table 2. Thematic analysis of resident needs and the dashboard elements developed to address them.

Resident Needs
Dashboard Element Quotes 1. Guidance through the assessment program 1.1 Calendar (Figure 1 The residents wanted to track the number of EPAs that they have completed during each block and whether this was an adequate amount. Selecting the 'View EPAs/Block' button at the top of the rotation schedule ( Figure 1) produces a visualization containing the number of EPAs observed on a given four-week block along with a heat-mapped percentage outlining how that number compares to the value expected by the program. Each percentage was calculated by dividing the number of EPAs observed for the resident on that block by the number of EPAs that they should be targeting for the rotation. The target number for each rotation was based upon the number of EPAs our program's residents have historically had observed on that rotation. In some cases, this value was modified by the program director (RW). A diverging color scale ranging from red (<25% of the expected number of EPAs) to green (>80% of the expected number of EPAs) allowed the residents to monitor their EPA acquisition relative to expectations over time.

Figure 1. Visual representation of the rotation schedule and the number of Entrustable Professional Activity observations per block observed relative to the expected number of observations per block.
Program-specific reference cards 35 were also incorporated into the rotation schedule (Figure 2). Clicking a rotation on the schedule loaded a reference card which guided the resident by indicating which EPAs should be targeted during that rotation. The design of the reference cards was based upon our program's curriculum map and described in detail in a previous manuscript. 35 The residents were told by our program leadership that they should aim for a minimum of two EPAs per week, so they were interested in tracking the number of EPA observations they received weekly. These data were visualized through numerical acquisition metrics and a graph visualizing weekly EPA numbers during the past 6 months (Figure 3). A date filter was available for this graph to provide the number of EPAs per week received in each interval and visualize it on the graph with blue highlighting.
Competence committee decisions and feedback were important to contextualize the residents' progress. They were presented within the dashboard using a graph outlining the state of the resident's progress (accelerated, as expected, not as expected, not progressing, or inactive) (Figure 4). Hovering over each data point displayed the competence committee's feedback. The start of each stage of training was incorporated into the graph with vertical lines indicating the date of promotion between stages.  Within the dashboard the residents sought indications that they had demonstrated adequate competence within individual EPAs so that they could increase their focus on other EPAs. As the final determination of competence was determined by the competence committee by stage, rather than by individual EPA, it was not possible for the dashboard to display this. This feature may be incorporated into future iterations of the dashboard if the competence committee modifies their approach to competence determination.
One feature discussed at length by the residents but not incorporated into the dashboard was a "To-Do" list. The residents noted that they had numerous nonclinical responsibilities within the program (e.g. courses, poison control shifts, research, etc.) that they found difficult to keep track of. They requested that an interactive dashboard element be developed outlining these requirements and whether they had been completed within each year. This feature was not developed for this iteration as our focus was on the resident assessment features.

Present workplace-based assessment data
The residents wanted a graphical visualization of their assessments for individual EPAs. During the first focus group they suggested many ways that this could be represented graphically. The final visualization was a line chart of each EPA ( Figure 5). Individual EPA assessments are presented as dots with the oldest on the left to newest on the right. Each dot is plotted on a Y-axis rating their entrustability using a 5-point entrustability score 36,37 (with 5 corresponding to "I did not have to be there" and 1 corresponding to "I had to do it"). Contextual information was also incorporated including the number of EPAs that needed to be observed within the assessment program, the number that had been observed, the number that still needed to be observed, and the number of each EPA that expired (EPAs that were sent e39 to a Faculty member by the resident but not completed). Some EPAs must be observed for specific clinical presentations or patient age groups. Tracking these presentations and patient demographics was a challenge. We added a filter option which allowed the resident to highlight data points tagged as a specific clinical presentations or age group in red ( Figure 5). This allowed residents to focus their requests for assessments on clinical presentations or patient demographics that had not previously been observed. Another challenge that the residents identified was the accessibility of the narrative comments associated with each EPA observation. We facilitated the review of these comments by developing two ways to look at narrative data within the line graph. First, hovering over the dots created a pop-up window containing the narrative comments and contextual information associated with that EPA observation ( Figure 6). Second, clicking an icon in the bottom left corner of the graph presents the narrative data in a sortable, searchable tabular format ( Figure  6). Both ways of viewing the narrative data met the needs identified by the residents in the focus groups with the former used for quick lookup or in-depth review. A separate line chart was developed which displayed all EPAs that had been observed recently (e.g. last 10 records, last 25 records, last one month of records, last three months of records) (Figure 7). The number of EPAs presented was selected arbitrarily but felt to be useful by the residents. The time intervals were used to correspond to the length of a single fourweek rotation and the timing of meetings with academic advisers (every three months). As in Figure  6, hovering over each of the dots displayed a pop-up window containing each EPA's associated narrative comments and contextual information. Residents often struggled to determine which preceptor(s) had been sent EPAs that they did not complete. We developed an expired EPA element to resolve this (Figure 8). Positioned at the bottom of the dashboard, it presents a table outlining the date each EPA expired, the number of the expired EPA, and the observer that it was sent to. This allowed the resident to be proactive in acquiring assessments as they could follow up with the attending physicians with outstanding EPAs.

Present other assessment data
The residents found other assessment data useful in tracking their learning and progress through residency and wanted all their assessment data available for review in a single place.
Exam scores (Figure 9) were presented as plots over time with the numerical scores out of 100 indicated by each data point. Written exam scores were recorded as percentiles compared to other residents of the same year across Canada. All previous exam scores were contained within a single written exam graph. Oral exam scores were presented in a similar fashion but with each dot representing the score on a single oral exam station rather than the overall score for a year. Each academic year could be selected using a drop-down menu. Hovering over each oral exam data point presented the narrative comments provided by the examiner for each oral exam. In addition to EPA observations, faculty within our program can submit general narrative assessments focused on either a resident's overall progress or on a particular event. Neither of these assessments are associated with a specific EPA. The residents identified that they wanted access to these assessments to supplement their EPA assessments. These data were presented in a searchable and sortable tabular format (Figure 10) containing the date submitted, the name of the faculty who submitted it, and the narrative comment.

Implementation and spread
The resident dashboard has been utilized by other local residency programs (Anesthesia, Pathology, Obstetrics and Gynecology, Internal Medicine, General Internal Medicine, and Neurosurgery) and modified for the University of Saskatchewan undergraduate medical education program. Nationally programs across Canada have expressed interest in the project. The senior author (BT) has met with representatives from the Royal College of Physicians and Surgeons, the University of Calgary, Laval University, the University of Alberta, the Elentra Consortium, and multiple national educators to discuss how it could be used by or might inform the work of other institutions with a similar goal of supporting resident learning. Resident Doctors of Canada has also requested additional information regarding the project via email. Feedback from these interactions was not incorporated into the qualitative data analysis as it was outside of the scope of the research ethics application and did not come from the perspective of resident learners.

Discussion
Building on our previous work exploring the needs of faculty members who serve on CBD competence committees, 31 we utilized a design-based research method to identify resident needs for data, analytics, and visualizations of their assessment data and created a dashboard to facilitate resident learning. This paper utilized design-based research to identify and address resident needs in a competency-based medical education program. While early program evaluation efforts have identified this type of data visualization as essential, 9,10,13 many of the studies on resident learning are largely theoretical. 12,15,16,18 More advanced work has investigated the use of dashboards to support the reflection of radiology residents when recording their exposure to specific radiographic findings. 38 Another study evaluated the McMaster Modular Assessment Program (McMAP) and found that EM residents valued the feedback generated by CBME and that workplace-based assessments allowed them to engage in informed e42 self-assessment. 39 The McMAP assessments generated improved verbal and written feedback from faculty to learners, 40 which then became a platform from which a trainee could springboard a data-driven approach to their own self-assessment. 39 Similarly, while the impact of the dashboard was not formally evaluated, the focus group comments suggest that it enhanced resident reflection and helped them create learning goals, thus supporting self-assessment. Such reflection is essential for resident development 16 and thought to support selfregulated learning . 15,17 While there are numerous overlapping elements between our competence committee and resident dashboard, it is important to note that they did identify different needs. This suggests that the use of a single dashboard for both groups without modification is not ideal. 31 For example, the residents discussed the potential value of anonymized normreferenced data, but the consensus was that it would foster competition within an otherwise cohesive resident group. The residents were also less focused on acquisition metrics (Figure 3) than the committee members. Conversely, residents were more focused on specific information regarding expired EPAs ( Figure 8) and facilitated guidance through assessments (Figures 1-4). The expired EPAs table (Figure 8) allowed the residents to follow up with assessors. These features were not as helpful for the competence committee. Both groups had their needs addressed by the inclusion of non-workplace-based assessment data (Figures 9-10), the tracking EPA assessments over rotations (Figure 1), and the individual EPA visualization (Figures 5-6).
There were several advantages to our research approach. It provided a scholarly framework within which we were able to support the creation and evolution of the dashboard, investigate residents' needs within a CBME program, and provide detailed visualizations to contextualize these needs. Conducting this study in a scholarly (rather than commercial) environment allowed direct access to the dashboard's end-users while also facilitating its open-access publication for use by anyone with adequate technical expertise. 34

Future directions
We anticipate that resident dashboards will become an important feature of CBME assessment programs as programs increasingly seek to utilize assessment data for learning. 14 As CBME dashboards are adopted more broadly, research will be needed to quantify the impact of their designs on resident on resident reflection, learning, and self-regulation as well as their impact on residents' relationships with their mentors/coaches. Dashboard designers should pay attention to the potential for their own perspectives and biases to be perpetuated within the design of resident dashboards in ways that could be detrimental to learning. Having developed dashboards to address the needs of both competence committee and residents, we plan to continue our design-based research process to investigate the needs of faculty developers and program leaders who are supporting CBME assessment programs.

Limitations
There were limitations to our project. First, the data were collected within one EM residency program at a single center in Canada which may limit its applicability to other disciplines or programs. However, the specifications of Competence by Design are consistent across Canadian medical specialties and its enthusiastic use by multiple non-EM training programs suggests that many of our findings are transferrable. Second, while we are confident that our thematic framework provides a robust representation of resident needs, as the residents gain more experience with CBME the dashboard elements will likely need to evolve. The involvement of residents (RC and GW) in the data collection process may have biased the results as they were active members of the program that was studied. We attempted to remediate this through the inclusion of a non-resident investigator (BT) in the qualitative analysis process, but he is also actively involved in the program in an academic role. We also note potential gender bias in our focus group participants due to the gender breakdown of our program. Notably, during development the substantial overlap between the competence committee and resident dashboards resulted in a core dashboard that served two purposes. Lastly, while there were no substantive design conflicts between the two groups, occasionally features were added to the core dashboard that were more desired by one of the groups than the other. This led to the dashboard being influenced by both e43 the competence committee focus groups and resident focus groups.

Conclusions
This project identified multiple needs for the presentation of assessment data to residents within CBME programs. The resulting dashboard and its supporting thematic framework should inform the development and evolution of resident dashboards at other institutions.