Building a CTU Orientation Handbook iPad® application for first-year residents.

Background
The General Internal Medicine Clinical Teaching Unit (CTU) is a challenging rotation for new residents and the optimal format of orientation has not been determined. We hypothesized that an iPad® application (app) would be a useful reference tool after residents completed their traditional large group orientation.


Methods
Postgraduate year 1 (PGY1) residents were sent a link to download the free app one week before the start of their rotation. A pre-usage survey at initial login collected basic demographics. Usage data was collected to determine the sections, duration, and the timeframe from which the app was utilized.


Results
Pre-usage survey data revealed that 63% of participants were female, 69% felt the app would improve orientation, and 94% were comfortable using mobile technology for medical education. Usage data showed "Teaching Sessions and Schedules," "The Consult Note," and "Admission Orders" were the three sections most commonly used. The most usage was during the evening call shift (10pm to 6am), followed by the morning shift (6am to 5pm).


Conclusion
The CTU Orientation App was a useful supplement to the traditional orientation. Researchers may not be able to predict what content would be most valuable in an iPad® app, thus pre-development needs-assessments and usage feedback are crucial.


Introduction
The General Internal Medicine Clinical Teaching Unit (CTU) is known to be a challenging rotation because of high patient load, high acuity, and complexity of patients. At the Ottawa Hospital, the CTU orientation is a one-hour PowerPoint ® lecture that is presented at the beginning of each rotation to all trainees by chief internal medicine residents. Many topics are covered in that time frame such as clinical responsibilities, call schedules, rotational objectives, and trainee question period. Our rotational evaluations indicate residents find their CTU orientation overwhelming. Lack of standardization, flexibility, accessibility, and information overload prevent effective orientations. [1][2][3] An ideal orientation should be comprehensive, focused on practical issues, frequently updated, standardized, and accessible regardless of time and place. Other modalities have also been explored, including online modules and videos, 2 simulations, mock clinical scenarios, 4 multiple choice sessions, 5 and web-based clinical orientations 3 , but they too fall short.
Since all trainees at our institution have access to the iPad ® for clinical duties, we created an iPad ® app to supplement their orientation ( Figure B1). Studies have shown technologically-savvy residents benefit from apps in medical education. [6][7][8][9] Although research in the area of technology and orientation is nascent, it has been demonstrated that learners perceive the integration of mobile technology will potentially improve their knowledge acquisition, facilitate collaborations, and enhance their efficiency, 10,11 However some authors have warned of the distraction and privacy issues. 12 In this paper we report the usage trends for this app, including anticipated versus actual app utilization. We will also discuss the impact of this study on future orientation app development, and for medical education in general.

Methods
Ethics approval was obtained from the Ottawa Health Science Network Research Ethics Board on April 14, 2015 (Protocol #20150130-01H).

App development
Local content experts used input from the preexisting paper-based orientation binder as well as from members in the training committee and the internal medicine residency program director to determine content (Appendix A) included in the app. They targeted the app towards Year 1 residents who were rotating through the CTU for the first time. A pre-usage survey was built into the app to collect basic demographics and identify baseline familiarity with mobile technology in medical education (see Appendix B). Once testing was complete, the app was made available as a free download in iTunes ® . The app requires an internet connection to refresh its content and the hospital has campus-wide Wi-Fi access. The Internal Medicine Training Committee and the Division of General Internal Medicine jointly funded the app development.

Research participants
The postgraduate medical program is organized into 13 rotations. Thus, rotation 1 represents the beginning and rotation 13 represents the end of the academic year. Participants were University of Ottawa PGY1 residents regardless of training program scheduled on CTU between rotation 12 to rotation 4 (6 clinical rotations).

Usage data
The software developer remitted the participants' anonymized usage data for the date range requested to be analyzed.

Survey results and demographics
Sixty-seven residents consented to participate within the study period, but we were only able to collect 35 pre-usage surveys. Overall, 63% of research participants were female, and 84% of users were between 25 to 29 years of age.
Of the 35 users surveyed, 33/35 (94%) reported being comfortable to very comfortable with using mobile technology, and 74% felt the app could improve the orientation experience. Participants anticipated that the app could help them with the admission algorithm, formulary, clinical guidelines, consultation rule and procedures, and call schedule, but there was no consensus on the section of the app they anticipated being the most commonly used. e113

Usage data
Microsoft Excel® was used for all descriptive statistics. The app was accessed 909 times by 41 participants with an average length of 78 seconds (Table 1: Analysis of Most Used Page.) However, some pages were used for a long time, but not frequently, while some pages were accessed frequently but only for a short time. To get a more accurate measure of usage, we created a "Popularity Score" defined as "minutes x frequency." The top four topics by Popularity Score were: 1. "The Consult Note" (the necessary parts of creating a consult note while on call)

"Admission Orders"
3. "Teaching Sessions and Schedules" (teaching sessions for the day) 4. "Junior Residents and Medical Students (on Call)" (responsibilities of a resident or medical student on call, including rounding at night) We also grouped certain topics into themes based on conceptually similar topics (Table 2); the most common themes were: 1. "On call" 2. "Daily expectations of junior house staff" 3. "Common forms"

Usage by time of day
We divided usage times into three sections based on the typical working day of our CTU rotation. In our academic CTU, the PGY1 residents are on-call in shifts, with the first senior resident working from 8am to 5pm. The next shifts are 5pm-12am and 12am-8am. The app was most used from 12am to 8am (46%), followed by 8am to 5pm (30%), and least used from 5pm to 12am (24%).

Usage by resident program
Only 8/35 participants identified themselves as internal medicine residents and 18/35 identified themselves as off-service residents; however, the app was utilized more frequently by internal medicine residents, as shown in Table 3

Usage by Rotation
The study period spanned the end of one academic year (Rotation 12 & 13) and the beginning of another year (Rotation 1 through 4). Usage increased from rotation 12 until it peaked at rotation 1, thereafter dropping off dramatically (see Table 4).

Usage by comfort with technology
We found an inverse relationship between residents' comfort level with mobile technology utilization and actual usage (Figure 1).

Discussion
Our study looked at the utility of an iPad as a supplement to the traditional orientation during an internal medicine rotation at an academic centre. Despite technical challenges, our results showed that this app was well used and could provide a useful supplement to traditional orientation. As we expected, the CTU Handbook App was used most often at the beginning of the academic year (July), when the first cohort of PGY-1 residents were just coming onto CTU for the first time. This suggests that orientation needs do not remain static over the course of the year. Also, as expected, the app was used most frequently by residents overnight when supportive resources were more scarce and work load may be the highest. The most frequently used sections ("on-call duties," "admissions and discharge protocols," "the consult note," and "common forms") suggest that it is during on-call hours that residents need the most support. More research is needed in this area.
We had anticipated that certain sections, such as the On-Call Schedules (Popularity Score 889.2), Medical Reconciliation (Popularity Score 169.2), and Drug Formulary (Popularity Score 652.8) would be most frequently accessed; this was not the case, and there could be several reasons. Firstly, the on-call schedule was also available through emails, making the content redundant. Secondly, while the Medical reconciliation was a relatively new and complicated process for junior residents, the instructions to use it was not frequently accessed in the app. Awareness may be insufficient, and residents would favour trial and error rather than reading lengthy instructions. Finally, the drug formulary was also available within the hospital intranet, and it was a rather large pdf document to load on the app. We felt these challenges highlighted that, sometimes, even medical education experts can't reliably anticipate what residents might need while on call. The more prudent way to proceed before the app development process may have been a formal needs assessment with the users. 12 This also highlights the need for ongoing post-app usage and feedback tracking, to allow for continuous improvement of the app after its initial rollout. It is essential to capture patterns of app usage to improve appropriate delivery of information on mobile app platforms. 13 Interestingly, we found that residents who were most comfortable with technology tended to spend less time on the app, and those who reported that they were less comfortable spent less time on the app (Figure 1). We speculate that this is because technology-savvy residents need less time to navigate, but it may also be that they scan information more and read less. 14 They may also have the ability to navigate other overlapping hospital resources to retrieve information quickly.
e115 This is an area where further research could provide insight.

Challenges and lessons learned
One of the main challenges was the availability of technical support. Many residents experienced login and registration difficulties, which required quick resolution with technicians. This may have led to some user dissatisfaction or early disengagement for some users. Additionally, necessary content updates were frequent after the app launch, making timely technical support necessary to keep information current.
Data collection was also a challenge. Although the pre-usage survey was mandatory prior to the app usage, we only received 35/41 surveys. Part of this could be explained by the fact that not all of the participants who actually consented used the app. Although we received usage data from 41 users, some participants were able to bypass the survey. After extensive investigations we could not determine how users could have bypassed the survey.
Users also did not provide detailed comments in the short answer sections of the survey, and we feel that better question design may have helped increase response. We had planned post-usage focus groups, however due to difficulties in recruiting residents after their rotations, this was not completed.
Our usage data tracking methods were not granular enough to yield usage data with sufficient precision to support more detailed analysis. For instance, if a user stayed on one page while leaving the iPad idle, the data will over-emphasize the usage of that particular page. To overcome this issue, we suggest that tracking actual user interaction may be superior to tracking simple time per page. Additionally, researchers can incorporate "search tracking" that records what information users were looking for.

Conclusion
The process of designing, building, and implementing the CTU Orientation Handbook iPad® app is feasible, practical, and may be a useful supplement to the traditional orientation.
When developing apps for medical education, developers and content experts are not always able to predict what features or contents will be the most useful, partly due to lack of understanding and evidence. Currently available tools only demonstrate moderate ability for predicting usage patterns of medical education apps. 15 Therefore, formal needs assessments and subsequent content updates following release, are likely to result in a more useful and sustainable app. Lastly, it may be helpful to use additional mechanisms to understand the needs from users, such as usage and search tracking for ongoing app improvement.
Conflicts of interest: No author associated with this manuscript has any conflict of interest that existseither due to a financial or a personal relationship that could potentially bias their work.  Figure B1. Screenshot of CTU Handbook App