Evaluation of a Virtual Sub-internship Experience for Senior Medical Students Applying to Otolaryngology During the COVID-19 Pandemic

Introduction

For fourth-year medical students, the COVID-19 pandemic limited learning experiences, altered professional development opportunities, introduced virtual interviews, and cancelled the “away rotation” or visiting sub-internship.^1,2 Medical educators had already been scrutinizing the final year of medical school due to learning inefficiencies and concerns for residency unpreparedness.³ In fact, some had encouraged the reformation of visiting sub-internships to provide more educational benefit based on the investment they require.⁴ Given safety concerns during the COVID-19 pandemic, the Coalition for Physician Accountability⁵ and other national organizations⁶ issued recommendations to cancel away rotations for the 2020–2021 residency application cycle. Thus, the COVID-19 pandemic offered an opportunity to reimagine how clinical departments could remotely meet the goals of the traditional visiting sub-internships: advanced medical student clinical training, mentorship, career exposure, and an introduction to residency life. We describe herein how we created a virtual space to meet some objectives of the traditional visiting sub-internship for Otolaryngology-Head and Neck Surgery (OHNS).

Methods

We first informally interviewed University of Michigan third-year medical students applying into OHNS, residents, and program leadership about the objectives of traditional visiting sub-internships. Based on these results, we designed a free, one-day virtual event to meet three objectives: (1) exposure to the breadth and depth of OHNS sub-specialties, (2) introduction to the University of Michigan OHNS department and residency program, and (3) compilation of resident-recommended didactic content for each OHNS sub-specialty. The event was held on June 13, 2020, and featured sessions focused on OHNS sub-specialties as well as specific topics such as scholarship during residency and diversity, equity, and inclusion (Table 1).

Invited faculty were asked to share aspects of their daily practice and insights on how they integrate leadership, educational philosophy, and scholarly activity into their careers. Similarly, resident presenters were chosen to represent junior- and senior-level trainees who were involved in a wide range of pursuits both inside and outside of the residency program. Each session was structured so that a resident moderator would introduce and then interview the presenters. Question and answer sessions were held in real time. All participants were sent guidelines for the event, presenter contact information, and electronic resources and book recommendations curated by University of Michigan OHNS residents.

Advertisement for the program took place through OHNS medical student forums, institutional diversity champions, social media, and word of mouth. The event was hosted as a Zoom webinar with two residents as moderators. Following the program, surveys were distributed to all participants and panelists through Qualtrics. Using 5-point Likert scale questions, participants were asked about the helpfulness of the program for learning about the scope of the field, University of Michigan residency, and OHNS electronic and book resources, as well as the program length and likelihood of recommending the program to other students.

Participant demographics and duration of attendance data were collected. Descriptive statistics were calculated for Likert scale questions on Qualtrics software. Two of the authors (JLF and EJM) created, combined, and reconciled themes for free response questions. The University of Michigan Institutional Review Board deemed this study exempt (HUM00182084).

Results

There were 448 registrants and 327 final participants, representing 138 institutions, 34 states and territories, and 10 countries. The majority (90%) of participants were senior medical students (fourth year only), with fewer junior medical students (6%), residents (3%), and participants with unknown training status (1%). The event had a total of 5 content hours, and participants on average were signed into the event for 4.1 hours (standard deviation 2.2).

There were 102 respondents to the participant survey (31% response rate). Of respondents, 99% stated that they would be very likely (79%) or somewhat likely (20%) to recommend the program to another student. Approximately half of participants felt the event was an appropriate length (n = 48, 47%) or a little too long (n = 46, 46%). The majority of participants found the event helpful for learning about the scope of the field (96%), the residency program (97%), and specialty-specific resources (86%) (Table 2). Themes extracted from free responses showed that participants found hearing from residents (n = 44, 43%) and faculty (n = 42, 41%) as the most helpful aspects of the program. A smaller number of responses noted description of program opportunities (n = 25, 25%), tips for the application process (n = 19, 19%), and live question and answer (n = 11, 11%) as the most helpful parts of the event. Suggestions for event improvement included more resident interaction (n = 15, 19%), a shorter event or more spread out programming (n = 13, 17%), and integrating more information regarding the application process (n = 12, 16%).

Participants stated that their biggest concern for the upcoming application cycle was getting to know programs in a remote application cycle (n = 25, 28%), interviews in a virtual setting (n = 23, 26%), the lack of away rotations (n = 18, 20%), and their individual competitiveness (n = 14, 16%), including students who felt their educational and career development opportunities had been limited by the pandemic (n = 7, 8%). Students also noted potential disadvantages for special applicant populations (n = 13, 15%), including international medical students, osteopathic students, students participating in the couples match, and students from institutions without OHNS departments.

As further described through free response, participants thought the most difficult aspect of a residency program to evaluate remotely was the program’s or institution’s culture (n = 65, 76%), followed by the program’s city (n = 22, 26%) or facilities (n = 16, 19%). Students were also concerned about the ability to assess their personal fit in a program (n = 16, 19%) and difficulty getting to know residents remotely (n = 11, 13%).

Discussion

Visiting medical student sub-internships, despite their cost and time investment, are a common and potentially vital portion of the transition to residency in OHNS and other specialties.⁴ With the notable limitation of visiting sub-internships during the COVID-19 pandemic, the University of Michigan OHNS department created a one-day virtual event to address some of the barriers to educational experiences.

We found that this event was well-received by participants and engaged a broad international audience of 327 participants in a field that offers only 350 new residency spots through the National Resident Matching Program annually.⁷ While visiting sub-internships have resumed, events such as the one described herein may still have a place as an introduction to the field or as a complement to mentorship programs for premedical students, early medical students, international students, and those without home residency programs. Earlier exposure to various residency programs may also lead to more informed application and interview decision-making, which would be particularly valuable if virtual interviews continue.

Even though participants on average stayed signed-in for the majority of this event, the program was not designed to measure direct engagement. While the majority of participants stated that the event length was just right or just a little long, we suggest that future events be shorter or more spread out, as greater adoption of similar experiences may contribute to virtual burnout.^8,9 Additionally, at the time of the event, students were generally not involved in clinical activity given guidance from the Association of American Medical Colleges,¹⁰ but at the time of writing this report, clinical responsibilities for students had resumed. The length and timing of future events should be planned accordingly. Evaluation of this program was limited by a 31% survey response rate and resulting selection bias of respondents. Additionally, the survey was designed to assess participant perceptions of whether program objectives were met and not the actual knowledge gained.

This program did not allow for in-depth didactic learning or breakout room discussion, compared to other multiple-week virtual programs.^11–15 Nonetheless, the majority (82%) of participants identified resident interaction as the most valuable aspect of the program. Given the specific concern of evaluating a program’s culture remotely per 76% of survey respondents, future events should create interactive experiences between faculty, residents, and participants. At our institution, we have started an initiative that pairs University of Michigan residents and faculty with senior medical students from across the country to discuss transition to residency topics and high-yield case-based reviews. While this program facilitates social networking and small group interaction, it does not replace individualized mentorship, meaningful interaction leading to a letter of recommendation, or other aspects of in-person visiting sub-internships. We encourage the development and continued support of specialty-wide virtual offerings to enhance didactic learning,¹⁵ create a national network of faculty and resident mentors,¹⁶ and provide other asynchronous virtual career development opportunities for students to further explore the field. For instance, Headmirror (www.headmirror.com), a website designed to provide “a centralized high-quality open-access [OHNS] resource,” has recently established a National Otolaryngology Interest Group that may simultaneously engage a wider student community and supplement sub-internships.

The COVID-19 pandemic has afforded the opportunity to create innovative educational experiences, such as the one described herein. Similar offerings could enhance or replace the traditional components of the final year of medical school training, such as the visiting sub-internship.¹⁷ As we move beyond the COVID-19 pandemic, medical educators should continue to critically examine and reimagine the sub-internship experience as well as the entire transition to residency process.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author Roles

Janice L. Farlow: design, participation, data collection, analysis, presentation; Emily J. Marchiano: design, participation, data collection, analysis, presentation; Andrew J. Neevel: presentation, manuscript review; Jeffrey S. Moyer: participation, manuscript review; Marc C. Thorne: participation, manuscript review; Lauren A. Bohm: design, participation, presentation, manuscript review.