Job Advertisement Sampling Strategy

We identified tenure-track university positions in the United States from postings listed on The Chronicle of Higher Education website as well as discipline-specific websites (e.g., American Historical Association for History) (see Appendix A). Selection criteria included positions for tenure-track, assistant professors, or open-rank positions inclusive of the assistant professor level, excluding those offered at two-year colleges, community colleges, and military academies. We identified job postings from nine academic disciplines—biology, business, chemistry, computer science, engineering (civil & environmental and mechanical), English, history, political science, and psychology—during a 4-week period (September 17–October 12, 2018). Within each discipline, sampling continued until we collected 200 postings or the 4-week sampling window closed. If a single department advertised multiple positions, we included each position in our sample. Overall, we selected 1,438 job postings for the study.

Survey Method

We used these job postings solely to identify departmental points of contact who would later receive a survey about how the hiring committee evaluated teaching effectiveness. To identify hiring chairs, we first located and emailed the department chairs corresponding to the 1,438 job postings and requested that they either fill out the survey, if they had served as the hiring chair, or provide the contact information for the hiring chair(s). When a department advertised for multiple positions, we sent surveys to as many hiring chairs as were provided by the department chair. Overall, the 1,438 job postings resulted in 1,130 online surveys distributed. We intended to use “faculty position” as our unit of analysis. However, because departments may have used one search committee to hire multiple faculty, conservatively, our unit of analysis is “hiring chair.”

Survey Response Sample

One hundred sixty-six individuals responded to the online survey for a response rate of approximately 15%. We categorized each survey response according to academic discipline and the Carnegie Classification (Table 1) provided in the fall 2019 Carnegie report (Center for Postsecondary Research, 2018). Due to small response rates for specific disciplines, we aggregated the nine disciplines into three categories based on academic divisions: Humanities (English and history), STEM (chemistry, biology, computer science, and engineering), and Social Sciences (psychology, political science, and business). Our disciplinary groupings by academic division are consistent with Meizlish and Kaplan’s 2008 study. Hereafter, we use the term academic division to refer to the three broad disciplinary categories articulated above. Table 1 classifies our job posting sample and our survey response sample according to institution type and academic division. Our survey respondents overrepresent U.S. Doctoral institutions relative to other institution types but not in terms of the number of annual hires per institution type. Doctoral institutions tend to be larger schools that hire more tenure-track instructors than smaller colleges and universities. Additionally, our sample is consistent with the distribution of Doctoral institutions in Meizlish and Kaplan’s 2008 study. Table 1 also shows that our survey responses are roughly representative of the postings we pulled. Therefore, while our response rate is relatively low, the distribution of academic divisions and Carnegie Classifications reflects our original sample of job postings.

Instrument

Our survey assessed how hiring chairs evaluate teaching effectiveness in tenure-track hiring at academic institutions across the United States. The 11-question survey consisted of Likert scale, closed-ended, and open-ended questions. Several survey questions were adapted from Meizlish and Kaplan’s 2008 study. A subset of the questions focused on three specific documents: the teaching philosophy statement, the diversity and inclusion statement, and student evaluations of teaching. We focused on these three documents because CTLs frequently consult on or are asked about them. Additionally, we sought to update data on teaching philosophy statements and student evaluations of teaching from previous studies (Bruff, 2007; Meizlish & Kaplan, 2008) and to begin exploring the use of diversity and inclusion statements in tenure-track hiring.

Data Analysis

For survey questions that yielded quantitative data, we computed descriptive statistics. In order to determine whether the importance of teaching in tenure-track hiring decisions varied by academic division and/ or institution type, we conducted a two-way ANOVA. With “importance of teaching” as our dependent variable, academic division and institution type functioned as fixed factors with three levels each. Additionally, the interaction term of academic division and institution type was included in the model. We tested model assumptions by examining boxplots of the residuals to identify outliers, using the Shapiro-Wilk test for normality and Levene’s test for homogeneity of variance. We identified several potential outliers, which remained in the data set. The assumption of homogeneity of variances held. However, the data violated the assumption of normality; the residuals were not normally distributed , even after a log transformation. Despite non-normality, we proceeded with the two-way ANOVA because this test is robust to violations of normality.

For all open-ended questions, we used the following approach to qualitatively analyze the data. Initially, one researcher holistically read through a subset of the responses and simultaneously created a codebook of emergent themes. Then two researchers each pilot-tested the initial codebook with another subset of the data. Next, pilot-testers discussed the codebook themes and definitions and arrived at consensus on a revised version. The two researchers then independently used the revised codebook to code the full data set $\left(n=166\right)$. The coders met to compare codes and discuss all discrepancies until they reached consensus for each response. Due to the open-ended nature of the questions, many of the responses received multiple codes. Thus, raters ensured consensus across all codes for each response. Throughout the entire process, we refined and clarified codebook definitions.

Research Question 1: How important is teaching in decisions about academic hiring? And does it vary by academic division and/or institution type?

On a 5-point Likert scale, teaching effectiveness received an overall average rating of 4.4, the highest average score of all the factors we asked about (publication record, external funding, service record) (see Table 2). Publication record received the second highest average rating overall (4.3). These two factors received conspicuously higher average ratings than external funding (2.7) and service record (2.6). Similarly, when we disaggregated our sample by institution type and academic division, teaching effectiveness and publication record consistently rated higher than external funding and service record.

Additionally, the two-way ANOVA¹ indicated that the perceived importance of teaching effectiveness differed significantly by institution type but not academic division. Specifically, the interaction between institution type and academic division was not statistically significant (i.e., any differences across institution types did not depend on academic division, and vice versa). Additionally, the main effect of academic division was not statistically significant (i.e., the importance of teaching effectiveness did not differ among STEM, Social Sciences, and Humanities). However, the main effect of institution type was statistically significant, $F\left(2,153\right)=14.3$, , ${\eta }_{\text{p}}^2=.16$, indicating a difference in the perceived importance of teaching effectiveness between at least two institution types. These results suggest that compared to hiring chairs at Baccalaureate and Master’s institutions, those at Doctoral institutions tend toward statistically lower ratings of the importance of teaching during the hiring process.

Research Question 2: What evidence do hiring committees use to assess teaching effectiveness? Does use vary by academic division and/or institution type? When they specifically request teaching philosophy statements, diversity and inclusion statements, and student evaluations of teaching, what is their rationale for doing so?

Our survey presented hiring chairs with a list of potential interactions and documents and asked respondents to identify all of the ones that they used for evaluating teaching effectiveness (Table 3). When examining the interactions, almost all hiring chairs (approximately 96%) reported using interview questions to ascertain an applicant’s teaching effectiveness. Nearly 83% of hiring chairs reported that research job talks were an important venue for evaluating teaching skills, and this varied little by institutional type. To observe teaching directly, roughly half of hiring chairs asked applicants to deliver a sample lesson (e.g., in a live course or for an ad hoc audience of faculty and/or students). Approximately two-thirds of hiring chairs also reported using meetings between applicants and students to diagnose teaching effectiveness. Teaching talks (i.e., presentations about how one would teach and why) occurred comparatively rarely.

When asked to identify all of the documents that hiring chairs used to evaluate teaching effectiveness, hiring chairs most commonly used teaching philosophy statements (approximately85%); faculty letters of recommendation (approximately 83%); and student evaluations of teaching (almost 48%) (see Table 3). Fewer than half of the respondents reported using each of the other sources of evidence mentioned in the survey, including diversity and inclusion statements.

Table 3 provides the number of hiring chairs who reported using various hiring documents as well as the number of hiring chairs that specifically requested teaching philosophy statements, diversity and inclusion statements, and student evaluations of teaching. Slightly fewer hiring chairs specifically requested each of these three documents than reported using each type of evidence. All further survey analysis regarding the use of hiring documents draws from the number of hiring chairs who specifically requested those documents.

Research Question 3: In using teaching philosophy statements, diversity and inclusion statements, and student evaluations of teaching to evaluate teaching effectiveness, what features are important? What makes such documents effective from a hiring committee perspective?

SoED Researchers

Our study provides empirical data regarding the role of teaching effectiveness and how it is evaluated in tenure-track hiring. Our results replicate those of previous studies across a broader range of disciplines and provide updated data. Specifically, our findings validate and update existing literature, showing the following:

1. In previous studies (Bruff, 2007; Meizlish & Kaplan, 2008), hiring chairs identified teaching and publication/research as the top two factors they considered, and our findings are consistent with these earlier results. After 10 years, teaching remains “very” or “extremely important” in tenure-track hiring, regardless of institution type or academic division. And while Doctoral institutions perceived teaching as less important in hiring than did Baccalaureate and Master’s institutions, STEM, Humanities, and Social Sciences disciplines did not differ significantly.²

2. Hiring committees use a variety of sources to evaluate teaching effectiveness, and the teaching philosophy statement is one of the most commonly used documents (Bruff, 2007; Meizlish & Kaplan, 2008). In our study, approximately 85% of hiring chairs reported using this document (Table 3) to evaluate applicants, which is considerably higher than the approximately 57% reported by Meizlish and Kaplan (2008).

3. Hiring committees continue to use student evaluations of teaching as a common data source to evaluate teaching effectiveness. In both our study and Meizlish and Kaplan (2008), approximately half of the hiring chairs (48% and 54%, respectively) reported using student evaluations of teaching. In Bruff (2007), approximately 75% of hiring chairs reported using student evaluations.

4. Hiring committees now somewhat more frequently ask applicants to teach a sample lesson: approximately 51% in our study compared to 41% in Meizlish and Kaplan (2008) and 35% to 45% in Bruff (2007).

Additionally, previous research (Meizlish & Kaplan, 2008) found evidence that hiring committees used an applicant’s research talk as a proxy for teaching effectiveness, and Bruff (2007) reported that 76% of Mathematics hiring chairs use this practice. Our study found that this practice continues and is slightly more common. Across all of our academic divisions, approximately 83% of hiring chairs reported using an applicant’s job talk as a measure of teaching effectiveness. The prevalence of this practice raises some questions, given that these talks tend to be designed for a more expert audience (e.g., other faculty members) rather than novice students. Additionally, while a research talk can demonstrate transferable lecture-based teaching skills (e.g., organizing content, implementing multimedia learning principles, responding to questions), these talks traditionally lack demonstration of evidence-based teaching strategies, such as active learning (e.g., Freeman et al., 2014; Prince, 2004), that may be desirable for evaluating teaching effectiveness. Additional research should be done to examine the rationale for this practice as a proxy for teaching as well as the criteria hiring committees use to evaluate a research talk as a measure of teaching effectiveness.

Our results indicate that approximately 34% of hiring chairs requested and approximately 48% used student evaluations of teaching, if provided. Additionally, our data show that hiring chairs have different views about the use of these documents to measure teaching effectiveness. These differing opinions may represent the disparity between what peer-reviewed research says about student evaluations of teaching and how that research gets (mis)interpreted in the popular press. Barre (2015) draws attention to this problem while also summarizing the research, stating that “student evaluations of teaching are a useful, but ultimately imperfect, measure of teaching effectiveness.” Additionally, Barre notes that scholars in this field recommend that other measures of teaching effectiveness be used in conjunction with student evaluations. As such, further research is needed to provide hiring chairs with recommendations on how to assess this data source that will then provide more guidance to applicants on how to present student evaluations in their application materials.

Previous research has not examined how hiring committees use diversity and inclusion statements to evaluate teaching effectiveness, and our study serves as an important first step in answering this question. Our results show that approximately 38% of hiring chairs use a diversity and inclusion statement to assess teaching effectiveness. Additionally, hiring chairs value an applicant’s understanding of diversity and inclusion issues in higher education as well as how diversity and inclusion is centered in one’s teaching, mentorship, and service. Our data raise some questions about the rationale for requesting this document. For example, approximately 25% of the hiring chairs who requested a diversity and inclusion statement expected it to contain information about the applicant’s identity (e.g., race, gender, ethnicity). The variety of responses and their potential implications point to the lack of shared understanding regarding the content of diversity and inclusion statements and underscore the need for hiring committees to explain their rationale for requesting these documents, as similarly found in Flaherty (2018). Approximately 25% of the hiring chairs who did not request a diversity and inclusion statement indicated that they had interest in requesting this document in the future. This finding, coupled with the events of 2020 (e.g., the murder of George Floyd, renewed emphasis on the Black Lives Matter movement), suggest that diversity and inclusion statements will be requested more often. We hope future research will continue to address the use and interpretation of this relatively new genre of hiring document.

In addition to the research directions noted above, we hope future research may address several of the limitations of our study. First, our study does not include any disciplines from the Fine Arts (e.g., music, drama, art, design) or health science professional schools (e.g., nursing, medicine, pharmacy, dentistry). These disciplines have specific pedagogies (e.g., individual, studio-based teaching in fine arts and clinical instruction for health sciences) that may result in different measures of teaching effectiveness than the ones described in our study. Second, our data were collected during the 2018–2019 academic year; therefore, it cannot speak to the impacts of the ubiquitous transition to remote and/or hybrid teaching that higher education experienced in response to the COVID-19 pandemic. Future studies can and should examine how (if at all) this shift in teaching modalities impacts how teaching effectiveness is evaluated in tenure-track hiring. For example, hiring committees may increasingly prioritize teaching effectiveness in remote or hybrid teaching formats, and evaluation criteria and data sources may change accordingly. Third, our overall response rate was lower than we would have liked. Nevertheless, our sample provides nationwide data with representation across STEM, Humanities, and the Social Sciences as well as institution types (Carnegie Classifications) in proportion to tenure-track job ads posted during our sampling period.

CTL Leadership

Our results hold implications for CTL leadership in terms of strategic planning and resource allocation. At the broadest level, hiring chairs tended to rate teaching effectiveness as very important in tenure-track hiring. Therefore, we recommend that CTLs who prioritize preparing graduate students and postdocs for the academic job market, and who have available resources and staff, develop and sustain programs and services to help these clients demonstrate their teaching effectiveness. When resources are scarce, CTL leadership may need to prioritize certain programs and services over others. For example, given that hiring committees both requested and used statements of teaching philosophy more often than any other hiring document (Table 3), CTLs might wish to prioritize workshops and individual consultations on these statements.

Our data suggest that CTLs should holistically consider how best to approach diversity, equity, and inclusion (DEI) in their offerings for future faculty. In terms of an applicant’s understanding of and demonstrated commitment to DEI, our findings suggest that hiring committees may look for this information across multiple data sources. For example, as shown in Table 4, hiring committees expect applicants to address inclusive teaching practices in their teaching philosophy statement. Additionally, we found that almost 35% of hiring committees specifically request a diversity and inclusion statement and use this statement to evaluate teaching effectiveness. However, as Table 6 shows, the content of diversity and inclusion statements does not pertain solely to teaching, and CTLs may not be equipped to fully address these other aspects of the statement. At minimum, CTL programs and services can make job applicants aware that they may be asked to demonstrate their perspectives and practices regarding DEI and alert them to the various ways they may be asked to do this on the academic job market (e.g., teaching philosophy statement, diversity and inclusion statement). CTLs can also adjust their offerings to help graduate students and postdocs develop their knowledge, practices, and capacity for enhancing DEI as it relates to their teaching. Such efforts might take the form of seminars or workshops explicitly targeting inclusive teaching, creating accessible instructional materials, or other DEI topics. In parallel, CTLs can endeavor to infuse a DEI lens into programs and services that do not overtly focus on DEI (e.g., syllabus consultations, seminars on designing and managing group work or effective grading and feedback practices). Additionally, CTLs could consider offering specific programming and services for diversity and inclusion statements, staff and resources permitting. Given that diversity statements encompass other aspects of future faculty life aside from teaching, CTLs leadership may benefit from collaborating with other campus units to support job applicants’ development of diversity and inclusion statements.

Our data suggest that providing opportunities for clients to reflect on and contextualize their student evaluations can best support future faculty. For example, CTLs could offer individual consultations that allow graduate students and postdocs to reflect on their evaluations of teaching and identify areas of strength as well as those for growth and development. Job applicants could then draw on this information when writing their teaching philosophy statement or considering what other documents to submit to a hiring committee (e.g., sample lesson plan, sample rubric). Such consultations would benefit job applicants, regardless of whether they actually submit their teaching evaluations, by allowing them an opportunity to reflect on and speak about their teaching practice.

In addition to the three hiring documents described above, our results show that hiring committees commonly assess teaching effectiveness in a variety of other ways. For instance, the practice of having a candidate teach a sample lesson has become more common, and this in an area where CTLs could support job applicants by helping them develop this type of lesson plan or provide opportunities to practice and receive feedback on the implementation of evidence-based teaching techniques in their sample lessons. Similarly, the practice of using a job applicant’s research talk as a proxy for teaching practice is quite prevalent (approximately 83% of hiring chairs reported doing this), but it’s less clear what type of support CTLs might provide here given the questions raised about this practice in the previous section.

While it is important to educate future faculty about the variety of ways in which they may need to demonstrate their teaching effectiveness, it may not be necessary for CTLs to develop services and programs to support all of these measures. Helping job applications prepare for one deliverable or interaction will likely assist with them with the preparation for others. For example, writing a teaching philosophy statement requires an applicant to identify and reflect on the types of teaching experiences that they have had or would like to have. These experiences might be concisely captured in a teaching philosophy statement, and an applicant might further elaborate on them during an interview. Similarly, an applicant might demonstrate a particular teaching method that they describe in their teaching philosophy statement during their sample teaching session.

Educational Developers

Our findings directly inform the work of educational developers who support graduate students and postdocs in demonstrating their teaching effectiveness, especially during one-on-one consultations or during workshops. In this section, we focus on how our data can inform one’s consulting toolkit, recognizing that these practices also translate to the design and delivery of seminars and workshops.

Overall, our findings highlight the importance of teaching in tenure-track hiring, and educational developers can leverage this point to validate the significant effort that job applicants put into demonstrating their teaching effectiveness. At our CTLs, educational developers frequently provide feedback to clients on hiring documents and may encounter resistance when they suggest clients iterate on a draft or create multiple versions tailored to different job offers. In particular, our data suggest that when hiring committees ask for specific documents, namely a teaching philosophy statement and a diversity and inclusion statement, they are doing so to understand how the applicant’s views on these subjects align with the mission and values of the institution. Consequently, educational developers might encourage job applicants to research the institutions/departments to which they are applying in order to tailor their hiring documents for different contexts. If hiring committees do not request specific documents, then educational developers can help job applicants consider the breadth of sources they can draw from to demonstrate their teaching effectiveness.

Educational developers can use our results to inform feedback or suggestions on the content of teaching philosophy statements. Hiring chairs appeared to have consensus, across institution types and academic divisions, about their rationale for requesting teaching philosophy statements as well as the characteristics of effective statements (Tables 4 and 5). While these statements are highly personal and individualized, our findings present some broad categories of information that should be present. For example, hiring chairs are looking for concrete, discipline-specific examples of an applicant’s teaching methods and learning objectives. Previous research (Bruff, 2007; Meizlish & Kaplan, 2008) reported similar findings, suggesting that hiring chairs have a stable and consistent view of what they are looking for in these documents. Approximately 30% of our respondents indicated that these concrete examples should be accompanied by reflection on one’s teaching, including a discussion of how one could improve as an educator. Bruff (2007) reported a similar finding, with 28% of hiring chairs indicating that reflection was a characteristic of an effective teaching statement. Hiring chairs in our study as well as the two previous ones (Bruff, 2007; Meizlish & Kaplan, 2008) recommended that applicants be specific with the content of their teaching philosophy statements and avoid the use of pedagogical buzzwords without accompanying descriptions of examples. When providing feedback to job applicants on their teaching philosophy statements, educational developers should look for this content and prioritize their feedback accordingly, perhaps through the use of rubrics or general guidance about writing teaching philosophy statements.

Educational developers may provide a variety of services to help graduate students and postdocs develop an understanding of diversity and inclusion with regard to their teaching. Based on our data, these offerings might focus on communicating one’s understanding, values, and/or current and future practices and engagement regarding enhancing DEI in higher education, inclusive teaching and mentorship, and service activities regarding DEI. For those who specifically help job applicants write diversity and inclusion statements, our results are aligned with the recommendations from NCID. NCID director Tabbye Chavous described diversity and inclusion statements: “Generally speaking, these statements should articulate the way a candidate understands DEI and show the candidate’s DEI skills and competencies” (as cited in Smith, 2019). As shown in Table 6, these skills and competencies do not pertain solely to teaching and need not include information about an applicant’s identity. Conveying this information to job applicants will ensure that they address these various aspects of DEI work and may help to clarify concerns about divulging protected identities. Chavous recommends that institutions be explicit about their rationale for requesting these statements (Flaherty, 2018), and educational developers can encourage job applicants to look for these rationales in the job requests.

Educational developers are frequently asked to provide guidance for how to present student evaluations of teaching to hiring committees. At a minimum, educational developers can make job applicants aware of the differing viewpoints on student evaluations expressed by hiring chairs. Given that fewer hiring chairs specifically requested student evaluations of teaching (approximately 34% requested compared to approximately 48% who used them), job applicants may want to consider whether to submit these documents if not required to do so. Educational developers can help job applicants to consider what other measures they might draw from and how best to package this information alongside their teaching evaluations (Linse, 2017). If an applicant submits their evaluations, regardless of whether they were asked to, educational developers should encourage applicants to provide summative data, interpretations or contextualization of the data, or additional descriptions about their teaching context to help the hiring committee interpret the evaluations. Additionally, educational developers should encourage job applicants to submit all of their teaching evaluations—the good and the bad—and use the negative ones to discuss opportunities for growth and improvement in their teaching practice.

Taken together, our results show that teaching effectiveness continues to be a top factor in tenure-track hiring and that job applicants are asked to demonstrate their teaching effectiveness in a wide variety of ways. Some “staples” of the hiring process remain (e.g., teaching philosophy statement), and new ones have been added (e.g., diversity and inclusion statement). CTLs and educational developers need updated data, representing a variety of academic disciplines and institution types, to inform the programs and services that they provide to job applicants. We are excited to provide this data, and we look forward to what future SoED researchers contribute to this ever-changing landscape.