Centers for teaching and learning (CTLs) often support graduate students and postdocs preparing to become future faculty members. Through programs (e.g., workshops, seminars), services (e.g., consultations), and web resources, CTLs aim to help graduate students and postdocs succeed in teaching and to prepare for their faculty roles (Adams, 2002; Cook & Kaplan, 2011; Hershock, Groscurth, et al., 2011; Palmer & Little, 2013; Pruitt-Logan & Gaff, 2004; Rozaitis et al., 2020). As part of this process, CTLs often provide graduate students and postdocs with guidance and assistance about how to demonstrate their teaching effectiveness to tenure-track hiring committees. For example, CTLs may help job applicants develop a teaching philosophy statement (Kaplan et al., 2007; Kearns & Sullivan, 2010) or a sample course syllabus.
To effectively support tenure-track applicants, CTLs and educational developers must have data about how important teaching is to search committees, the types of teaching-related materials that search committees request, and how search committees evaluate these materials. This type of evidence-based approach (e.g., Beach et al., 2016; Hershock, Cook, et al., 2011) can be difficult because few empirical studies exist regarding how teaching effectiveness gets evaluated in tenure-track hiring. To help fill these gaps in the literature, advance data-informed educational development through action research (Hershock, Cook, et al., 2011; POD Network, 2018), and, most importantly, better support our CTL clients, we surveyed the hiring chairs for 166 university-level, tenure-track searches across nine academic disciplines. Our study, drawing from the methodology of Meizlish and Kaplan (2008), aims to supplement and update previous research regarding how teaching is evaluated in tenure-track hiring by (re)examining the following:
How important is teaching in decisions about tenure-track hiring? And does it vary by academic division and/or institution type?
What evidence do hiring committees use to assess teaching effectiveness? Does use vary by academic division and/or institution type? When they specifically request teaching philosophy statements, diversity and inclusion statements, and student evaluations of teaching, what is their rationale for doing so?
In using teaching philosophy statements, diversity and inclusion statements, and student evaluations of teaching to evaluate teaching effectiveness, what features are important? What makes such documents effective from a hiring committee perspective?
The most recent studies on tenure-track hiring are from the mid-2000s. Meizlish and Kaplan (2008) analyzed job postings and surveyed hiring chairs in six academic disciplines: English, history, political science, psychology, biology, and chemistry. Bruff (2007) used the Meizlish and Kaplan methodology and survey instrument to extend this work to mathematics. Generally speaking, these studies found that, across institutions and disciplines, teaching effectiveness is an important component in the tenure-track hiring process. Additionally, they found that, while hiring committees use various types of evidence to determine an applicant’s teaching effectiveness, some of the most prevalent ones include interview questions, faculty letters of recommendation, teaching philosophy statements, and student evaluations of teaching. This research resulted in valuable, evidence-based practices for CTLs and educational developers, specifically with regard to supporting job applicants in writing teaching philosophy statements. At the time, this research suggested that this genre of hiring document was one that many job applicants would encounter on the job market.
Our study builds on this foundation in several key ways. First, given the ever-changing nature of the academic job market, our study provides an updated picture of the role of teaching in tenure-track hiring and the types of evidence hiring committees use to evaluate it. Second, our study expands the number of academic disciplines surveyed in previous studies by adding engineering, computer science, and business. Third, with regard to student evaluations of teaching, we aimed to gather data about how frequently hiring committees request these documents, why they request them, and how job applicants can best present this information. Fourth, our study explores how hiring committees evaluate a new genre of hiring document—diversity and inclusion statements—in addition to some of the more traditionally requested teaching documents (e.g., teaching philosophy statements). Peer-reviewed scholarship on diversity and inclusion statements is limited (e.g., Sylvester et al., 2019), and we are not aware of any studies examining how hiring committees use and evaluate diversity statements. However, given the attention to them in the popular press (e.g., Canning & Reddick, 2019; Flaherty, 2018; Golash-Boza, 2016; Reyes, 2018), including a growing sense that they are commonly requested on the academic job market (e.g., Smith, 2019; Whitaker, 2020; Willis, 2017), we wanted to gather data about how frequently these statements are used to evaluate teaching effectiveness and why they are requested to evaluate candidates, in general. The results of our study have implications for different, yet often overlapping audiences, namely scholarship of educational development (SoED) researchers, CTL leadership, and educational developers. After presenting our study and its results, we focus specifically on these three audiences to provide practical, data-driven advice for each.
Job Advertisement Sampling Strategy
We identified tenure-track university positions in the United States from postings listed on The Chronicle of Higher Education website as well as discipline-specific websites (e.g., American Historical Association for History) (see Appendix A). Selection criteria included positions for tenure-track, assistant professors, or open-rank positions inclusive of the assistant professor level, excluding those offered at two-year colleges, community colleges, and military academies. We identified job postings from nine academic disciplines—biology, business, chemistry, computer science, engineering (civil & environmental and mechanical), English, history, political science, and psychology—during a 4-week period (September 17–October 12, 2018). Within each discipline, sampling continued until we collected 200 postings or the 4-week sampling window closed. If a single department advertised multiple positions, we included each position in our sample. Overall, we selected 1,438 job postings for the study.
We used these job postings solely to identify departmental points of contact who would later receive a survey about how the hiring committee evaluated teaching effectiveness. To identify hiring chairs, we first located and emailed the department chairs corresponding to the 1,438 job postings and requested that they either fill out the survey, if they had served as the hiring chair, or provide the contact information for the hiring chair(s). When a department advertised for multiple positions, we sent surveys to as many hiring chairs as were provided by the department chair. Overall, the 1,438 job postings resulted in 1,130 online surveys distributed. We intended to use “faculty position” as our unit of analysis. However, because departments may have used one search committee to hire multiple faculty, conservatively, our unit of analysis is “hiring chair.”
Survey Response Sample
One hundred sixty-six individuals responded to the online survey for a response rate of approximately 15%. We categorized each survey response according to academic discipline and the Carnegie Classification (Table 1) provided in the fall 2019 Carnegie report (Center for Postsecondary Research, 2018). Due to small response rates for specific disciplines, we aggregated the nine disciplines into three categories based on academic divisions: Humanities (English and history), STEM (chemistry, biology, computer science, and engineering), and Social Sciences (psychology, political science, and business). Our disciplinary groupings by academic division are consistent with Meizlish and Kaplan’s 2008 study. Hereafter, we use the term academic division to refer to the three broad disciplinary categories articulated above. Table 1 classifies our job posting sample and our survey response sample according to institution type and academic division. Our survey respondents overrepresent U.S. Doctoral institutions relative to other institution types but not in terms of the number of annual hires per institution type. Doctoral institutions tend to be larger schools that hire more tenure-track instructors than smaller colleges and universities. Additionally, our sample is consistent with the distribution of Doctoral institutions in Meizlish and Kaplan’s 2008 study. Table 1 also shows that our survey responses are roughly representative of the postings we pulled. Therefore, while our response rate is relatively low, the distribution of academic divisions and Carnegie Classifications reflects our original sample of job postings.
|Job posting positions pulled (%)||Survey responses received (%)|
|Baccalaureatea||191 (13.3)||38 (22.9)|
|Master’s||372 (25.9)||43 (25.9)|
|Doctoral||875 (60.8)||85 (51.2)|
|Humanities (English, history)||294 (20.4)||55 (33.1)|
|Social Sciences (business, political science, psychology)||419 (29.1)||50 (30.1)|
| STEM (biology, chemistry, engineering, and computer science)
||725 (50.4)||61 (36.8)
Note. This table does not include U.S. institutions that were excluded from our sampling process (e.g., two-year colleges, special focus colleges, community colleges, and military academies).
Baccalaureate colleges and Baccalaureate/Associate’s colleges.
Our survey assessed how hiring chairs evaluate teaching effectiveness in tenure-track hiring at academic institutions across the United States. The 11-question survey consisted of Likert scale, closed-ended, and open-ended questions. Several survey questions were adapted from Meizlish and Kaplan’s 2008 study. A subset of the questions focused on three specific documents: the teaching philosophy statement, the diversity and inclusion statement, and student evaluations of teaching. We focused on these three documents because CTLs frequently consult on or are asked about them. Additionally, we sought to update data on teaching philosophy statements and student evaluations of teaching from previous studies (Bruff, 2007; Meizlish & Kaplan, 2008) and to begin exploring the use of diversity and inclusion statements in tenure-track hiring.
For survey questions that yielded quantitative data, we computed descriptive statistics. In order to determine whether the importance of teaching in tenure-track hiring decisions varied by academic division and/ or institution type, we conducted a two-way ANOVA. With “importance of teaching” as our dependent variable, academic division and institution type functioned as fixed factors with three levels each. Additionally, the interaction term of academic division and institution type was included in the model. We tested model assumptions by examining boxplots of the residuals to identify outliers, using the Shapiro-Wilk test for normality and Levene’s test for homogeneity of variance. We identified several potential outliers, which remained in the data set. The assumption of homogeneity of variances held. However, the data violated the assumption of normality; the residuals were not normally distributed , even after a log transformation. Despite non-normality, we proceeded with the two-way ANOVA because this test is robust to violations of normality.
For all open-ended questions, we used the following approach to qualitatively analyze the data. Initially, one researcher holistically read through a subset of the responses and simultaneously created a codebook of emergent themes. Then two researchers each pilot-tested the initial codebook with another subset of the data. Next, pilot-testers discussed the codebook themes and definitions and arrived at consensus on a revised version. The two researchers then independently used the revised codebook to code the full data set . The coders met to compare codes and discuss all discrepancies until they reached consensus for each response. Due to the open-ended nature of the questions, many of the responses received multiple codes. Thus, raters ensured consensus across all codes for each response. Throughout the entire process, we refined and clarified codebook definitions.
Research Question 1: How important is teaching in decisions about academic hiring? And does it vary by academic division and/or institution type?
On a 5-point Likert scale, teaching effectiveness received an overall average rating of 4.4, the highest average score of all the factors we asked about (publication record, external funding, service record) (see Table 2). Publication record received the second highest average rating overall (4.3). These two factors received conspicuously higher average ratings than external funding (2.7) and service record (2.6). Similarly, when we disaggregated our sample by institution type and academic division, teaching effectiveness and publication record consistently rated higher than external funding and service record.
|Teaching effectiveness mean (SD)||Publication record mean (SD)||External funding mean (SD)||Service record mean (SD)|
|Baccalaureate||4.7 (.6)||3.8 (1.0)||1.9 (.8)||2.7 (.8)|
|Master’s||4.6 (.5)||4.0 (.9)||2.7 (1.0)||2.6 (.8)|
|Doctoral||4.1 (.7)||4.6 (.6)||3.2a (1.3)||2.5 (.8)|
|Humanities||4.5 (.6)||4.3 (.8)||2.2b (.9)||2.8 (.7)|
|Social Sciences||4.4 (.7)||4.2 (.9)||2.5 (1.2)||2.4 (.9)|
|STEM||4.3 (.7)||4.3 (.9)||3.4 (1.2)||2.5 (.7)|
|Overall||4.4 (.7)||4.3 (.9)||2.7c (1.2)||2.6 (.8)|
Note. Scale: 1 = not at all important; 2 = not so important; 3 = somewhat important; 4 = very important; 5 = extremely important.
Additionally, the two-way ANOVA1 indicated that the perceived importance of teaching effectiveness differed significantly by institution type but not academic division. Specifically, the interaction between institution type and academic division was not statistically significant (i.e., any differences across institution types did not depend on academic division, and vice versa). Additionally, the main effect of academic division was not statistically significant (i.e., the importance of teaching effectiveness did not differ among STEM, Social Sciences, and Humanities). However, the main effect of institution type was statistically significant, , , , indicating a difference in the perceived importance of teaching effectiveness between at least two institution types. These results suggest that compared to hiring chairs at Baccalaureate and Master’s institutions, those at Doctoral institutions tend toward statistically lower ratings of the importance of teaching during the hiring process.
Research Question 2: What evidence do hiring committees use to assess teaching effectiveness? Does use vary by academic division and/or institution type? When they specifically request teaching philosophy statements, diversity and inclusion statements, and student evaluations of teaching, what is their rationale for doing so?
Our survey presented hiring chairs with a list of potential interactions and documents and asked respondents to identify all of the ones that they used for evaluating teaching effectiveness (Table 3). When examining the interactions, almost all hiring chairs (approximately 96%) reported using interview questions to ascertain an applicant’s teaching effectiveness. Nearly 83% of hiring chairs reported that research job talks were an important venue for evaluating teaching skills, and this varied little by institutional type. To observe teaching directly, roughly half of hiring chairs asked applicants to deliver a sample lesson (e.g., in a live course or for an ad hoc audience of faculty and/or students). Approximately two-thirds of hiring chairs also reported using meetings between applicants and students to diagnose teaching effectiveness. Teaching talks (i.e., presentations about how one would teach and why) occurred comparatively rarely.
|Number of respondents using (%)||Number of respondents who specifically requested (%)|
|Interview questions||159 (95.8)||N/A|
|Research job talk||137 (82.5)||N/A|
|Student meetings||111 (66.9)||N/A|
|Teach a sample lesson||85 (51.2)||N/A|
|Teaching talk||13 (7.8)||N/A|
|Teaching philosophy statement||141 (84.9)||115 (73.2)b|
|Letters of recommendation from faculty||137 (82.5)||N/A|
|Student evaluations of teaching||79 (47.6)||53 (34.0)b|
|Course materials||64 (38.6)||N/A|
|Diversity and inclusion statement||63 (38.0)||54 (34.6)c|
|Teaching portfolio||34 (20.5)||N/A|
|Letters of recommendations from students||4 (2.4)||N/A|
a “Other” responses include interactions with faculty and administration as well as other versions of the traditional job talk.
d “Other” responses included documents such as a cover letter/application letter , CV or list of courses taught , or research statement . More unique responses included a search committee asking for a written discussion of how an assignment achieves pedagogical aims and peer/supervisor class observations and evaluations.
Note. Totals exceed 100% because participants could select multiple items.
When asked to identify all of the documents that hiring chairs used to evaluate teaching effectiveness, hiring chairs most commonly used teaching philosophy statements (approximately85%); faculty letters of recommendation (approximately 83%); and student evaluations of teaching (almost 48%) (see Table 3). Fewer than half of the respondents reported using each of the other sources of evidence mentioned in the survey, including diversity and inclusion statements.
Table 3 provides the number of hiring chairs who reported using various hiring documents as well as the number of hiring chairs that specifically requested teaching philosophy statements, diversity and inclusion statements, and student evaluations of teaching. Slightly fewer hiring chairs specifically requested each of these three documents than reported using each type of evidence. All further survey analysis regarding the use of hiring documents draws from the number of hiring chairs who specifically requested those documents.
Teaching Philosophy Statements
Our data show that hiring chairs specifically requested a teaching philosophy statement approximately twice as often (73% of respondents) as they requested student evaluations of teaching or diversity and inclusion statements (34% of respondents each) (Table 3). While all institution types requested teaching philosophy statements frequently, Baccalaureate and Master’s institutions did (approximately 77%, 27 of 35 hiring chairs, and 81%, 34 of 42 hiring chairs, respectively) more often than Doctoral institutions (68%, 54 of 80 hiring chairs). In contrast, STEM fields requested teaching philosophy statements in larger numbers (90%, 53 of 59 STEM hiring chairs) than did Humanities and Social Sciences hiring chairs (61%, 30 of 49 hiring chairs, and 65%, 32 of 49 hiring chairs, respectively).
When asked to provide a rationale for why they did or did not request a teaching philosophy statement, the most common reasons for requesting one were to assess how applicants view the practice of teaching and to determine if an applicant’s views on teaching aligned with the institution or department. For example, one hiring chair stated, “[The] teaching philosophy shows whether they value teaching and whether their philosophy fits with the department.” Of the hiring chairs who said that they did not request a teaching philosophy statement, the most commonly cited rationale was because they do not find these statements to accurately reflect an instructor’s teaching practice. As one hiring chair stated, “[Teaching philosophy] statements are too contrived and do not necessarily reflect the reality of what a candidate actually does.”
Diversity & Inclusion Statements
Approximately 35% of hiring chairs indicated that they requested these documents as a way to assess teaching effectiveness (Table 3). We found that hiring chairs in STEM and the Humanities (38%, 22 of 58, and 37%, 18 of 49, respectively) requested diversity statements more than did those in the Social Sciences (29%, 14 of 49) and that hiring chairs at Baccalaureate and Master’s institutions request diversity statements more often than did those at Doctoral institutions (43%, 15 of 35, and 43%, 18 of 42, compared to 27%, 21 of 79, respectively).
We asked hiring chairs why they did or did not request a diversity and inclusion statement. Of the 54 respondents who requested it, the most common reason was to see if the applicant’s views on diversity and inclusion aligned with the institution’s or department’s values. For example, “This is an important mission of our College. These statements give us important insight into whether a candidate has thought about these issues and considered how to address them.” Those that did not request diversity statements gave a range of responses. Some respondents noted that while they have not yet requested diversity statements, they would consider doing so in the future. Others indicated that they gathered this information via other means (e.g., on-campus interviews), so they did not need to request a stand-alone statement. Finally, some respondents thought that the diversity and inclusion statement could reveal aspects of a job applicant’s identity (e.g., race, ethnicity, gender).
Student Evaluations of Teaching
Approximately 35% of the hiring chairs requested student evaluations of teaching from candidates. Social Sciences departments reported requesting them most often (63%, 31 of 49 hiring chairs), followed by Humanities departments (29%, 14 of 48 hiring chairs) and STEM departments (14%, 8 of 59 hiring chairs). Less variation existed among Carnegie Classifications with Master’s and Doctoral institutions requesting at similar rates (36%, 15 of 42 hiring chairs, and 35%, 28 of 79 hiring chairs, respectively) and Baccalaureate institutions requesting student evaluations of teaching slightly less frequently (29%, 10 of 35 hiring chairs).
Hiring chairs expressed conflicting views about the usefulness of student evaluations as a measure of teaching effectiveness. For example, one hiring chair indicated their potential helpfulness: “This is often the only indicator of effective teaching that is available. In addition, it often coincides with students teaching a course as [an] instructor of record (as opposed to a TA assignment), which is [an] important experience.” Other hiring chairs indicated that student evaluations of teaching are not helpful, either because graduate students do not tend to have teaching evaluations or because the evaluations (and student responses to them) are biased. For example, one respondent replied, “We didn’t want to ask for too many materials from applicants and student evaluations can be subjective, with many variables, and are not always an effective way of ascertaining good teaching.”
Research Question 3: In using teaching philosophy statements, diversity and inclusion statements, and student evaluations of teaching to evaluate teaching effectiveness, what features are important? What makes such documents effective from a hiring committee perspective?
Teaching Philosophy Statement
Table 4 lists the mean importance ratings for specific topics contained within teaching philosophy statements, all of which were drawn from previous research (Kaplan et al., 2007; Meizlish & Kaplan, 2008). Hiring chairs tended to agree that teaching methods, student learning goals, discipline-specific considerations, inclusive teaching, and student assessment are each important topics in this genre of hiring document.
|Features||Mean rating (SD)|
|Teaching methods||4.5 (.7)|
|Student learning goals||4.1 (.8)|
|Discipline-specific considerations||4.0 (.7)|
|Inclusive teaching||4.0 (.9)|
|Student assessment||3.7 (.8)|
Note. Scale: 1 = strongly disagree; 2 = disagree; 3 = neither agree nor disagree; 4 = agree; 5 = strongly agree.
Additionally, when hiring chairs described characteristics of an effective teaching statement, five themes emerged qualitatively (Table 5). These themes do overlap with responses from closed-ended questions (Table 4), especially regarding instructional strategies, student-centered inclusive teaching, and learning goals, but additional themes emerge, such as describing and more specifically reflecting on past teaching experiences.
|Characteristic and percentage of respondents||Strategy emerging from hiring chair responses||Suggested way to implement strategy (if reported)||Illustrative survey response|
|Instructional strategies (44.8%)||Applicants should describe specific instructional strategies.||Highlighting innovative ways to work with students
Discussing active learning or student-centered instruction
Reporting on specific approaches for various components of teaching (i.e., assessment)
|“Addresses some creative ways to approach students … ”
“Whether the candidate discusses high impact teaching practices.”
|Attitudes toward students (31.0%)||Applicants should specifically highlight their views regarding students.||Describing inclusive practices||“If they are aware of different student-centered and active learning approaches and are concerned with inclusivity in the classroom.”|
|Reflection broadly on teaching practice or experience (29.9%)||Applicants should reflect upon their teaching and discuss how to improve future practice based on student feedback or experiences.||N/A||“They have reflected on what went well and what did not, and are reflective of what improvements they plan to make.”|
|Teaching experience (27.6%)||Applicants should discuss past teaching experiences and highlight the ways in which they have worked with students. This theme often (but not always) overlapped with the theme of “reflection on teaching.”||N/A||“I find it most compelling when a candidate can reflect concretely on experiences they have had as a teacher or as a student and how those shape her/his approach to teaching…”|
|Attention to learning outcomes (20.7%)||Applicants should address what students will learn in the course (learning outcomes, goals for students to attain, etc.)||N/A||“An effective teaching statement provides specific examples from classroom activities, assignments, and/or approaches that illustrate the overall learning outcomes and teaching goals referenced.”|
We also asked hiring chairs to describe common shortcomings in the teaching philosophy statements they review. The top three shortcomings reported were being too generic (31%), lacking classroom examples (24%), and lacking explanations for pedagogical buzzwords (16%). The unifying theme among these shortcomings is an absence of description and detail. For example, one hiring chair wrote:
Common shortcomings included a lack of connection between ideas/ theories articulated and the examples offered, a lack of specific examples, reliance on clichés (e.g., my classroom is student centered), and a lack of vocabulary for discussing teaching practices, assessment, and student learning outcomes.
When including educational terms, theories, or common pedagogical buzzwords, several hiring chairs encouraged applicants to demonstrate their understanding by explaining these things or providing a rationale or application of them, rather than merely listing educational ideas. The following two quotations from hiring chairs exemplify this theme: “Buzzwords need to be unpacked. For example, don’t just speak to critical thinking, speak to what it is and how it can be driven in the classroom” and “statements that employ the current jargon without clear purpose or understanding—e.g. they ‘flip the classroom’ but don’t quite see why this would be effective or have mechanisms to monitor and adjust.”
Diversity and Inclusion Statements
We asked hiring chairs to indicate all of the formats in which they might request a diversity and inclusion statement. Therefore, the following percentages total more than 100%. Of the hiring chairs who indicated that they requested a statement of diversity and inclusion , approximately 59% requested this information as a stand-alone document, and 46% requested that this information be included in the applicant’s cover letter. Notably, almost 26% of respondents said that they requested the information be included in the applicant’s teaching philosophy statement.
To solicit feedback about the content of diversity and inclusion statements, we asked hiring chairs to rate the importance of four content domains (Table 6) adapted from research done at the University of Michigan’s National Center for Institutional Diversity (NCID) (Flaherty, 2018; Sylvester et al., 2019). On average, hiring chairs rated research on diversity equity and inclusion as less important than an understanding of diversity and inclusion in higher education, inclusive teaching and mentorship, and service and/or engagement regarding diversity and equity.
|Understanding of diversity and inclusion in higher education||Teaching and mentorship||Service and/or engagement||Research (e.g., current or planned research relevant to diversity, equity, and inclusion)|
||4.6 (.6)||4.4 (.6)||4.1 (.7)||3.2 (.9)|
Note. Scale: 1 = strongly disagree; 2 = disagree; 3 = neither agree nor disagree; 4 = agree; 5 = strongly agree.
Student Evaluations of Teaching
We asked hiring chairs to provide advice to applicants on how to present their student evaluations of teaching. Of those who responded to the open-ended question, almost 54% encouraged applicants to provide an interpretation of the evaluations or relevant information to help the committee contextualize the data. For example, applicants could consider including data tables, graphs, or department averages for course evaluations. Furthermore, some hiring chairs cautioned against selecting only the most favorable evaluations, to avoid the perception of “cherry picking.” While 10% of respondents (8 of 84) advocated for including favorable student evaluations only, approximately 27% of respondents (23 of 84) specifically stated that applicants should include all student evaluations, both favorable and unfavorable, and use them as an opportunity to reflect on how they have changed and developed as an instructor. One hiring chair summarized this approach by saying, “Provide the full and complete evaluations and provide context when necessary. Don’t try to ignore poor results, but recognize weaknesses and discuss efforts to improve.”
The results of our study hold implications for three broad and overlapping audiences—SoED researchers, CTL leadership engaged in strategic planning, and educational developers delivering the programs and services to prepare future faculty. Below, we discuss the implications of our results for each of these audiences in turn.
Our study provides empirical data regarding the role of teaching effectiveness and how it is evaluated in tenure-track hiring. Our results replicate those of previous studies across a broader range of disciplines and provide updated data. Specifically, our findings validate and update existing literature, showing the following:
In previous studies (Bruff, 2007; Meizlish & Kaplan, 2008), hiring chairs identified teaching and publication/research as the top two factors they considered, and our findings are consistent with these earlier results. After 10 years, teaching remains “very” or “extremely important” in tenure-track hiring, regardless of institution type or academic division. And while Doctoral institutions perceived teaching as less important in hiring than did Baccalaureate and Master’s institutions, STEM, Humanities, and Social Sciences disciplines did not differ significantly.2
Hiring committees use a variety of sources to evaluate teaching effectiveness, and the teaching philosophy statement is one of the most commonly used documents (Bruff, 2007; Meizlish & Kaplan, 2008). In our study, approximately 85% of hiring chairs reported using this document (Table 3) to evaluate applicants, which is considerably higher than the approximately 57% reported by Meizlish and Kaplan (2008).
Hiring committees continue to use student evaluations of teaching as a common data source to evaluate teaching effectiveness. In both our study and Meizlish and Kaplan (2008), approximately half of the hiring chairs (48% and 54%, respectively) reported using student evaluations of teaching. In Bruff (2007), approximately 75% of hiring chairs reported using student evaluations.
Additionally, previous research (Meizlish & Kaplan, 2008) found evidence that hiring committees used an applicant’s research talk as a proxy for teaching effectiveness, and Bruff (2007) reported that 76% of Mathematics hiring chairs use this practice. Our study found that this practice continues and is slightly more common. Across all of our academic divisions, approximately 83% of hiring chairs reported using an applicant’s job talk as a measure of teaching effectiveness. The prevalence of this practice raises some questions, given that these talks tend to be designed for a more expert audience (e.g., other faculty members) rather than novice students. Additionally, while a research talk can demonstrate transferable lecture-based teaching skills (e.g., organizing content, implementing multimedia learning principles, responding to questions), these talks traditionally lack demonstration of evidence-based teaching strategies, such as active learning (e.g., Freeman et al., 2014; Prince, 2004), that may be desirable for evaluating teaching effectiveness. Additional research should be done to examine the rationale for this practice as a proxy for teaching as well as the criteria hiring committees use to evaluate a research talk as a measure of teaching effectiveness.
Our results indicate that approximately 34% of hiring chairs requested and approximately 48% used student evaluations of teaching, if provided. Additionally, our data show that hiring chairs have different views about the use of these documents to measure teaching effectiveness. These differing opinions may represent the disparity between what peer-reviewed research says about student evaluations of teaching and how that research gets (mis)interpreted in the popular press. Barre (2015) draws attention to this problem while also summarizing the research, stating that “student evaluations of teaching are a useful, but ultimately imperfect, measure of teaching effectiveness.” Additionally, Barre notes that scholars in this field recommend that other measures of teaching effectiveness be used in conjunction with student evaluations. As such, further research is needed to provide hiring chairs with recommendations on how to assess this data source that will then provide more guidance to applicants on how to present student evaluations in their application materials.
Previous research has not examined how hiring committees use diversity and inclusion statements to evaluate teaching effectiveness, and our study serves as an important first step in answering this question. Our results show that approximately 38% of hiring chairs use a diversity and inclusion statement to assess teaching effectiveness. Additionally, hiring chairs value an applicant’s understanding of diversity and inclusion issues in higher education as well as how diversity and inclusion is centered in one’s teaching, mentorship, and service. Our data raise some questions about the rationale for requesting this document. For example, approximately 25% of the hiring chairs who requested a diversity and inclusion statement expected it to contain information about the applicant’s identity (e.g., race, gender, ethnicity). The variety of responses and their potential implications point to the lack of shared understanding regarding the content of diversity and inclusion statements and underscore the need for hiring committees to explain their rationale for requesting these documents, as similarly found in Flaherty (2018). Approximately 25% of the hiring chairs who did not request a diversity and inclusion statement indicated that they had interest in requesting this document in the future. This finding, coupled with the events of 2020 (e.g., the murder of George Floyd, renewed emphasis on the Black Lives Matter movement), suggest that diversity and inclusion statements will be requested more often. We hope future research will continue to address the use and interpretation of this relatively new genre of hiring document.
In addition to the research directions noted above, we hope future research may address several of the limitations of our study. First, our study does not include any disciplines from the Fine Arts (e.g., music, drama, art, design) or health science professional schools (e.g., nursing, medicine, pharmacy, dentistry). These disciplines have specific pedagogies (e.g., individual, studio-based teaching in fine arts and clinical instruction for health sciences) that may result in different measures of teaching effectiveness than the ones described in our study. Second, our data were collected during the 2018–2019 academic year; therefore, it cannot speak to the impacts of the ubiquitous transition to remote and/or hybrid teaching that higher education experienced in response to the COVID-19 pandemic. Future studies can and should examine how (if at all) this shift in teaching modalities impacts how teaching effectiveness is evaluated in tenure-track hiring. For example, hiring committees may increasingly prioritize teaching effectiveness in remote or hybrid teaching formats, and evaluation criteria and data sources may change accordingly. Third, our overall response rate was lower than we would have liked. Nevertheless, our sample provides nationwide data with representation across STEM, Humanities, and the Social Sciences as well as institution types (Carnegie Classifications) in proportion to tenure-track job ads posted during our sampling period.
Our results hold implications for CTL leadership in terms of strategic planning and resource allocation. At the broadest level, hiring chairs tended to rate teaching effectiveness as very important in tenure-track hiring. Therefore, we recommend that CTLs who prioritize preparing graduate students and postdocs for the academic job market, and who have available resources and staff, develop and sustain programs and services to help these clients demonstrate their teaching effectiveness. When resources are scarce, CTL leadership may need to prioritize certain programs and services over others. For example, given that hiring committees both requested and used statements of teaching philosophy more often than any other hiring document (Table 3), CTLs might wish to prioritize workshops and individual consultations on these statements.
Our data suggest that CTLs should holistically consider how best to approach diversity, equity, and inclusion (DEI) in their offerings for future faculty. In terms of an applicant’s understanding of and demonstrated commitment to DEI, our findings suggest that hiring committees may look for this information across multiple data sources. For example, as shown in Table 4, hiring committees expect applicants to address inclusive teaching practices in their teaching philosophy statement. Additionally, we found that almost 35% of hiring committees specifically request a diversity and inclusion statement and use this statement to evaluate teaching effectiveness. However, as Table 6 shows, the content of diversity and inclusion statements does not pertain solely to teaching, and CTLs may not be equipped to fully address these other aspects of the statement. At minimum, CTL programs and services can make job applicants aware that they may be asked to demonstrate their perspectives and practices regarding DEI and alert them to the various ways they may be asked to do this on the academic job market (e.g., teaching philosophy statement, diversity and inclusion statement). CTLs can also adjust their offerings to help graduate students and postdocs develop their knowledge, practices, and capacity for enhancing DEI as it relates to their teaching. Such efforts might take the form of seminars or workshops explicitly targeting inclusive teaching, creating accessible instructional materials, or other DEI topics. In parallel, CTLs can endeavor to infuse a DEI lens into programs and services that do not overtly focus on DEI (e.g., syllabus consultations, seminars on designing and managing group work or effective grading and feedback practices). Additionally, CTLs could consider offering specific programming and services for diversity and inclusion statements, staff and resources permitting. Given that diversity statements encompass other aspects of future faculty life aside from teaching, CTLs leadership may benefit from collaborating with other campus units to support job applicants’ development of diversity and inclusion statements.
Our data suggest that providing opportunities for clients to reflect on and contextualize their student evaluations can best support future faculty. For example, CTLs could offer individual consultations that allow graduate students and postdocs to reflect on their evaluations of teaching and identify areas of strength as well as those for growth and development. Job applicants could then draw on this information when writing their teaching philosophy statement or considering what other documents to submit to a hiring committee (e.g., sample lesson plan, sample rubric). Such consultations would benefit job applicants, regardless of whether they actually submit their teaching evaluations, by allowing them an opportunity to reflect on and speak about their teaching practice.
In addition to the three hiring documents described above, our results show that hiring committees commonly assess teaching effectiveness in a variety of other ways. For instance, the practice of having a candidate teach a sample lesson has become more common, and this in an area where CTLs could support job applicants by helping them develop this type of lesson plan or provide opportunities to practice and receive feedback on the implementation of evidence-based teaching techniques in their sample lessons. Similarly, the practice of using a job applicant’s research talk as a proxy for teaching practice is quite prevalent (approximately 83% of hiring chairs reported doing this), but it’s less clear what type of support CTLs might provide here given the questions raised about this practice in the previous section.
While it is important to educate future faculty about the variety of ways in which they may need to demonstrate their teaching effectiveness, it may not be necessary for CTLs to develop services and programs to support all of these measures. Helping job applications prepare for one deliverable or interaction will likely assist with them with the preparation for others. For example, writing a teaching philosophy statement requires an applicant to identify and reflect on the types of teaching experiences that they have had or would like to have. These experiences might be concisely captured in a teaching philosophy statement, and an applicant might further elaborate on them during an interview. Similarly, an applicant might demonstrate a particular teaching method that they describe in their teaching philosophy statement during their sample teaching session.
Our findings directly inform the work of educational developers who support graduate students and postdocs in demonstrating their teaching effectiveness, especially during one-on-one consultations or during workshops. In this section, we focus on how our data can inform one’s consulting toolkit, recognizing that these practices also translate to the design and delivery of seminars and workshops.
Overall, our findings highlight the importance of teaching in tenure-track hiring, and educational developers can leverage this point to validate the significant effort that job applicants put into demonstrating their teaching effectiveness. At our CTLs, educational developers frequently provide feedback to clients on hiring documents and may encounter resistance when they suggest clients iterate on a draft or create multiple versions tailored to different job offers. In particular, our data suggest that when hiring committees ask for specific documents, namely a teaching philosophy statement and a diversity and inclusion statement, they are doing so to understand how the applicant’s views on these subjects align with the mission and values of the institution. Consequently, educational developers might encourage job applicants to research the institutions/departments to which they are applying in order to tailor their hiring documents for different contexts. If hiring committees do not request specific documents, then educational developers can help job applicants consider the breadth of sources they can draw from to demonstrate their teaching effectiveness.
Educational developers can use our results to inform feedback or suggestions on the content of teaching philosophy statements. Hiring chairs appeared to have consensus, across institution types and academic divisions, about their rationale for requesting teaching philosophy statements as well as the characteristics of effective statements (Tables 4 and 5). While these statements are highly personal and individualized, our findings present some broad categories of information that should be present. For example, hiring chairs are looking for concrete, discipline-specific examples of an applicant’s teaching methods and learning objectives. Previous research (Bruff, 2007; Meizlish & Kaplan, 2008) reported similar findings, suggesting that hiring chairs have a stable and consistent view of what they are looking for in these documents. Approximately 30% of our respondents indicated that these concrete examples should be accompanied by reflection on one’s teaching, including a discussion of how one could improve as an educator. Bruff (2007) reported a similar finding, with 28% of hiring chairs indicating that reflection was a characteristic of an effective teaching statement. Hiring chairs in our study as well as the two previous ones (Bruff, 2007; Meizlish & Kaplan, 2008) recommended that applicants be specific with the content of their teaching philosophy statements and avoid the use of pedagogical buzzwords without accompanying descriptions of examples. When providing feedback to job applicants on their teaching philosophy statements, educational developers should look for this content and prioritize their feedback accordingly, perhaps through the use of rubrics or general guidance about writing teaching philosophy statements.
Educational developers may provide a variety of services to help graduate students and postdocs develop an understanding of diversity and inclusion with regard to their teaching. Based on our data, these offerings might focus on communicating one’s understanding, values, and/or current and future practices and engagement regarding enhancing DEI in higher education, inclusive teaching and mentorship, and service activities regarding DEI. For those who specifically help job applicants write diversity and inclusion statements, our results are aligned with the recommendations from NCID. NCID director Tabbye Chavous described diversity and inclusion statements: “Generally speaking, these statements should articulate the way a candidate understands DEI and show the candidate’s DEI skills and competencies” (as cited in Smith, 2019). As shown in Table 6, these skills and competencies do not pertain solely to teaching and need not include information about an applicant’s identity. Conveying this information to job applicants will ensure that they address these various aspects of DEI work and may help to clarify concerns about divulging protected identities. Chavous recommends that institutions be explicit about their rationale for requesting these statements (Flaherty, 2018), and educational developers can encourage job applicants to look for these rationales in the job requests.
Educational developers are frequently asked to provide guidance for how to present student evaluations of teaching to hiring committees. At a minimum, educational developers can make job applicants aware of the differing viewpoints on student evaluations expressed by hiring chairs. Given that fewer hiring chairs specifically requested student evaluations of teaching (approximately 34% requested compared to approximately 48% who used them), job applicants may want to consider whether to submit these documents if not required to do so. Educational developers can help job applicants to consider what other measures they might draw from and how best to package this information alongside their teaching evaluations (Linse, 2017). If an applicant submits their evaluations, regardless of whether they were asked to, educational developers should encourage applicants to provide summative data, interpretations or contextualization of the data, or additional descriptions about their teaching context to help the hiring committee interpret the evaluations. Additionally, educational developers should encourage job applicants to submit all of their teaching evaluations—the good and the bad—and use the negative ones to discuss opportunities for growth and improvement in their teaching practice.
Taken together, our results show that teaching effectiveness continues to be a top factor in tenure-track hiring and that job applicants are asked to demonstrate their teaching effectiveness in a wide variety of ways. Some “staples” of the hiring process remain (e.g., teaching philosophy statement), and new ones have been added (e.g., diversity and inclusion statement). CTLs and educational developers need updated data, representing a variety of academic disciplines and institution types, to inform the programs and services that they provide to job applicants. We are excited to provide this data, and we look forward to what future SoED researchers contribute to this ever-changing landscape.
The institutional review board (STUDY2017_00000575) approved this research at the university where it was conducted. The authors thank Doug Piper and Sasha Cohen Ioannides, who retrieved and coded job advertisements and located contact information for department chairs. We also thank Nicole Tuttle for advice and resources on diversity and inclusion statements and Michael Melville for assistance with deploying the survey.
Katharine Phelps Walsh is Associate Director of Learning Support at the Student Academic Success Center at Carnegie Mellon University. Katie directs the Supplemental Instruction and peer tutoring programs, which provide learning support to students enrolled in historically challenging courses. Additionally, Katie teaches courses in the CMU History Department and in Dietrich College’s Grand Challenge Program.
Laura Ochs Pottmeyer is a Data Science Research Associate at the Eberly Center for Teaching Excellence and Educational Innovation at Carnegie Mellon University. Laura works with instructors to collect and analyze classroom data in order to make data-informed decisions for iterative improvement of their courses.
Deborah Meizlish is Associate Director of the Center for Research on Learning and Teaching at the University of Michigan. In addition to her work on academic hiring, she was a coinvestigator on a study of metacognition funded by the Teagle and Spencer Foundations, and she co-edited the volume Using Reflection and Metacognition to Improve Student Learning: Across the Disciplines, Across the Academy.
Chad Hershock is the Director of Faculty and Graduate Student Programs at the Eberly Center for Teaching Excellence and Educational Innovation at Carnegie Mellon University. Chad leads a team of teaching consultants and data science research associates who support instructors through data-driven consultation services and evidence-based educational development programming. Together, they collect, leverage, and disseminate data on student outcomes to help instructors iteratively refine their teaching and course design.
- For the two-way ANOVA, teaching effectiveness was the dependent variable, while institution type and academic division were fixed factors. Results indicated that the interaction between institution type and academic division was not statistically significant, , . Similarly, the main effect of academic division was not statistically significant, , . The main effect of institution type was statistically significant, , , . Bonferroni-adjusted, pairwise comparisons indicate significantly higher means for both Baccalaureate and Master’s institutions compared to Doctoral institutions in our sample (0.58 points higher for Baccalaureate, 95% CI [.28,.89] and 0.48 points higher for Master’s, 95% CI [.19,.77]). Means did not differ significantly between Baccalaureate and Master’s institutions . [^]
- Correction notice: As of Dec. 16, 2022, this sentence has been revised from “...perceived teaching as more important in hiring...” to “...perceived teaching as less important in hiring...” (emphases added). [^]
Adams, K. A. (2002). What colleges and universities want in new faculty. Number 7.
Association of American Colleges and Universities.
Barre, B. (2015, July 9). Do student evaluations of teaching really get an “F”? Reflections on Teaching and Learning. https://cte.rice.edu/blogarchive/2015/07/09/studentevaluations
Beach, A. L., Sorcinelli, M. D., Austin, A. E., & Rivard, J. K. (2016). Faculty development in the age of evidence: Current practices, future imperatives. Stylus Publishing.
Bruff, D. (2007). Valuing and evaluating teaching in the mathematics faculty hiring process. Notices of the American Mathematical Society, 54(10), 1308–1315.
Canning, C. M., & Reddick, R. J. (2019, January 11). In defense of diversity statements. The Chronicle of Higher Education. https://www.chronicle.com/article/In-Defense-of-Diversity/245463
Center for Postsecondary Research. (2018). Carnegie Classifications of Institutions of Higher Education: 2018 update facts and figures. Indiana University School of Education. http://carnegieclassifications.iu.edu/downloads/CCIHE2018-FactsFigures.pdf
Cook, C., & Kaplan, M. (2011). Advancing the culture of teaching on campus: How a teaching center can make a difference. Stylus Publishing.
Flaherty, C. (2018, November 19). Breaking down diversity statements. Inside Higher Ed. https://www.insidehighered.com/news/2018/11/19/new-paper-explores-what-faculty-candidates-include-their-diversity-equity-and
Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415.
Golash-Boza, T. (2016, June 10). The effective diversity statement. Inside Higher Ed. https://www.insidehighered.com/advice/2016/06/10/how-write-effective-diversity-statement-essay
Hershock, C., Cook, C. E., Wright, M. C., & O’Neal, C. (2011). Action research for instructional improvement. In Cook, C. E. & Kaplan, M. (Eds.), Advancing the culture of teaching on campus: How a teaching center can make a difference (pp. 167–182). Stylus Publishing.
Hershock, C., Groscurth, C. R., & Milkova, S. (2011). Approaches to preparing future faculty for teaching. In Cook, C. E. & Kaplan, M. (Eds.), Advancing the culture of teaching on campus: How a teaching center can make a difference (pp. 97–117). Stylus Publishing.
Kaplan, M., Meizlish, D. S., O’Neal, C., & Wright, M. C. (2007). A research-based rubric for developing statements of teaching philosophy. To Improve the Academy, 26, 242–262. https://doi.org/10.3998/tia.17063888.0026.020
Kearns, K. D., & Sullivan, C. S. (2010). Resources and practices to help graduate students and postdoctoral fellows write statements of teaching philosophy. Advances in Physiology Education, 35(2), 136–145. https://doi.org/10.1152/advan.00123.2010
Linse, A. R. (2017). Interpreting and using student ratings data: Guidance for faculty serving as administrators and on evaluation committees. Studies in Educational Evaluation, 54, 94–106. https://doi.org/10.1016/j.stueduc.2016.12.004
Meizlish, D., & Kaplan, M. (2008). Valuing and evaluating teaching in academic hiring: A multidisciplinary, cross-institutional study. The Journal of Higher Education, 79(5), 489–512. https://doi.org/10.1080/00221546.2008.11772114
Palmer, M. S., & Little, D. (2013). Tomorrow’s professor today: Tracking perceptions of preparation for future faculty competencies. To Improve the Academy, 32(1), 251–268. https://doi.org/10.1002/j.2334-4822.2013.tb00709.x
POD Network. (2018). 2018–2023 strategic plan. https://podnetwork.org/content/uploads/POD_Strategic-Plan_2018-23.pdf
Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231. https://doi.org/10.1002/j.2168-9830.2004.tb00809.x
Pruitt-Logan, A. S., & Gaff, J. G. (2004). Preparing future faculty: Changing the culture of doctoral education. In Wulff, D. H. & Austin, A. E. (Eds.), Paths to the professoriate: Strategies for enriching the preparation of future faculty (pp. 177–193). Jossey-Bass.
Reyes, V. (2018, January 25). Demystifying the diversity statement. Inside Higher Ed. https://www.insidehighered.com/advice/2018/01/25/how-write-effective-diversity-statement-job-candidate-opinion
Rozaitis, B., Baepler, P., Gonzalez, A., Ching, P., Wingert, D., & Alexander, I. D. (2020). Preparing future faculty: Pedagogical practice in graduate school. New Directions for Teaching and Learning, 2020(163), 35–43. https://doi.org/10.1002/tl.20408
Smith, B. (2019). The case for diversity statements. University of Michigan College of Literature, Science, and the Arts. https://lsa.umich.edu/lsa/news-events/all-news/search-news/diversity-statements.html
Sylvester, C.-Y. C., Sánchez-Parkinson, L., Yettaw, M., & Chavous, T. (2019). The promise of diversity statements: Insights and an initial framework developed from a faculty search process. Currents: Connecting Diversity Scholarship to Practice and Society, 1(1). https://doi.org/10.3998/currents.17387731.0001.112
Whitaker, M. (2020, November 24). 5 don’ts in writing your DEI statement. The Chronicle of Higher Education. https://www.chronicle.com/article/5-donts-in-writing-your-dei-statement?cid2=gen_login_refresh&cid=gen_sign_in&cid2=gen_login_refresh
Willis, D. S. (2017). Getting up to speed on diversity. Inside Higher Education. https://www.insidehighered.com/advice/2017/08/21/how-graduate-students-can-demonstrate-commitment-diversity-job-interviews-essay
Appendix A: Sources for Job Ad Sampling
During a 4-week period (September 17–October 12, 2018), we located job ads for the following disciplines on the disciplinary websites listed. In addition to each of these websites, we searched The Chronicle of Higher Education’s website for jobs in all of the same disciplines. We continued to pull job ads until either the 4-week period expired or we reached 200 jobs ads for a discipline.
|Discipline||Professional organization website|
|History||American Historical Association|
|English||Modern Language Association|
|Biology||American Society for Microbiology|
|Chemistry||Chemical & Engineering News|
|Psychology||American Psychological Association|
|Political Science||American Political Science Association|
|Computer Science||Association for Computing Machinery|
|Civil & Environmental Engineering||American Society of Civil Engineers|
|Association of Environmental Engineering and Science Professors|
|Mechanical Engineering||American Society of Mechanical Engineers|
|Business||American Finance Association|
|American Marketing Association|