Methodology—Faculty Feedback

The CTB-MFI was implemented on a limited scale in Spring 2021 with 11 faculty from the COAS only. At the end of the semester, researchers reached out to faculty who had conducted the CTB-MFI in their courses to request feedback on the CTB-MFI format, clarity, and usefulness. In Fall 2022, after implementing the CTB-MFI process with a wider audience, faculty participants were again invited to complete a version of this survey focused on faculty perceived usefulness of the quantitative and qualitative feedback responses included in the CTB-MFI by responding to the following questions:

• When reviewing your midterm feedback report, how useful was the overview (data table and overview chart) of feedback students provided by answering the ranking questions?

• Please explain your rating. Consider, for example, what did you find useful/ not useful? How did you use the data from ranking questions? What would make this type of data more useful?

• When reviewing your midterm feedback report, how useful was the narrative overview of students’ open response feedback on your teaching behaviors, including representative student comments?

• Please explain your rating. Consider, for example, what did you find useful/ not useful? How did you use the data from narrative feedback questions? What would make this type of data more useful?

• CTB Questions: Anything missing? After reviewing the ranking questions on the survey, were you hoping to receive feedback on any additional ranking items not included in the survey?

Results & Discussion—Faculty Feedback

A total of 37 faculty from COAS and COE responded across the two semesters (see Table 1). In Spring 2021, most faculty indicated that they did not feel anything was missing from the questions asked and offered no direct suggestions for changes, though few provided important feedback in response to prompts asking them to explain their ratings that led to minor revisions of survey items to increase clarity and relevance. For example, based on feedback received, researchers added a list of potential technology options, including the university’s learning management system (LMS), to the question that asked students to report the frequency with which their instructor “[u]ses technologies and/or apps that enhance your learning experience in the course.” Additionally, a faculty comment stating “I was looking for more focused questions on the clarity of assignments and grading rubrics, rather than just the opportunities to have learning assessed” prompted the revision of questions to more specifically gather student input on how frequently assignments expectations are clearly communicated. Faculty responses to the survey questions rating the usefulness of the CTB-MFI tool are summarized in Table 1.

These data show faculty consider both types of feedback to be useful, but overall they perceive the insights gained from qualitative student responses to be more useful than those provided by quantitative data, a finding that aligns with insights from an earlier study showing that faculty prefer student comments to Likert scores (van Wyk & Mclean, 2007). Interestingly, this does not hold true for engineering faculty, who on average considered both qualitative and quantitative feedback to be equally useful, indicating a marked difference between faculty preferences in arts and sciences (COAS) and engineering (COE).

When asked to explain their ratings, faculty elaborated that the quantitative scores were useful because they provided an at-a-glance overview of effective teaching behaviors and how “well” faculty were integrating them into their courses; one COE instructor stated, “[t]he quantitative data of different categories were helpful to see how I was doing in the different categories of effective teaching behavior, and quick [sic] understand the areas I need to improve on.” Faculty also mentioned that the data provided a baseline that they would be able to track over time: “I suspect that tracking these changes over time would be more helpful, especially if they could be cross-referenced with end-of-term evaluations to show trends of growth over time.” Faculty were aware of how the data might assist them in summative evaluations. One faculty member stated, “I’m glad we had [quantitative data] to consider, especially since numbers are often what higher admin are most likely to pay attention to!” That said, several survey respondents mentioned that they found quantitative data useful only in combination with the qualitative feedback provided.

The qualitative student comments provided faculty with more specific insights and concrete guidance for understanding student concerns and making pedagogical changes. Specifically, faculty stated that comments helped them identify pedagogical opportunities: “The narrative data helped me understand what the students are confused about, where I need to change my teaching methods and what I need to clarify moving forward.” Faculty also appreciated how the qualitative feedback provided much needed context: “Having some explanation of the scores in the quant section and specific feedback about what’s happening in the course (and how widespread agreement with that concern is) was the most helpful part.”

Finally, one aspect of the CTB-MFI that goes beyond student feedback responses and separates it from other instruments is its explicit purpose to be used as a framework intended to guide both instructor reflection and discussion around pedagogy. Faculty feedback indicated that they found the instrument extremely useful in both areas. For instance, one participant said, “[the CTB-MFI] gave us [the educational developer and faculty member] a structure for the discussion,” recognizing how the tool was able to productively guide both the discussion facilitator and the faculty receiving feedback through the debrief conversation. Another shared how the analysis of the feedback data was a positive experience: “I enjoyed seeing how [the CTB-MFI] was explained and [it] gives me concrete feedback for areas I need to focus on to improve my work in the future.”

Summary—Faculty Feedback

Overall, faculty found both quantitative and qualitative feedback helpful in understanding student learning experiences and perceptions of teaching in their courses. Some faculty emphasized that quantitative data serves as a useful addition that provides at-a-glance insights and a foundation for documenting teaching strengths for an external audience. Other faculty emphasized that qualitative feedback must remain an essential component of midterm feedback as it provided context and depth to the survey data. The instrument and associated report contribute to the establishment of a common definition and framework of effective teaching and shared language for pedagogical review and discussion.

Methodology—CFA

Teaching center staff administered the midterm feedback survey via Microsoft Forms in Fall 2022 in 151 sections taught by 76 faculty. Table 2 provides more details on the disciplines represented and course level of the sections in which the CTB-MFI was administered. Based on faculty preference, teaching center staff either visited the classroom to introduce the survey and facilitate its completion in the absence of the faculty or provided a link to the faculty member that was shared with students via the institutional LMS (Canvas). When shared online, students would complete the survey asynchronously. In the classroom, the survey was administered as a standalone feedback tool or in combination with a whole-group class discussion (modeled on the last step in the SGID protocol); when combined with a discussion, students first individually completed the survey before discussing their feedback and observations with classmates. In all cases, students completed the survey anonymously; no identifying data was collected. For this study, only the quantitative portion of the CTB-MFI data collected was considered for the CFA.

Quantitative responses were exported from MS Forms into a spreadsheet; data were randomized and divided into two comparable groups for analysis. A CFA was conducted using one of these data groups to assess the degree to which the specified survey or scale measurement fits the observed data. The CFA helps researchers test whether the items within the survey measure the intended underlying constructs or factors. By doing so, a CFA enables researchers to evaluate the construct validity of surveys and other measurement instruments and validate their theoretical assumptions. Within the output of a CFA, the fit indices, standardized regression weights, assessments of discriminant and convergent validity, and correlations between factors are of particular interest. The following sections present the results of the CFA.

Results & Discussion—CFA

All analyses were conducted utilizing AMOS graphics 26.0. We assessed the fit of the survey using a chi-squared test, the Tucker-Lewis index (TLI), comparative fit index (CFI), and the root mean square error of approximation (RMSEA) (Byrne, 2016; Hu & Bentler, 1999).

Fit indices are used to assess how well the measurement model fits the data. The fit of the survey will be evaluated using several factors including chi-squared, which tests the difference between the observed and expected covariance matrices (Byrne, 2016). In this case, the chi-squared test is significant, indicating a good fit. The TLI and CFI measure how well the model reproduces the observed covariance matrix (Byrne, 2016). Values above 0.95 indicate good fit; therefore, in this case, both values indicate that collected data support the proposed five-category model as a good fit, as shown in Table 3. Finally, the RMSEA value estimates the discrepancy between the hypothesized model and the population covariance matrix (Byrne, 2016). For RMSEA, values below 0.08 indicate good fit, which, as seen in Table 3, is the case for the present model.

The standardized regression weights represent the strength and direction of the relationships between the latent constructs, or factors (e.g., CTB categories such as Align, Include), and their observed variables (e.g., aligned items or questions within the survey) (Byrne, 2016). Each factor has multiple items within the survey, and the weights indicate how strongly each item is related to its corresponding factor. Standardized regression weights typically range from -1 to +1 (Byrne, 2016), with larger absolute values indicating stronger relationships between the constructs (latent variables) and their related survey items. Table 4 shows the different items for each construct (e.g., in the Align category: LEARNING_OUTCOMES, TIME_MANAGEMENT, ALIGNED_ASSESSMENT; see Appendix A for item details) and their standardized regression weights (e.g., 0.737, 0.759, 0.667).

A CFA also allows us to assess convergent and discriminant validity. Convergent validity assesses the degree to which the indicators within a construct are related to each other, whereas discriminant validity evaluates the degree to which different constructs are distinct from one another (Byrne, 2016; Hu & Bentler, 1999). To interpret convergent and discriminant validity within a model, several tests are considered including the average variance extracted (AVE) and maximum shared variance (MSV). The AVE measures the amount of variance captured by the indicators of a construct, with values above 0.5 indicating good convergent validity, as shown in Table 5 for the present model (all above 0.5). This indicates that all items within each construct converge to measure that corresponding single construct. The MSV represents the maximum amount of variance that a construct shares with any other construct (Byrne, 2016; Hu & Bentler, 1999). To have good discriminant validity, the AVE should be greater than the MSV. Table 5 provides the AVE, MSV, and other related measures for each construct (e.g., Tech Integration, Align, Include).

Furthermore, the composite reliability (CR) value represents the reliability or internal consistency of the constructs in the model, with a value of 0.7 or higher being considered acceptable. While the model appears to have excellent convergent validity, the data indicates moderately poor discriminant validity. First, the square root of the AVE for Align, Include, Engage, Assess, and Tech Integration is less than one, the absolute value of the correlations with another factor. Second, the AVE values for Align, Include, Engage, Assess, and Tech Integration are all less than the MSV. This could be an indication of overlapping items within several constructs or concept overlaps. Data showing correlations between different constructs, presented in Table 6, support this assumption; the numbers on the diagonal represent the squared correlation between each construct and its manifest variables (survey items), all of which portray high correlations with one another (with values > 0.5 indicating a strong, positive correlation). In other words, if we theoretically merged these constructs into one overarching latent variable, such as “effective teaching,” the combined constructs would measure it very well.

Summary—CFA

Overall, the results suggest that the model had excellent fit across multiple indicators (e.g., RMSEA, TLI, CFI). This tells us that our data covariance matrix aligns very well with the hypothesized model’s covariance matrix and that our respective measures (e.g., the survey items) are reasonably good indicators of their latent constructs (e.g., the CTB categories). As for construct validity, the model had excellent convergent validity, indicating that the items for each construct are good indicators of that construct. However, the model had moderately poor discriminant validity, which assesses whether the individual latent variables (e.g., the CTB categories) are distinct from one another. In some instances, our test for discriminant validity demonstrates that constructs are moderately related to each other, as seen in Table 5. The overlap between constructs shown in the data comes as no surprise as any instructional behavior may often accomplish multiple purposes. For example, while active learning tasks allow an instructor to Engage students in the learning process, they also have been linked to promoting equitable learning environments (Include) and, if they Align well with learning outcomes, allow us to Assess student progress and understanding. Active learning tasks might also be facilitated by strategic Tech Integration, such as student response systems. In short, connections between categories of critical teaching behaviors often serve as an asset rather than a shortcoming. While good convergent validity of the model allows us to assume that items within each construct provide us with reliable insights into achievements related to the instructional behavior assessed in each category, the low discriminant validity reminds us that ultimately all behaviors are correlated, and good teaching must be considered holistically.

Methodology—Student Feedback

Researchers solicited student feedback to evaluate consistency of student interpretations of each question and to understand students’ thought processes when completing the CTB-MFI; student comments were expected to provide further insights into the construct validity of survey items by allowing researchers to assess whether student interpretations were consistent across participants and aligned with the intended purpose of each item. Undergraduate and graduate students (n = 8) participated in a 45-minute feedback session after being recruited through outreach to student organizations and by posting flyers on approved student communication boards on campus. An overview of participant demographics can be seen in Table 7.

Three feedback sessions with two to three participants each were conducted. First, participants completed a survey that asked them to subsequently read each of the 15 ranking items included on the CTB midterm feedback tool and then (1) restate the question in their own words and (2) describe what kind of evidence they would consider when rating their instructor for this item. Next, students participated in a short focus group conversation to discuss their insights and reaction to the midterm feedback items and to share general comments. Focus group conversations were audio recorded and transcribed. After completing the focus group sessions, the researchers analyzed the results from the written responses and the focus group conversations to identify themes.

Results & Discussion—Student Feedback

Student survey responses, summarized in Appendix C, enrich our understanding of student interpretations of CTB-MFI items and the evidence they would consider in rating their instructors’ behavior. Written survey responses suggest that students’ interpretations of the ranking questions asked on the CTB-MFI were largely consistent across participants with small variations. For example, when restating the item asking whether the instructor “selects examples and activities that represent a range of perspectives and experiences,” student responses showed general agreement with the idea that the instructor should incorporate “a multitude of examples and applications that can benefit the entire class”; specifically, though, some students expected that the examples included would show a diversity of applications across different careers, whereas other students expected instructors to demonstrate that “history has suppressed minorities and that they are actively working to be more inclusive and diverse.”

Students’ responses restating the CTB-MFI items indicate each of the CTB-MFI items is clearly perceived as asking a distinct question. However, responses to the second survey question—“What evidence would you consider when evaluating your instructors’ efforts in this area?”—indicated some overlap between evidence considered in the assessment of different teaching behaviors. For example, students considered instructor availability outside of class time—in-office hours and by email—as an important piece of evidence informing their feedback on how often their instructor “Invites students’ questions, examples, and experiences and listens carefully when students speak” (Include) as well as how frequently their instructor “Establishes regular and open communication” (Engage). This insight mirrors CFA findings indicating some expected overlap between the categories in this model.

Completing the survey required students to engage with and reflect on the ranking items included in the CTB-MFI and provided them with a thorough overview of the tool and the process. In the focus group conversations that followed, students were asked to provide specific feedback on the CTB-MFI tool, particularly if they thought any questions were redundant, irrelevant, or missing, and general impressions of the midterm feedback process. Students agreed that none of the questions were confusing or unclear. They had few suggestions for specific changes to the tool. One group considered that the questions associated with the Tech Integration category were similar and suggested eliminating the following item “Uses technologies and/or apps that enhance your learning experience in the course (e.g., Canvas, multimedia content, polling)”; none of the other groups suggested eliminating any of the items. One focus group responded especially positively to the CTB-MFI items:

S1:

“Well, I’d say these questions are a lot more helpful than those [on the EOC evaluation]… . They are more helpful to evaluate the teaching and learning experience of the student. Whereas some of the questions [on the EOC evaluation]—I can’t remember.”

S2:

“Well, some of them, I would agree, they felt a little bit more course specific.”

This conversation suggests that students appreciate the request to provide feedback on specific, observable teaching behaviors rather than more abstract or ambiguously defined ideas or qualities. Students in all three focus group sessions remarked that midterm feedback sessions should be more strongly encouraged and incentivized for all instructors. One student stated, “I feel like a midterm feedback is more important than end-of-semester survey. End-of-semester survey is just as important, but it’s kind of like when in your classes you don’t know how you’re doing until just like the very end and that’s, that’s not helpful at all.” Several students agreed that midterm feedback is at least as important as EOC feedback, maybe even more so, because it provides valuable information for the instructor to make changes and improvements to benefit students during the current course.

Summary—Student Feedback

Overall, insights from the student feedback sessions support that the CTB-MFI items are clear, relevant, and interpreted consistently across respondents. Each item on the survey is perceived as clearly distinct from the others, although there is some overlap in the evidence students consider when evaluating each item. Students further agreed that they consider midterm feedback in general an important avenue for providing feedback that should be incentivized so more instructors seek formative feedback from students rather than wait until the end of the semester.