In this section, we compare and contrast our CTL’s C&SD consultation service models for faculty clients and graduate students/postdocs participating in our Future Faculty Program (FFP). Similarities are numerous. All consultations are voluntary, confidential, and provided by full-time educational developers. Consultants receive the same training, including shadowing and being shadowed by more experienced colleagues on consultations and participating in monthly professional development sessions to compare strategies and discuss challenging consultation scenarios. Clients meet with consultants one or more times. Discussions typically focus on learning objectives, assessments, instructional strategies, alignment of course design features, and/or course policies. Clients may be designing a course from scratch, revising a syllabus they inherited from previous instructors, or inspired by syllabi encountered elsewhere. Consultants provide resources, as needed, such as a syllabus template, a set of heuristic questions, and/or recommendations from our institution’s Faculty Senate (Eckhardt, 2017). Rather than follow a strict protocol, consultants draw from the same, established C&SD and learning principles (e.g., Ambrose et al., 2010; Fink, 2013; Wiggins & McTighe, 2005) regarding learning objectives, assessments, alignment, inclusive teaching, course policies, and how students learn to provide feedback on clients’ C&SDs. However, to strategically “meet the client where they are,” including what to prioritize, how much feedback to provide, and when to gently push (or not), consultants also have to rely on intuition and be flexible in their approach. After receiving feedback, clients may revise syllabi and receive additional feedback. Given that contextual, client-specific factors influence the exact content of a consultation, one expects some variation in the client experience. Given this variation, our tests of the impacts of consultations are conservative. Detecting a strong signal regarding syllabi improvements after consultations or relative to a comparison group, in spite of this potential variation, actually strengthens interpretations of the benefits of the fundamental, shared parameters of the C&SD service model.

C&SD consultations may differ between faculty and FFP clients. Time on task regarding C&SD tends to be greater during FFP consultations. FFP consultations focus exclusively on C&SD, because it’s a specific requirement of the FFP program. Faculty consultations often include other CTL services and have shorter timelines for iteration. Many faculty clients request a C&SD consultation proximal to the start of a semester when they will teach the targeted course. Fewer FFP clients are (or will be) teaching the courses they are designing at CMU. Thus, consultants may need to prioritize feedback differently for faculty, based on time constraints and what’s possible or most critical.

Additionally, FFP consultations tend to be more scaffolded. Unlike faculty, FFP clients participate in a structured program designed to support skill development and success as educators in a faculty career (Eberly Center, 2021). FFP participants must (a) complete a course and syllabus (re)design project; (b) attend at least eight CTL seminars on evidenced-based teaching and learning; (c) receive at least two teaching feedback consultations; and (d) write a teaching philosophy statement. FFP clients receive feedback on syllabi drafts using a rubric assessing course descriptions, learning objectives, description of assessments, assessment criteria for evaluation (if applicable), course policies, and more (see Appendix A). Faculty clients may also receive feedback on their syllabi, but it is not scaffolded using the FFP rubric. Typically, FFP clients do not receive the rubric prior to submitting their first draft. However, consultants may conduct a generative interview with FFP clients before they draft their syllabi. Generative interviews do not “workshop” elements of course or syllabus design in detail. Instead, they pose guiding questions to help clients get started. Finally, after FFP syllabi pass minimum requirements (on the first draft or after iteration), all FFP clients complete a written reflection on their course designs. This assignment challenges clients to communicate the alignment of course design features, especially regarding instructional strategies, assessments, and learning objectives, because this information is rarely explicit in a traditional course syllabus. Consultants do not request this reflection document from faculty clients.

We compared syllabi for 94 CTL clients before and after they received a C&SD consultation. Syllabus pairs came from the same course.

Two-thirds of our sample came from FFP clients. From January 2017 through June 2018, 62 FFP participants (16 master’s students, 38 doctoral students, and eight postdocs) received a C&SD consultation and submitted syllabi before and after consultations (Table 1).

One-third of our sample came from faculty clients. We identified 57 unique faculty-course combinations that requested a C&SD consultation between August 2017 and June 2019 and shared syllabi with consultants at the beginning of the process. We emailed these clients to request the syllabi they implemented after the consultation and received syllabi from 43 client-course combinations. Eleven respondents submitted syllabi for multiple courses receiving consultations. For repeat clients, we only analyzed syllabus pairs from their earliest consultation, resulting in a final sample of 32 faculty clients (Table 2).

To better assess CTL impacts, we also analyzed syllabi from two comparison groups of faculty who did not receive a C&SD consultation. In 2018, CMU established a course syllabus registry to better support its students. We identified registry syllabi from faculty who did not receive CTL consultations from 2016 through June 2019. We closely matched client and registry syllabi regarding course discipline and level as well as faculty employment track and rank (Table 2). When multiple registry matches occurred, we randomly selected one course. Our first comparison group included a single syllabus from 32 unique faculty. Ten faculty matches in the course registry posted syllabi for the same course taught in consecutive semesters. These faculty formed our second comparison group, accounting for “syllabus drift,” ambient changes in syllabi across semesters from instructors independently revising courses. Together, these comparison groups provide reference points for comparing faculty clients’ syllabi before versus after a consultation. We could not procure a viable comparison group for graduate students and postdocs who did not receive a consultation.

Additionally, we included data on clients’ attendance at CTL seminars on C&SD prior to the completion of consultations. Seminar attendance data spanned July 2016 to the time of consultation. For faculty, we used attendance at the CTL’s annual Incoming Faculty Orientation event, which contains a seminar devoted to C&SD. We excluded faculty attending these seminars from our syllabus registry comparison groups. No other CTL events for faculty focused exclusively on C&SD during this period. For FFP participants, we used attendance data from the CTL’s annual Graduate Student Seminar Series, which offers at least one seminar on C&SD each year.

The FFP consultation rubric was not designed as a research instrument. Therefore, to assess the quality of syllabi in our study, we created a different rubric informed by course design and evidence-based teaching literature (Ambrose et al., 2010; Palmer et al., 2014; Wiggins & McTighe, 2005) and CMU Faculty Senate recommendations (Eckhardt, 2017).

Six of the authors (C&SD experts) served as syllabus coders and calibrated by individually coding two anonymous syllabi not included in this study. Two of the remaining authors (assessment experts) compiled the codes, identified areas of disagreement, and facilitated a discussion in which the coders revised the rubric. With the revised rubric, the coders independently scored a third anonymous syllabus, followed by another round of rubric revision. The final rubric included six broad categories—course description, learning objectives, assessments, assessment and grading policies, other policies, and organization—each with subcategories for specific features (see Appendix A).

Given the consensus among raters following the norming activities, the six coders independently scored a unique subset of 220 syllabi in the study. Syllabi were randomly assigned, but we ensured that coders did not analyze a syllabus from one of their clients or analyze both pre- and post-consultation syllabi from the same client. Prior to coding, we de-identified syllabi regarding instructor identity and study conditions.

With a maximum of 56 points possible on the rubric, coders rated each syllabus on 17 of the 20 rubric items (Appendix A). However, coders only rated the remaining three items (9 total points possible) if they were present in the syllabus. Coders agreed that these items (participation grading, other policies, and explanation for other policies) were not necessarily essential in syllabi and thus “optional” from a consultant’s perspective. Consequently, we did not count their absence against the score of a syllabus and reduced the total possible points for that syllabus. We converted rubric scores to a proportion to account for the variation in possible point totals and to compare across rubric categories.

To investigate the impact of consultations on C&SD practices (Research Question 1), we conducted separate analyses for faculty and FFP clients because we were unable to obtain an FFP comparison group. We analyzed FFP syllabus scores using within-subjects, repeated measures ANOVA with syllabus condition (pre-consultation, post-consultation) as a fixed factor. We applied a general linear model to faculty client data and the first syllabus registry comparison group (n=32 matched course syllabi). Independent variables included syllabus condition (pre-consultation, post-consultation, or syllabus registry Comparison Group 1) as a fixed factor and unique faculty identity (dummy coded) as random effects variables. The faculty identity variables account for repeated measures across pre- and post-consultation syllabi for clients. Because we identified only 10 syllabus registry matched pairs meeting selection criteria (Comparison Group 2), we deemed this sample too small for a reliable statistical test. Instead, we include that data below as an observational comparison.

To determine the extent to which syllabus scores and rates of change differed between faculty or FFP clients (Research Question 2) as well as the influence of CTL seminars on the consultation outcomes (Research Question 3.b), we conducted a three-way mixed ANOVA on syllabus rubric scores. Independent variables included syllabus condition (pre-consultation, post-consultation) as a repeated measures factor as well as client type (faculty, FFP) and prior seminar attendance (yes, no) as between-subjects factors. We included all possible two-way and three-way interactions in the model. The syllabus condition×client type interaction term directly tests the null hypothesis of Research Question 2—that is, impacts of consultations on syllabi are independent of the type of client. The syllabus condition × seminar attendance interaction term directly tests the null hypothesis of Research Question 3.b—that is, impacts of consultations on syllabi are independent of whether clients previously attended a CTL seminar on C&SD. The three-way interaction of syllabus condition×client type×seminar attendance directly tests whether impacts of consultations are independent of both client type and previous seminar attendance.

To measure the influence of CTL workshops on the initial condition of clients’ syllabi (Research Question 3.a), we conducted a two-way ANOVA on pre-consultation syllabus scores. Independent variables included client type (faculty, FFP) and previous seminar attendance (yes, no) as between-subjects factors.

In all statistical analyses, we checked model assumptions by evaluating boxplots of residuals to identify outliers, the Shapiro-Wilk test and Q-Q plots of residuals to verify normality, Levene’s test for homogeneity of variance, and Box’s M test for homogeneity of covariances. We did not identify outliers or violations of homogeneity of variance or covariance in our data sets. We only detected violations of normality in the post-consultation syllabus scores in the three-way ANOVA. We proceeded with the ANOVA because this type of analysis is robust to violations of normality (Norman, 2010).

Overall changes in syllabus scores following consultations resulted in strong effect sizes for both faculty and non-faculty clients (Tables 4 and 5), representing increases of 0.73 and 1.04 standard deviations, respectively. However, changes in syllabus scores following consultations differed between faculty and FFP clients. The three-way mixed ANOVA on all client syllabus scores did not exhibit a statistically significant three-way interaction among syllabus condition (pre-consultation, post-consultation), client type (faculty, FFP), or previous seminar attendance (yes, no), F1,90<0.0, p=.995. However, this ANOVA revealed a statistically significant interaction between client type (faculty, FFP) and syllabus condition (pre-consultation, post-consultation), F1,92=5.109, p=.026, partial n2=.053 (Figure 2). While pre-consultation syllabus scores did not differ between client types (FFP M=.80, SD=.09; faculty M=.81, SD=.10), FFP clients received higher post-consultation scores (M=.90, SD=.07) than faculty clients (M=.87, SD=.08). While total syllabus scores improved for both types of clients, FFP clients improved more than faculty (10% vs. 6%, respectively). When calculated as a relative growth rate, F−I÷Max−I where F = final syllabus score, I = initial syllabus score, and Max = maximum possible syllabus score, FFP and faculty clients demonstrated 50% and 31% gains, respectively, relative to possible growth. Faculty clients’ growth was driven primarily by categories related to assessments and policies (Table 4), whereas FFP clients’ scores increased in almost every category (Table 5).

Seminar attendance prior to a consultation did not impact clients’ initial syllabus scores. The two-way ANOVA testing the influence of seminar attendance on clients’ pre-consultation syllabus scores found no significant interaction between prior attendance at a C&SD seminar and client type (faculty, FFP) on pre-consultation scores F1,90=.23, p=.63 (Figure 3). Additionally, there were no significant main effects for either prior seminar attendance or client type on pre-consultation syllabus scores. Similarly, as reported above, prior seminar attendance did not explain the clients’ changes in syllabus scores, either within or across client types (i.e., no statistically significant main effects or two- or three-way interactions including seminar condition).

Our approach to formative assessment is motivated by our collective desire to iteratively refine our programs and services, informed by data from direct measures of outcomes. The data collected are not used to evaluate the performance of individual consultants. Instead, the primary purpose is to foster a team-based, growth mindset and data-informed collaborative search for areas for improvement. To establish and maintain buy-in as a team, CTL leadership sought consensus and staff input at every step, from rubric development to study design to data collection, analysis, and interpretation. Both rubric norming and data debrief sessions led to rich exchanges of consulting techniques and experiences as well as discussions about the implementation of service models.

To illustrate, the data from this study instigated deliberations on several changes to one or both service models. Foremost, discussions of the study rubric led to changes in how FFP consultations are scaffolded. The revised rubric more clearly aligns with our evidence-based objectives and priorities for clients’ development. We made similar minor changes to our generative interview protocol often used in FFP consultations. The limitations of our data source (syllabi) inspired revisions to the course design reflection memo that is part of the required FFP C&SD project. We clarified prompt questions to foster clients’ reflections on instructional methods and their alignment with learning objectives, two important components of course design that are rarely captured fully in a syllabus. In future formative assessments, this additional deliverable for FFP clients can complement the data available in syllabi. Furthermore, we reevaluated when to provide the rubric during the FFP service model to best support clients and mirror what we recommend to instructors in general (i.e., provide the rubric in advance, rather than after receiving the client’s first draft, as standard practice).

The difference in faculty and FFP clients’ gains suggests that components of the FFP service model may be beneficial to incorporate into the faculty consultations. In particular, we discussed using the rubric or other resources to scaffold faculty consultations. Time constraints and/or competing priorities often prevent holistic feedback and iteration on faculty syllabi. Previously, consultants varied in what resources they used to support faculty clients. Some employed recommendations from our Faculty Senate regarding suggested syllabus components (Eckhardt, 2017). None reported leveraging the FFP rubric with faculty. Increasing the scaffolding in faculty consultations may enhance faculty clients’ gains. However, increasing the focus of faculty consultations on particular growth areas highlighted in our data (e.g., assessments) may also be advantageous for “meeting clients where they are” and impacting C&SD, given the constraints discussed above.

Without this project, these team-wide conversations likely would not have happened, at least not in the same inclusive, systematic, data-driven way. Conspicuous benefits of the data-driven approach to reflective practice presented here include the ability to (a) more precisely measure the impact of a CTL service on instructors’ course design practices; (b) leverage the power of group collaboration to generate data-informed, actionable steps to iteratively enhance a consultation service model; and (c) normalize dialogue across CTL staff members to share and enhance consultation practices and techniques. We acknowledge that this model requires both time and resources and may need to be adjusted to be feasible in other CTL contexts. Nevertheless, we believe the potential benefits merit consideration as one option for formatively evaluating CTL services.

When evaluating CTL services using direct measures of outcomes, it can be difficult to attribute causality without a comparison group. Without comparison groups, outcomes assessments document “what is/happens” but not “what causes.” And, as with our FFP client sample, finding a comparison group is non-trivial. Even without a comparison group, pre/post measurements can provide rich, evidence-based fodder for reflection and iterative refinement of CTL practices. However, one must interpret data cautiously regarding causality. Given these challenges, we advise investigating multiple data sources and/or outcomes measures to demonstrate the added value of CTL services.

What if a CTL has limited resources, including staff with assessment expertise? The approaches reported here would likely provide actionable, formative data with a smaller sample or when only using descriptive statistics or qualitative methods rather than parametric statistical analyses. Using study designs with large samples and inferential statistics to evaluate CTL impacts is rigorous. However, effective, data-driven, reflective practice is not limited to only that particular analytical “way of knowing.” We recommend starting small with direct outcomes assessments and adopting a formative lens, if possible. CTLs can prioritize efforts by asking two questions: What would you most like to know about client outcomes that you don’t? And what would you be willing to change in response to that data, if you had it?

Our formative assessments using syllabus analyses greatly informed our educational development practices regarding C&SD. Consequently, our reflective dialogues also explored other sustainable options for measuring impacts of CTL C&SD services that provide insights unavailable via syllabus analyses.

Classroom observations can provide systematic, objective data absent from syllabi (Campbell et al., 2019; Smith et al., 2013; Stanny et al., 2015; Wheeler & Bach, 2021), especially regarding classroom teaching strategies. For instance, observation data can directly measure frequency, duration, and timing of particular teaching strategies (e.g., active learning, inclusive teaching techniques) during synchronous course sessions. Moreover, trained observers can document fine-grained teaching and learning behaviors, including patterns of participation, the nature of instructor-student interactions, how feedback is delivered, student time on task, and more. Observational data is rich and actionable. However, to measure CTL outcomes, especially at scale, observations are a logistically daunting commitment when resources are limited, even with efficient and reliable observation protocols.

Analyses of other teaching artifacts, above and beyond syllabi, may provide complementary and less resource-intensive alternatives. Collecting assignment prompts and/or grading rubrics from clients before and after participating in CTL services could measure alignment of assessments with learning objectives in syllabi. A similar approach could be taken for analyzing the impacts of CTL services targeting the adoption of multimedia learning principles (Clark & Mayer, 2016) in the design of online learning modules, instructional videos, or lecture slides. Likewise, one could analyze exam item construction (Parkes & Zimmaro, 2016; Piontek, 2008), exam item analytics (Livingston, 2006), or rubric construction before and after CTL assessment design services. Regarding inclusive teaching, one could measure patterns in the representation of authors, perspectives, or ways of knowing in assigned course readings or materials. For all of the examples above, one could potentially develop evidence-based rubrics for formative outcomes assessment. To our knowledge, no SoED studies have included these data sources in CTL outcomes assessments. We hope future research will explore these possibilities and more.