Case Study 1: Transparency in Learning and Teaching in Higher Education (TILT Higher Ed)

Transparency in Learning and Teaching in Higher Education (TILT Higher Ed) was initiated in 2008 by Mary-Ann Winkelmes, who is currently the executive director of the Center for Teaching and Learning at Brandeis University. The research project developed quickly into a collaborative national study between Winkelmes and the American Association of Colleges and Universities (AAC&U) in 2014–2015. TILT is now a national and international collaborative project that is available to all instructors who sign up through the TILT website to participate in the project. Faculty can participate in TILT either by entering their student data into the TILT database (which provides them with feedback) or by applying for access to the database for research purposes under the current institutional review board (IRB). Winkelmes intentionally expanded the scope of TILT beyond a single institution in an effort to model a pathway toward creating larger, more robust data sets related to teaching and learning in higher education.

The basis of transparency work is the engagement of the faculty/ instructor in aligning, and communicating, the purpose, task, and criteria of course assignments and activities before students start working on them (Winkelmes, 2019). TILT is an evidence-based framework that has been proven to improve students’ learning experience as well as institutional retention. The results of multiple TILT studies have confirmed that “students who received more transparent instruction” reported significantly larger gains in three areas that are important predictors of student success: (a) academic confidence, (b) sense of belonging, and (c) awareness of their mastery of the skills that employers value most when hiring (Winkelmes, 2019, p. 8). This finding demonstrates how we can transform higher education pedagogy to make our teaching and learning interactions more equitable.

The Teaching and Learning Center (TLC) at University of Tennessee, Knoxville joined TILT in 2015 to promote evidence-based teaching strategies on campus. TILT offered a clear structure of the research process as well as confidential individual reports to the instructors about their students’ learning experience at the beginning and the end of the semester. According to Olsen et al. (2019, pp. 119–120), the TLC brought training and survey tools from the TILT project to develop a program called Teaching for IMPACT that incorporated key elements of TILT’s Transparent Assignment framework (purpose, task, and criteria):

• IMportance of the task (e.g., test, project, paper) that demonstrates student learning;

• Purpose and the Activities supporting that purpose, connecting the task to the course and unit goals;

• Criteria for task assessment, such as a rubric or criteria sheet; and

• the Task itself, wherever it fits in the course structure, with instruction of course content and related activities leading up to the task required of students to demonstrate learning.

The IMPACT project was implemented at University of Tennessee, Knoxville in 2017 and 2018 following a limited pilot study in 2015. The IRB for the study authorized coinvestigators from the TLC and TILT to access and work with student data and add them to the TILT database. Furthermore, TLC staff requested additional IRB approval to access participating students’ grades so that they could examine the impact of using transparent teaching strategies on students’ retention rates.

In order to encourage University of Tennessee, Knoxville faculty and graduate students to participate in this research project, a call for proposals was released, and participants were selected from the pool of instructors who had applied for the small incentive funding. The participants were then asked to attend a workshop in which TLC staff discussed evidence-based strategies to make their courses more transparent. The next step for the instructors was to give the pre-survey to their students, use the strategies to make a given course unit more transparent, and then administer to students the post-survey that asked about students’ learning experience. After instructors completed the process, they received an individual report about their students’ learning experience and had a follow-up meeting with TLC staff to discuss how the transparent strategies that they used impacted their students’ learning experiences.

The instructors showed interest in continuing to use transparent teaching strategies in their practice and recommended broadening the calls for participation on campus. Furthermore, according to Olsen et al. (2019), “a pre- and post-knowledge test for instructors, about the concepts related to transparency, would yield helpful information and could increase instructor metacognition” (p. 130). The last iteration of the IMPACT project at University of Tennessee, Knoxville was in 2018, and the institution has not pursued this study since that time.

Case Study 2: ManyClasses

Originating from an educational psychology laboratory at Indiana University, ManyClasses is a research model that explicitly acknowledges the limitations of current research in teaching and learning, especially single-classroom studies, and suggests that in the absence of an experimental model, stronger conclusions can be drawn if researchers can measure an experimental variable across multiple independent research contexts (i.e., classes). For its first iteration, the project focused on measuring the effect of differences in feedback turnaround times—that is, whether it makes a difference if feedback on student assignments is given immediately or delayed. While the project began by focusing on these feedback modalities, the project team plans to launch further studies that will integrate a wide range of experimental variables and include the impact of different versions of assignments, assessment methods, active learning techniques, and more. Fyfe et al. (2021) reiterate this significance, stating that each class in this ManyClasses study represents a “micro-experiment” that allows the research team to examine the “generalizable effect size of a manipulation beyond each individual classroom implementation” (p. 3).

The ManyClasses team has given careful consideration to the logistics of gathering data at this level of scale. They have partnered with a third party, the Unizin Data Platform (UDP), a multi-institutional data clearinghouse collaborative, to facilitate the collection of data directly from learning management system (LMS) platforms connected to participating and consenting class sites. They also developed an LMS-embedded widget that encodes consent, which has proven to increase student participation rates in the online interventions, which are managed by the ManyClasses team directly. Participating faculty need only to agree to participate and (possibly) provide any appropriate credit to students.

At Penn State, the Schreyer Institute for Teaching Excellence chose to participate in the first iteration of the ManyClasses project, which took place in Fall 2019, an initiative that fit well with a recently approved 5-year SoTL strategic plan that originated in one of the central CTLs. Because the institution was not yet a full member of Unizin, this participation required a special Family Educational Rights and Privacy Act (FERPA) exemption that would allow the Indiana University researchers to have access to student responses within the Canvas course shells. In this case, a faculty member in the CTL served as the liaison between the ManyClasses team and the registrar’s office, which approves FERPA exceptions. After several rounds of intensive negotiations, a custom solution was found that passed muster with the registrar, the IRB office, and the office of research protection.

The same CTL consultant then chose to send multiple communications calling for volunteers, netting eight faculty, 12 courses, and approximately 500 students from a range of levels and disciplines. The recruitment letter emphasized both low-stakes involvement and comparatively high rates of return—participating faculty were offered co-authorship credit. Although the resulting paper will feature a long list of co-authors, this incentive proved to be critical to the success of the project, as the majority of faculty participants were full-time, non-tenure-track teaching faculty for whom the institution had recently created a ranking and promotion system that explicitly rewards SoTL work, often without specific distinctions for level of contribution. After the first full semester of participation (Fall 2019), the faculty reported minor annoyances with how the Canvas assessments were implemented, but otherwise the project team handled the majority of logistics without incident. The circumstances of the global pandemic slowed down publication of the first paper until June 2021, but Penn State had already formally agreed to participate in the second iteration of the project, tentatively scheduled for Spring 2022.

Challenges and Opportunities

These two cases of MegaSoTL suggest a shifting of roles for educational developers and CTL staff (Hanson, 2013). Rather than, or in addition to, the provision of a slate of conventional SoTL support services, such as fellowships, grants, workshops, and related programming, we can also play a different role in enabling faculty to connect to MegaSoTL projects. Our primary role becomes one of connectors, using our external networks to identify opportunities for MegaSoTL work and then drawing upon our internal, campus-based connections to bring faculty to the table. Subsequent to that, we also serve as ground agents, anchoring the project in the culture of the institution; guide posts, shepherding faculty through the project pipeline; and amplifiers, ensuring that faculty participation is recognized and, hopefully, rewarded appropriately.

Recognition and reward for MegaSoTL projects like these work a bit differently from other forms of engagement with SoTL. From the perspective of faculty participants, these projects often sit somewhere along the hazy border between scholarly teaching and the scholarship of teaching and learning. Most projects utilize crowdsourcing as a means for offering low-stakes engagement, with potential benefits for classroom-level instruction, but publication credit is either not offered (TILT) or offered at diminished contribution (ManyClasses). The authors’ experiences with other MegaSoTL projects has tended toward the former, so it has become increasingly incumbent on the educational developer to negotiate additional benefits that could swing the perceived value equation and sweeten the participation pot. With another project, for example, the outside researchers were not offering co-authorship, but they did agree to provide a tailored webinar for faculty participants that included a close conversation with the lead researcher, a person whose status could perhaps justly be described as a “rock star” in the field of educational psychology.

A further challenge with low-stakes engagement is that faculty can also feel less invested in the project’s success, so educational developers may be tasked not just with finding ways to recruit faculty but also with motivating them to persist through the project. With IMPACT, the CTL provided a monetary incentive, and with ManyClasses, the research team encouraged participants with multiple tokens of appreciation, including buttons, digital badges, and thank-you cards. In both cases, the campus point person also was tasked with providing regular updates and progress reports. Educational developers at both institutions worked closely with university communications to ensure that faculty participation received coverage in the campus press. In the case of ManyClasses, for example, this coverage included a feature story on the project as a whole as well as related stories about individual faculty members.

Those news stories also included mention of the role of the CTL, of course, so the good publicity was shared by multiple parties, but this does raise the question of recognition and reward for the work of the educational developers. In our shared experiences, the role of ground agent can be a breeze, as it largely involves providing a conduit of communication between institutions, but it can also constitute a more significant investment of time, particularly when working across the varied standards held by IRBs. Both of our case studies involved projects that had IRB approval at their originating institutions, but they also involved the negotiation of separate, at times contested, IRBs at the local campuses (Stockley & Balkwill, 2013). In some cases, educational developers may need to collect local data, manage de-identification processes, and offer other research support services that may or may not be in their regular job descriptions. The additional time spent may be justified through higher participation numbers, but evidence of direct impact may be in shorter supply. This state of affairs is symptomatic of larger conversations taking place around assessment and organizational development. It should be noted that in some cases, it may be possible for educational developers to negotiate co-authorship for themselves, an inquiry that has worked at least twice for the authors of this article.

Discussion and Implications

There have been an increasing number of calls for educational developers to imagine different roles for themselves in an age of organizational development. Embracing the promise of this age necessitates an expansion of both the scale and scope for our work; it also suggests that we will need to identify modalities that foster educational development and/or teaching transformation without their having to be implemented directly through the CTL or by its staff, as these resources tend to be more finite (Stark & Smith, 2016). In other words, we may need to think of ourselves less as providers of direct services and more as levers of change. Support for MegaSoTL projects falls into this latter category, as we have found ourselves in the role of connectors more often than in the role of coaches or creators.

That is not to say that we as educational developers cannot serve as the originators of MegaSoTL projects, as was the case with TILT, but we are not limited to engaging only in those projects that we lead. Rather, our role(s) in MegaSoTL can serve to ensure that such projects are in keeping with the signature pedagogy model, providing an additional avenue through which faculty/instructors can navigate the SoTL ecosystem. It could be argued that by lowering a number of barriers to entry, crowdsourced projects like these may serve to increase both access and inclusion in the work of SoTL, particularly for those for whom time, data access, and/or data analysis may be significant barriers (Boshier, 2009; Nelson Laird & Ribera, 2011; Walker et al., 2008). At our respective institutions (both large, public institutions with very high research activity), this would seem to be the case, as the number of faculty participating in various MegaSoTL projects has risen precipitously. It could be an interesting follow-up study to survey participating faculty regarding their perceptions of this phenomenon and their role within it.

That said, because these initial MegaSoTL projects have been informed by strongly positivist, social science methods, there is a risk that they could also be creating additional barriers for faculty who are not in disciplines related to the social sciences. In this case, MegaSoTL may be exacerbating a long-standing thorn in SoTL’s side. Several meta-analyses of published SoTL work both highlight and, at times, lament the predominance of social science methods in the field (Hubball et al., 2010; Miller-Young & Yeo, 2015; Potter & Wuetherick, 2015). Leading advocates have sought to frame SoTL as a “bigger tent” by, for example, bringing together a multitude of disciplinary and methodological approaches. The “big tent” serves not only to foster broader access but also to bring a diversity of perspectives to bear on the often complex and multifaceted practices of teaching and learning (Chick & Poole, 2013). Efforts continue to extend the tent poles and to bring scholarly practice in closer alignment to the inclusive values of both fields—SoTL and educational development (Friberg & Scharff, 2020). MegaSoTL projects, however, can allow low-stakes faculty participation without necessitating a mastery of social science research methods or the availability of experienced data analysts. And while the proliferation of MegaSoTL projects is currently being driven by the need for numbers, there is nothing to prevent future large-scale teaching and learning projects that consist predominantly of non-quantitative analysis, whether of texts, images, or other related artifacts.

To facilitate those future projects, we suggest building the MegaSoTL infrastructure, starting with developing a set of shared practices for educational developers, a conversation this article is intended to start. There is an especially pressing need for collaborative guidance in the context of working across IRBs, which remains a struggle for researchers, participants, and educational developers alike. On a related note, the idea of a global clearinghouse for multi-institutional projects, in which researchers could post opportunities and data could be (voluntarily) shared, would likely be a boon to teaching transformation and educational research alike. As Bass (2020) notes, there is a growing need for convergent (i.e., interdisciplinary, even multi-institutional) research on the incredibly complex process of learning, and it will take all of us thinking, even dreaming, bigger to tame this wicked beast.