Digital Repository User Studies

There is a substantial amount of work published on UX and usability assessment, collectively known as user studies, of digital library collections and repositories in the library and information science literature. Chapman et al. (2015) surveyed the state of user studies in digital library literature, covering articles published in 2010–2014. However, they did not include articles on solely scholarly repositories such as institutional, disciplinary, and research data repositories. Therefore, the literature review in this paper emphasizes work that has been published since Chapman et al., as well as key publications on user studies of scholarly repositories in overlapping years.

Digital Library Collections Usability Studies

Most of the writing in usability studies of digital repositories has focused on digital libraries hosted by academic libraries and government-sponsored cultural heritage organizations. Research on the UX of digital library collections and repositories’ search and discovery is a growing area of interest, with emphasis on exploratory and controlled vocabulary search (Bernard et al., 2015; Gaona-García et al., 2017; García et al., 2014; Goodale et al., 2014; Walker & Halvey, 2017) and data visualization-based discovery (Gaona-García et al., 2018). More research is needed, however, in search and discovery within digital library collections of aggregated metadata (Matusiak, 2017; Zavalina, 2014).

Accessibility and Bias Testing

Accessibility is a factor of usability, but there has been little in the literature about accessibility testing of library technology and electronic resources in general (Fernandez, 2018; Michalak & Rysavy, 2020; Rysavy & Michalak, 2020), with equally little devoted to accessibility in digital repositories and collections. Babu and Xie (2017) identified issues in digital library design that inhibited the use of these systems by blind and visually impaired users. Xie et al. (2020) built on this work by implementing and testing help features tailored specifically to the needs of blind and visually impaired users. Exciting new research has begun to develop methods of critically evaluating systems for bias (Cunningham et al., 2016) and inclusion and equity (Stobbs et al., 2018).

Institutional Repository Usability Studies

Most usability studies of IRs in the literature are about repository services provided by academic libraries (Hee & Ho, 2008; Koshiyama et al., 2015; McKay & Burriss, 2008; Zhang et al., 2013). Caccialupi et al. (2009) described a usability study on an instance of DSpace hosted by the University Milano-Bicocca’s Multimedia Production Centre. Although they mentioned the importance and scope of IRs, they used their DSpace instance as an internal digital asset management system. However, their results echoed findings of usability studies of traditional DSpace implementations, such as difficulty browsing within communities and collections, unclear terminology and mandatory fields in the submission interface, and frustration with the lack of authority control (2009).

Zhang et al. (2013) undertook a usability study of a system featuring a digital repository as well as bibliography and research collaboration features. They found that users had no problems with the collaboration and searching aspects of the system. However, like findings of usability studies of traditional IRs, there was noted confusion when depositing materials or adding information to the bibliography, as the submitters didn’t understand why they were being required to enter descriptive metadata multiple times, and the deposit interface was unintuitive.

Betz and Hall (2015) designed user testing for their IR’s self-archiving workflow based on “microinteraction theory”, which allowed them to focus on highly specific aspects of the deposit process. Some of their findings are typical for IR user studies; for example, free-text input was confusing for depositors, and authority control was a problem. Other findings were more specific; for example, they found that the optional field to assign a Creative Commons License was confusing to their depositors.

Development of New Institutional Repository Programs

Most user needs assessments for institutional repositories are about informing planning and implementation of new institutional repository services. Maness et al. (2008) described how they employed user personas, which are profiles of fictional people that represent a particular user group. They found that the features users need for an IR usually contradicted their assumptions. Specifically, they found that researchers want a tool more akin to researcher information or course management systems rather than a traditional institutional repository. Foster and Gibbons (2005) discussed the findings of a grant project examining faculty work practices and how the IR might naturally support them. They found that faculty wanted to do their research, publish, control access to it, make their publications discoverable, and that they did not want to do additional work.

Green et al. (2007) also undertook a survey and interviews to inform development of a workflow and metadata tool called RepoMMan, which was being built as part of the University of Hull’s Fedora repository. Their faculty expressed interest in a tool that would allow them to work on in-progress research without having to carry around storage media, essentially describing cloud storage services such as Box or Google Drive before they became widespread. Similarly, Seaman (2011) found, “If librarians think of IRs in terms of access, scholarly communication, and institutional promotion, then these interviews underscore the fact that faculty members think in terms of storage, services, and the marketing of self” (Conclusion paragraph 2).

Next Generation Repositories

A couple of studies have put forth broad visions of how the field of scholarly repository technologies should move forward from siloed individual repositories to a more distributed, interoperable, and equitable network of repositories. In 2017, the Confederation of Open Access Repositories’ Next Generation Repositories Working Group (NGRWG) put out a report outlining eleven new behaviors, that the next generation of scholarly repositories should adopt (NGRWG, 2017). The report also made recommendations for technical solutions that would facilitate these behaviors. They acknowledged, however, that these [technologies] were “a snapshot of the current status of technology, standards and protocols available, but we are aware that technologies will continue to evolve” (p.14).

In another study, Shen (2019) approached the future of digital libraries through the lens of “intelligent infrastructure for human-centered communities” or smart infrastructure. Informed by interviews with researchers of smart technologies, Shen encouraged libraries to fully embrace smart technologies. According to NetLingo, “The term ‘smart’ originally comes from the acronym ‘Self-Monitoring, Analysis and Reporting Technology’ but become[sic] widely known as ‘smart’ because of the notion of allowing previously inanimate objects—from cars to basketballs to clothes—to talk back to us and even guide our behavior” (NetLingo, n.d.).

Digital Library Migration

Stein and Thompson (2015) wrote about the results of a survey of digital library managers who were planning to migrate repository systems or had already migrated to discover their motivations for doing so. Although this article excluded scholarly repositories from its scope, one of the findings was that when institutions migrated from one software platform to another, the trend was to consolidate multiple repositories services, e.g., digital library, institutional repository, and data repository, into a single unified repository. Only recently have we begun to see writing on redesigns and migration of existing IRs to next-generation platforms (Bell & Sarr, 2010; Luca & Narayan, 2016) and iterative improvements based on usability testing (González-Pérez & Ramirez, 2019).

User Groups and Roles

As noted in Stein et al. (2017), little research in this area has considered user groups besides faculty and researcher depositors (McKay, 2007; Russel & Day, 2010). In the context of submitting tickets to library technology help, which is applicable to the administration of IRs, Hoffman (2020) called for using UX data-gathering techniques and strategies to shape internal services for our co-workers, just as we do for our public-facing services. Our project builds on the existing literature by interviewing and hosting focus groups for three distinct user groups to inform the development of, and migration to, a new institutional repository system.

A key reason for considering different user groups is they may express unique needs, and there may be resonances and conflicts between or within groups. These differences may stem from competing positions that shape assumptions of how particular sociotechnical systems may work. Some early work such as studies by Rieger (2008) and Guedon (2009) focused on the way power—in the sense of the ability of social actors to exert influence and shape outcomes—and sociotechnical assumptions shape the implementation of IRs and their positioning in the scholarly ecosystem. The intermediary group we discuss here relies in part on other librarians, such as subject specialists, who, as Salo (2009) explored as part of her analysis of the obstacles facing repositories, may have varying interest in, or perhaps outright hostility toward, institutional repositories. Her work is suggestive of how the various interpretive lenses that different stakeholders bring to technical solutions such as repositories can tie to organizational politics within the library.

Organizational politics, as used here, relates to the perception of self-interested ends and use of power by individuals or groups in competition with others within an organization. See, for example, Hochwarter et. al (2010) and Rosen et. al (2014) for discussions in studies of organizational behavior. Our study considers whether differences between the expressed needs and user behaviors of different user groups illuminate different issues related to power and political position of those groups vis-à-vis the repository.

Spotlight on Intermediaries

Themes unique to intermediaries (Appendix C) tended to be about ways the IR could better help these individuals manage their communities. For example, provision of high-level statistics information could be used to communicate impact of works in IDEALS. Intermediaries also expressed a need for more training as they were not aware of everything they could do as community administrators. Although intermediaries were not always sure what their permissions did entail, they were well aware of what they could not do. They described this awareness in discussions of batch ingests. The vast majority of content in IDEALS comes from batch ingests, which a web developer must upload into the repository via the server. Intermediaries who regularly curate content batches for IDEALS wanted to be able to perform batch ingests themselves, with appropriate training, rather than go through the current process, in which the IR management team reviews all the metadata for adherence to the metadata policy and best practices (Stein et al., 2017). While this request is not unreasonable, it is at odds with existing practice designed to account for the fact that the vast majority of metadata errors in IDEALS are introduced via batch ingests.

What is most significant about the intermediaries’ feedback is that it illuminates internal politics both within the library as well as on campus. Somewhat contrary to Salo’s (2009) findings, the intermediaries, including other subject specialists, were invested enough in IDEALS to want control over relevant communities and collections and thus a more decentralized library management model that would allow them to work directly with their communities rather than having to negotiate with the repository services team. To some extent this brings IDEALS into a familiar push-pull for centralized control that we’ve seen locally in other library services such as cataloging and reference.

Similarly, frustration from intermediaries over certain facets of information organization in IDEALS reflect a sense among some members of that group that subject specialists are better positioned to know their departments. For instance, DSpace provides a list of top-level communities. In IDEALS, this list is generally organized to reflect the organizational structure of the university. Intermediaries identified several communities for schools and departments they felt were incorrectly placed as sub-communities of colleges or other departments on campus or were correctly placed but historically the subject of complaints from faculty that an electronic thesis or dissertation was “incorrectly cataloged.” There are also top-level communities for recurring campus conferences that they felt should not be moved under the umbrella of any specific campus community because no one department could claim ownership. This boundary-drawing, while certainly reflecting an interest shared among repository developers and the intermediaries to correctly represent the university, is also about who owns what: that is, not just what departments, but which intermediaries should be able to control particular content.

We did not receive the same level of pushback or skepticism regarding the purpose of the service as was reported by early user studies of IRs, although one user did wonder why they should deposit into IDEALS instead of a disciplinary repository or pre-print server such as arXiv. However, this is likely due to recruiting that prioritized active intermediaries, people who had deposited work in IDEALS, or those who happened to be actively using it at the time of the study.

Known Usability Issues

Many of the findings from this study have been known issues for quite some time and were not a surprise. For example, the IDEALS website needs to adhere to Web Content Accessibility Guidelines 2.1; the login button is difficult to find; the lack of authority control for author and contributor names, as well as campus units, is frustrating; depositors cannot edit previously submitted items; consumers and depositors do not know what “communities” are; intermediaries frequently mix-up communities and collections; search results are often excessive and irrelevant; and, much like most online terms of service, no one reads the deposit agreement.

New Findings

The study did unearth some surprising findings, most of which pertained to anxiety about creating or interpreting metadata. For example, every submission is required to have at least one depositor-generated keyword, which has been stored in the Dublin Core element “dc.subject.” The display label for this field is “Subject” and the corresponding “Browse by” category is also called “Subject.” Although terms from controlled vocabularies can be found as values in this field—mostly due to repurposed catalog metadata—they function as keywords, not a controlled vocabulary. This conflation of keywords and subject terms turns out to be confusing for depositors and consumers. In particular, depositors feel anxious about “assigning subject terms” when they are only creating keywords. Similarly, depositors and consumers do not understand the difference between “authors” and “contributors.” Additionally, depositors said they were uncertain what a complete record looked like or what data entry rules there are for metadata.

Other findings unrelated to metadata also surprised us: despite IDEALS’ community structure reflecting campus departmental hierarchies, it is unclear to depositors which collections they can submit materials to and why; and consumers want to be able to browse image-based collections as a gallery rather than as a list of items.

Recommendations

The recommendations generated for internal use reveal tension between the needs and desires of the three user groups and the IDEALS management team that raise issues of internal university politics similar to those identified in the analysis of codes and themes. Certainly, many of the themes led to improvements we want to implement, or successes we want to take advantage of in workflows, interfaces, and outreach and engagement.

Other themes, though, led to suggestions that gave us pause. For some of these, usually in cases related to policy where the implications for changing features could pose challenges, our recommendation for a next step was to “consider” a set of actions. For example, “[consider] widening permissions levels for batch editing to additional groups of intermediaries beyond the core IR team.” These recommendations are unlikely to result in immediate implementation but may become part of the ongoing development of IDEALS, or consideration may lead to a different outcome than what users recommended.

Some recommendations, though, were simply to take no action in response to a suggestion that arose in a theme because the suggestion would violate policy or law. For example, providing more detailed information about consumer traffic to intermediaries could violate the library’s privacy policy—although some consideration of improved usage reporting is warranted.

Power, Politics, and UX

The tensions over control and permissions that arose in the intermediary discussions, as well as the user requests for features that run against what is possible due to policy and law, are important reminders that UX is tied to questions of power and politics. It can be easy for studies such as ours to imagine that systems and interfaces are just a matter of improvement through user testing or meeting compliance standards. But UX cornerstones like “usefulness” cannot be separated from who the various user groups are, the relationships between the user groups, and that they are made up of human beings. Issues of permissions for use of advanced features and of information organization in a system of campus units are tied to questions of who controls collections. Most notably, the balance of subject specialist control over their IR communities with central repository management control ensuring consistency with policy and quality standards reflects long-standing internal politics of the library.

Additionally, repository administrators may need to balance some expressed user needs against legal or policy restrictions. For example, one user requested that we provide more information on consumers of repository content, including IP addresses. Tracking and providing access to this level of user information would violate the library’s privacy policy.

Alignment with Visions of Next Generation Repositories

Our findings align with nine of eleven areas emphasized in the Confederation of Open Access Repositories’ report (NGRWG, 2017) on next-generation repositories to varying degrees: Exposing Identifiers, Declaring Licenses at the Resource Level, Interacting with Resources (Annotation, Commentary, and Review), Collecting and Exposing Activities, Batch Discovery, Identification of Users, Authorization of Users, Exposing Standardized Usage Metrics, and Preserving Resources. The two remaining areas none of our findings aligned with are gaps: Discovery of Navigation and Resource Transfer.

Discovery of Navigation is a particularly interesting behavior because it essentially boils down to thinking of machines as users: “While a human user can intelligently move around between these various resources, understanding that they pertain to the same scholarly object, a machine cannot” (NGRWG, 2017, p. 17). The nature of our methods, interviews and focus groups, do not lend themselves to categorizing machines as users. This is a significant limitation in the study design since “machine users” have consequential implications for accessibility (Arlitsch, 2018).

Resource Transfer entails “supporting by value content transfer of their resources to support text/data mining and preservation applications. By value content transfer entails allowing third parties to efficiently access and transfer the actual content of scholarly objects” (NGRWG, 2017, p.20). Although features such as supporting text and data mining of IDEALS’ content had come up during informal conversations previously, they did not arise in this study.

Overall, it is encouraging that the needs of IDEALS’ users seem to align with the confederation’s vision for the future of scholarly repositories. However, this is not the case when compared to other work in this area. As mentioned previously, based on interviews with experts in smart technologies, Shen (2019) called for libraries to fully embrace smart technologies and artificial intelligence. In our study, no participant brought up smart infrastructure and technologies; recommender systems were mentioned once, but in the relatively limited capacity of pointing to other items in the same collection from a single item’s access page. Furthermore, there are significant privacy concerns in regard to smart infrastructure and artificial intelligence that need to be carefully considered and addressed prior to and simultaneously with integrating such systems in libraries.