Introduction

Institutional repositories (IRs) have existed for around two decades (Lynch, 2003). In that time, as human technological capabilities have evolved, so have IR users’ needs; IRs as receptacles of static documents are no longer good enough. To better address new use cases, libraries with IRs are starting to investigate and migrate to new systems, a shift that is in progress at the University of Illinois Urbana-Champaign University Library’s institutional repository: the Illinois Digital Environment for Access to Learning and Scholarship (IDEALS).

In this paper, we use “digital repositories” as an umbrella term for systems that preserve and provide access to digitized and born-digital content from cultural heritage, research, and academic institutions. Within these bounds, digital repositories are further divided into two broad types: digital libraries and scholarly repositories. Digital libraries may also be called digital collections or digital archives, which typically feature digitized content from special collections libraries, archives, museums, or historical societies but may include born-digital materials. Scholarly repositories include institutional repositories, research data repositories, disciplinary repositories, or pre-print servers such as the famed arXiv.

Typically, institutional repositories collect and provide access to the intellectual output of an academic institution but can be affiliated with other types of organizations. Similarly, disciplinary repositories allow researchers within a certain knowledge domain to deposit research, or sometimes instructional, materials. Finally, data repositories curate and provide access to digital research data and may be institutional or disciplinary in nature. This study focuses on this second category—scholarly repositories.

Research Question

As one of many libraries seeking to migrate IR systems, we decided to undertake a study of our existing IR users in order to inform local system development. We identified three distinct user groups:

  • Consumers: users who are looking for resources in the IR.

  • Depositors: users who submit works to the IR.

  • Intermediaries: users who manage one or more IR communities or facilitate the IR-related activities of the other two groups.

Given these distinctive user roles and advancement in technologies, our study seeks to answer two research questions:

  1. What user needs and desires should drive next-generation repository services?

  2. What are the unique needs of different user groups (consumers, depositors, and intermediaries)?

    What does inclusion of intermediaries bring to our understanding of user experience (UX) in digital repositories in particular?

Literature Review

Digital Repository User Studies

There is a substantial amount of work published on UX and usability assessment, collectively known as user studies, of digital library collections and repositories in the library and information science literature. Chapman et al. (2015) surveyed the state of user studies in digital library literature, covering articles published in 2010–2014. However, they did not include articles on solely scholarly repositories such as institutional, disciplinary, and research data repositories. Therefore, the literature review in this paper emphasizes work that has been published since Chapman et al., as well as key publications on user studies of scholarly repositories in overlapping years.

Digital Library Collections Usability Studies

Most of the writing in usability studies of digital repositories has focused on digital libraries hosted by academic libraries and government-sponsored cultural heritage organizations. Research on the UX of digital library collections and repositories’ search and discovery is a growing area of interest, with emphasis on exploratory and controlled vocabulary search (Bernard et al., 2015; Gaona-García et al., 2017; García et al., 2014; Goodale et al., 2014; Walker & Halvey, 2017) and data visualization-based discovery (Gaona-García et al., 2018). More research is needed, however, in search and discovery within digital library collections of aggregated metadata (Matusiak, 2017; Zavalina, 2014).

Accessibility and Bias Testing

Accessibility is a factor of usability, but there has been little in the literature about accessibility testing of library technology and electronic resources in general (Fernandez, 2018; Michalak & Rysavy, 2020; Rysavy & Michalak, 2020), with equally little devoted to accessibility in digital repositories and collections. Babu and Xie (2017) identified issues in digital library design that inhibited the use of these systems by blind and visually impaired users. Xie et al. (2020) built on this work by implementing and testing help features tailored specifically to the needs of blind and visually impaired users. Exciting new research has begun to develop methods of critically evaluating systems for bias (Cunningham et al., 2016) and inclusion and equity (Stobbs et al., 2018).

Institutional Repository Usability Studies

Most usability studies of IRs in the literature are about repository services provided by academic libraries (Hee & Ho, 2008; Koshiyama et al., 2015; McKay & Burriss, 2008; Zhang et al., 2013). Caccialupi et al. (2009) described a usability study on an instance of DSpace hosted by the University Milano-Bicocca’s Multimedia Production Centre. Although they mentioned the importance and scope of IRs, they used their DSpace instance as an internal digital asset management system. However, their results echoed findings of usability studies of traditional DSpace implementations, such as difficulty browsing within communities and collections, unclear terminology and mandatory fields in the submission interface, and frustration with the lack of authority control (2009).

Zhang et al. (2013) undertook a usability study of a system featuring a digital repository as well as bibliography and research collaboration features. They found that users had no problems with the collaboration and searching aspects of the system. However, like findings of usability studies of traditional IRs, there was noted confusion when depositing materials or adding information to the bibliography, as the submitters didn’t understand why they were being required to enter descriptive metadata multiple times, and the deposit interface was unintuitive.

Betz and Hall (2015) designed user testing for their IR’s self-archiving workflow based on “microinteraction theory”, which allowed them to focus on highly specific aspects of the deposit process. Some of their findings are typical for IR user studies; for example, free-text input was confusing for depositors, and authority control was a problem. Other findings were more specific; for example, they found that the optional field to assign a Creative Commons License was confusing to their depositors.

Development of New Institutional Repository Programs

Most user needs assessments for institutional repositories are about informing planning and implementation of new institutional repository services. Maness et al. (2008) described how they employed user personas, which are profiles of fictional people that represent a particular user group. They found that the features users need for an IR usually contradicted their assumptions. Specifically, they found that researchers want a tool more akin to researcher information or course management systems rather than a traditional institutional repository. Foster and Gibbons (2005) discussed the findings of a grant project examining faculty work practices and how the IR might naturally support them. They found that faculty wanted to do their research, publish, control access to it, make their publications discoverable, and that they did not want to do additional work.

Green et al. (2007) also undertook a survey and interviews to inform development of a workflow and metadata tool called RepoMMan, which was being built as part of the University of Hull’s Fedora repository. Their faculty expressed interest in a tool that would allow them to work on in-progress research without having to carry around storage media, essentially describing cloud storage services such as Box or Google Drive before they became widespread. Similarly, Seaman (2011) found, “If librarians think of IRs in terms of access, scholarly communication, and institutional promotion, then these interviews underscore the fact that faculty members think in terms of storage, services, and the marketing of self” (Conclusion paragraph 2).

Next Generation Repositories

A couple of studies have put forth broad visions of how the field of scholarly repository technologies should move forward from siloed individual repositories to a more distributed, interoperable, and equitable network of repositories. In 2017, the Confederation of Open Access Repositories’ Next Generation Repositories Working Group (NGRWG) put out a report outlining eleven new behaviors, that the next generation of scholarly repositories should adopt (NGRWG, 2017). The report also made recommendations for technical solutions that would facilitate these behaviors. They acknowledged, however, that these [technologies] were “a snapshot of the current status of technology, standards and protocols available, but we are aware that technologies will continue to evolve” (p.14).

In another study, Shen (2019) approached the future of digital libraries through the lens of “intelligent infrastructure for human-centered communities” or smart infrastructure. Informed by interviews with researchers of smart technologies, Shen encouraged libraries to fully embrace smart technologies. According to NetLingo, “The term ‘smart’ originally comes from the acronym ‘Self-Monitoring, Analysis and Reporting Technology’ but become[sic] widely known as ‘smart’ because of the notion of allowing previously inanimate objects—from cars to basketballs to clothes—to talk back to us and even guide our behavior” (NetLingo, n.d.).

Digital Library Migration

Stein and Thompson (2015) wrote about the results of a survey of digital library managers who were planning to migrate repository systems or had already migrated to discover their motivations for doing so. Although this article excluded scholarly repositories from its scope, one of the findings was that when institutions migrated from one software platform to another, the trend was to consolidate multiple repositories services, e.g., digital library, institutional repository, and data repository, into a single unified repository. Only recently have we begun to see writing on redesigns and migration of existing IRs to next-generation platforms (Bell & Sarr, 2010; Luca & Narayan, 2016) and iterative improvements based on usability testing (González-Pérez & Ramirez, 2019).

User Groups and Roles

As noted in Stein et al. (2017), little research in this area has considered user groups besides faculty and researcher depositors (McKay, 2007; Russel & Day, 2010). In the context of submitting tickets to library technology help, which is applicable to the administration of IRs, Hoffman (2020) called for using UX data-gathering techniques and strategies to shape internal services for our co-workers, just as we do for our public-facing services. Our project builds on the existing literature by interviewing and hosting focus groups for three distinct user groups to inform the development of, and migration to, a new institutional repository system.

A key reason for considering different user groups is they may express unique needs, and there may be resonances and conflicts between or within groups. These differences may stem from competing positions that shape assumptions of how particular sociotechnical systems may work. Some early work such as studies by Rieger (2008) and Guedon (2009) focused on the way power—in the sense of the ability of social actors to exert influence and shape outcomes—and sociotechnical assumptions shape the implementation of IRs and their positioning in the scholarly ecosystem. The intermediary group we discuss here relies in part on other librarians, such as subject specialists, who, as Salo (2009) explored as part of her analysis of the obstacles facing repositories, may have varying interest in, or perhaps outright hostility toward, institutional repositories. Her work is suggestive of how the various interpretive lenses that different stakeholders bring to technical solutions such as repositories can tie to organizational politics within the library.

Organizational politics, as used here, relates to the perception of self-interested ends and use of power by individuals or groups in competition with others within an organization. See, for example, Hochwarter et. al (2010) and Rosen et. al (2014) for discussions in studies of organizational behavior. Our study considers whether differences between the expressed needs and user behaviors of different user groups illuminate different issues related to power and political position of those groups vis-à-vis the repository.

Background

We decided to highlight the role of intermediaries specifically because it factors heavily into the service model for the University of Illinois Urbana-Champaign’s institutional repository IDEALS. Until recently, IDEALS was powered by the widely used turnkey institutional repository software DSpace, which has a hierarchical data model that consists of communities, collections, and items. Communities are the top-level entity and may contain child communities—called sub-communities—and/or collections. Collections hold items, which are the records that represent the object being deposited along with corresponding descriptive metadata.

Originally, IDEALS was designed to operate via a distributed management model. That is, every time someone requests a new top-level community, they must name an individual who serves as the contact and administrator for that community. Administrators manage normal operational tasks for a community, such as assigning permissions, creating sub-communities and collections, and reviewing deposits. In IDEALS, community administrators tend to either be representatives from campus departments or subject librarians. Aside from batch ingests and deposits into a general-purpose community, materials are not mediated when deposited into IDEALS unless a community administrator sets up such a workflow.

IDEALS first came online in 2006 charged with collecting, preserving, and providing access to the research and scholarship of the University of Illinois Urbana-Champaign research community. IDEALS has grown substantially over the years to steward over 104,500 items as of May 2021. Envisioned primarily as a document repository, IDEALS contains texts, images, audiovisual files, datasets, software, websites, and other types of content. The vast majority of works in IDEALS are text-based. However, as the library creates new services to support and facilitate the needs of campus researchers, IDEALS has struggled to keep up in ways that are both seemingly simple (supporting personal identifiers such as ORCiD) and complex (displaying and preserving dynamic digital publications). Therefore, the library made the decision to migrate IDEALS from DSpace to a node on the library’s homegrown repository infrastructure (Prom et al., 2019). While we were aware of many unmet needs or desired improvements to the system, we also realized a more systematic understanding of the user experience of scholars using the repository would improve the development of the new system.

Methods

After IRB approval, we recruited participants in three general categories corresponding to roles they might play: depositors of content to the repository, consumers of content already in the repository, and intermediaries who facilitate work with the other two groups to ensure their success. The intermediaries did not include the site-level administrators of the repository but did include administrators of particular repository communities. Community administrators are often library subject specialists and representatives from key campus partners.

In practice, an individual could play more than one, and even all three, roles at different moments, but we developed different interview and focus group protocols (see Appendix A) to ensure we explored issues related to these key roles in depth. We recruited intermediaries through the all-library email list and direct invitations to known individuals who played this role. We recruited depositors and consumers through a banner message on the IDEALS website and direct outreach via email to individuals who had deposited content in the repository (who may or may not be the authors of that content).

Since we were not looking for larger patterns of behavior, we chose interviews and focus groups to explore a range of perspectives from our three user groups in detail and to allow follow-up questions. We interviewed depositors and consumers because we had particular tasks we wanted the “consumer” individuals to complete, and we wanted members of both groups to feel free to speak as non-experts who might not always feel comfortable with the technology. In the case of intermediaries, we were dealing with a relatively small known cohort of individuals who all had expertise in the area but also very different positions in relation to repository work. As a result, we chose focus groups in order to help draw out points of agreement and disagreement among intermediaries and otherwise use the conversation to create a more dynamic back and forth in response to the questions.

We interviewed twelve depositors and twelve consumers and conducted two focus groups that included a total of nine intermediaries. We asked each group questions about experiences with IDEALS’ current interfaces and workflows, what worked well and did not, and what would improve the repository experience. We kept these questions open ended to allow participants to express concerns, frustrations, or ideas they had related to IDEALS. We did not design questions to prompt them to respond to whether specific imagined features might be useful because we wanted to focus on what arose from their user experience and the needs they felt as users of IDEALS without leading them to particular answers. Interviewees received a gift card for their participation, and the focus groups included light refreshments. After a paid graduate student transcribed the interviews, they were added to Atlas.ti for coding.

We initially coded the transcripts in Atlas.ti for broad thematic areas. We began with some anticipated thematic areas (for example, “search process and results,” “browsing,” “item-level display”) but added to and refined them using a grounded coding approach to capture unanticipated areas of emphasis. A passage could receive more than one code. After a test to normalize the approach, each researcher coded half of the transcripts, and then reviewed and suggested changes to the codes of the other researcher before we jointly discussed and reconciled any lingering discrepancies.

We then extracted the passages for each code into spreadsheets and identified more specific themes, including trends in positive feedback, challenges, or suggestions from users. For each theme that revealed a user challenge or unmet need, we developed a recommendation for internal use in upcoming repository development plans. In cases of positive feedback, we recommended continuing or reemphasizing relevant features or approaches. We also divided the initial work for this second round of coding and reviewed one another’s work to provide suggestions before reconciling the results.

An important procedural note related to the grounded coding approach is that we applied codes in terms of how the respondent discussed a given issue, even if a passage implicated themes that the user did not directly express. We addressed solutions and underlying problems as we interpreted the themes and translated them into specific recommendations.

Results

The analysis of the transcripts resulted in twenty-six general codes, varying from the very specific (Deposit Agreement, Versioning) to very general (Social and Legal Issues, used as a catch-all to capture issues not coded otherwise). In general, the codes fell into four groups: 1) metadata, 2) interfaces, 3) workflows and infrastructure, and 4) policy, legal, and social issues. Appendix B shows a list of codes and their scopes.

During analysis, we broke down the coded passages into 205 themes. Of these, 183 themes were discussed only in one code, and twenty-two themes repeated across two or more codes, typically due to the intersection of two codes. For example, the theme “Would like to be able to indicate relation as part of metadata” appeared under the Deposit Workflow, Item-Level Metadata, and Relationships codes. Depending on the specificity of the code and how central it was to the initial set of questions, a code might have as few as one theme (as with the Relationships code) or as many as thirty-two (in the case of the most general code, Social and Legal Issues).

Because we asked questions tailored to specific groups and due to the different dynamics of focus groups and interviews, we cannot quantify the differences in rates at which different codes or themes appeared. However, in the case of a few codes, we only identified relevant passages in transcripts from one or two of the user groups. Depositors were the only user group to mention issues related to user accounts and deposit agreements. Only depositors and intermediaries discussed batch processing, versioning, or the administrative interface. Depositors and consumers were the only groups to mention the general user interface. Only the intermediaries referred to relationships between items of other kinds beyond versioning.

At the level of themes, unique contributions from different user groups are more numerous, with twenty-four themes exclusively communicated by intermediaries, twenty-four by consumers, and forty-eight by depositors (see Appendix C). Each group’s unique contributions tended to cluster in particular areas. Consumers made the most unique observations related to item-level display, as well as search process and results. Intermediaries had the most unique contributions related to community and collection metadata, and also had more unique insights into issues with general information organization. Depositors offered the most unique observations, clustering in issues related to user accounts, the deposit workflow, and social and legal issues.

Discussion

In this project we sought feedback from three distinct user groups: consumers, depositors, and intermediaries. While the literature has focused on depositors’ user requirements, and on consumers to a lesser extent, researchers have paid little attention to intermediaries who make the distributed management model of IDEALS possible. Therefore, we dug deeper into our findings specific to this user group. Considering the role of intermediaries (people who manage a community, usually liaison librarians but also campus colleagues who are typically affiliated with a particular project or service), the thematic areas most central to their feedback—Community and Collection Level Metadata, Information Organization, and Social & Legal Issues—are not surprising.

Spotlight on Intermediaries

Themes unique to intermediaries (Appendix C) tended to be about ways the IR could better help these individuals manage their communities. For example, provision of high-level statistics information could be used to communicate impact of works in IDEALS. Intermediaries also expressed a need for more training as they were not aware of everything they could do as community administrators. Although intermediaries were not always sure what their permissions did entail, they were well aware of what they could not do. They described this awareness in discussions of batch ingests. The vast majority of content in IDEALS comes from batch ingests, which a web developer must upload into the repository via the server. Intermediaries who regularly curate content batches for IDEALS wanted to be able to perform batch ingests themselves, with appropriate training, rather than go through the current process, in which the IR management team reviews all the metadata for adherence to the metadata policy and best practices (Stein et al., 2017). While this request is not unreasonable, it is at odds with existing practice designed to account for the fact that the vast majority of metadata errors in IDEALS are introduced via batch ingests.

What is most significant about the intermediaries’ feedback is that it illuminates internal politics both within the library as well as on campus. Somewhat contrary to Salo’s (2009) findings, the intermediaries, including other subject specialists, were invested enough in IDEALS to want control over relevant communities and collections and thus a more decentralized library management model that would allow them to work directly with their communities rather than having to negotiate with the repository services team. To some extent this brings IDEALS into a familiar push-pull for centralized control that we’ve seen locally in other library services such as cataloging and reference.

Similarly, frustration from intermediaries over certain facets of information organization in IDEALS reflect a sense among some members of that group that subject specialists are better positioned to know their departments. For instance, DSpace provides a list of top-level communities. In IDEALS, this list is generally organized to reflect the organizational structure of the university. Intermediaries identified several communities for schools and departments they felt were incorrectly placed as sub-communities of colleges or other departments on campus or were correctly placed but historically the subject of complaints from faculty that an electronic thesis or dissertation was “incorrectly cataloged.” There are also top-level communities for recurring campus conferences that they felt should not be moved under the umbrella of any specific campus community because no one department could claim ownership. This boundary-drawing, while certainly reflecting an interest shared among repository developers and the intermediaries to correctly represent the university, is also about who owns what: that is, not just what departments, but which intermediaries should be able to control particular content.

We did not receive the same level of pushback or skepticism regarding the purpose of the service as was reported by early user studies of IRs, although one user did wonder why they should deposit into IDEALS instead of a disciplinary repository or pre-print server such as arXiv. However, this is likely due to recruiting that prioritized active intermediaries, people who had deposited work in IDEALS, or those who happened to be actively using it at the time of the study.

Known Usability Issues

Many of the findings from this study have been known issues for quite some time and were not a surprise. For example, the IDEALS website needs to adhere to Web Content Accessibility Guidelines 2.1; the login button is difficult to find; the lack of authority control for author and contributor names, as well as campus units, is frustrating; depositors cannot edit previously submitted items; consumers and depositors do not know what “communities” are; intermediaries frequently mix-up communities and collections; search results are often excessive and irrelevant; and, much like most online terms of service, no one reads the deposit agreement.

New Findings

The study did unearth some surprising findings, most of which pertained to anxiety about creating or interpreting metadata. For example, every submission is required to have at least one depositor-generated keyword, which has been stored in the Dublin Core element “dc.subject.” The display label for this field is “Subject” and the corresponding “Browse by” category is also called “Subject.” Although terms from controlled vocabularies can be found as values in this field—mostly due to repurposed catalog metadata—they function as keywords, not a controlled vocabulary. This conflation of keywords and subject terms turns out to be confusing for depositors and consumers. In particular, depositors feel anxious about “assigning subject terms” when they are only creating keywords. Similarly, depositors and consumers do not understand the difference between “authors” and “contributors.” Additionally, depositors said they were uncertain what a complete record looked like or what data entry rules there are for metadata.

Other findings unrelated to metadata also surprised us: despite IDEALS’ community structure reflecting campus departmental hierarchies, it is unclear to depositors which collections they can submit materials to and why; and consumers want to be able to browse image-based collections as a gallery rather than as a list of items.

Recommendations

The recommendations generated for internal use reveal tension between the needs and desires of the three user groups and the IDEALS management team that raise issues of internal university politics similar to those identified in the analysis of codes and themes. Certainly, many of the themes led to improvements we want to implement, or successes we want to take advantage of in workflows, interfaces, and outreach and engagement.

Other themes, though, led to suggestions that gave us pause. For some of these, usually in cases related to policy where the implications for changing features could pose challenges, our recommendation for a next step was to “consider” a set of actions. For example, “[consider] widening permissions levels for batch editing to additional groups of intermediaries beyond the core IR team.” These recommendations are unlikely to result in immediate implementation but may become part of the ongoing development of IDEALS, or consideration may lead to a different outcome than what users recommended.

Some recommendations, though, were simply to take no action in response to a suggestion that arose in a theme because the suggestion would violate policy or law. For example, providing more detailed information about consumer traffic to intermediaries could violate the library’s privacy policy—although some consideration of improved usage reporting is warranted.

Power, Politics, and UX

The tensions over control and permissions that arose in the intermediary discussions, as well as the user requests for features that run against what is possible due to policy and law, are important reminders that UX is tied to questions of power and politics. It can be easy for studies such as ours to imagine that systems and interfaces are just a matter of improvement through user testing or meeting compliance standards. But UX cornerstones like “usefulness” cannot be separated from who the various user groups are, the relationships between the user groups, and that they are made up of human beings. Issues of permissions for use of advanced features and of information organization in a system of campus units are tied to questions of who controls collections. Most notably, the balance of subject specialist control over their IR communities with central repository management control ensuring consistency with policy and quality standards reflects long-standing internal politics of the library.

Additionally, repository administrators may need to balance some expressed user needs against legal or policy restrictions. For example, one user requested that we provide more information on consumers of repository content, including IP addresses. Tracking and providing access to this level of user information would violate the library’s privacy policy.

Alignment with Visions of Next Generation Repositories

Our findings align with nine of eleven areas emphasized in the Confederation of Open Access Repositories’ report (NGRWG, 2017) on next-generation repositories to varying degrees: Exposing Identifiers, Declaring Licenses at the Resource Level, Interacting with Resources (Annotation, Commentary, and Review), Collecting and Exposing Activities, Batch Discovery, Identification of Users, Authorization of Users, Exposing Standardized Usage Metrics, and Preserving Resources. The two remaining areas none of our findings aligned with are gaps: Discovery of Navigation and Resource Transfer.

Discovery of Navigation is a particularly interesting behavior because it essentially boils down to thinking of machines as users: “While a human user can intelligently move around between these various resources, understanding that they pertain to the same scholarly object, a machine cannot” (NGRWG, 2017, p. 17). The nature of our methods, interviews and focus groups, do not lend themselves to categorizing machines as users. This is a significant limitation in the study design since “machine users” have consequential implications for accessibility (Arlitsch, 2018).

Resource Transfer entails “supporting by value content transfer of their resources to support text/data mining and preservation applications. By value content transfer entails allowing third parties to efficiently access and transfer the actual content of scholarly objects” (NGRWG, 2017, p.20). Although features such as supporting text and data mining of IDEALS’ content had come up during informal conversations previously, they did not arise in this study.

Overall, it is encouraging that the needs of IDEALS’ users seem to align with the confederation’s vision for the future of scholarly repositories. However, this is not the case when compared to other work in this area. As mentioned previously, based on interviews with experts in smart technologies, Shen (2019) called for libraries to fully embrace smart technologies and artificial intelligence. In our study, no participant brought up smart infrastructure and technologies; recommender systems were mentioned once, but in the relatively limited capacity of pointing to other items in the same collection from a single item’s access page. Furthermore, there are significant privacy concerns in regard to smart infrastructure and artificial intelligence that need to be carefully considered and addressed prior to and simultaneously with integrating such systems in libraries.

Conclusion

This study examined three user groups of one institutional repository. It identified distinct and overlapping needs among those groups, as well as underlying tensions between and within the groups. Additional studies in other repository contexts that look across user groups would help develop better understanding of the needs of these groups that repository developers and administrators must navigate. Moreover, further research could target how UX across a variety of contexts carries implicit assumptions of values, prioritization of the needs of some groups versus others, or negotiation between expressed needs of user groups and professional and professional ethics of information service providers. Another area of future research is exploring people with disabilities’ user experiences with IRs, in particular the way different technologies mediate their experience of the repository.

One of the biggest takeaways from this study was the impact of socio-technical factors on users’ experiences with IDEALS. For example, due to interdepartmental university politics, some users may disagree with the placement of certain collections under one department’s unit or another, as it implies the unit has jurisdiction, and thus control, over those materials. Intermediaries’ desire for enhanced metrics, including more detailed information about web traffic and content consumers, have implications for privacy protections and must be balanced not only against university policies or applicable laws, but our professional ethics as well.

Finally, this study also provides a user-driven approach to identifying next-generation repository needs that has some different outcomes than some of the leading public documents on next-generation repositories. Expert opinion surveys often drive these documents, and based on our study, some of the recommendations in these reports, such as those related to “smart” features, may be more related to what some systems designers think would be the cool new thing rather than addressing expressed user needs. Another way of putting this is that while information service providers may have ethical obligations or policy constraints that provide a check on features users express a desire for, understanding user needs can put a check on information service providers when they imagine new features that they find interesting but don’t have a true expressed need for.

Like other library systems and services, institutional repositories are not neutral; they are shaped by and can reinscribe different power structures and value assumptions. The UX of repositories is one formation where librarians and library users can read those values.

Appendix A:  Interview and Focus-Group Questions

Depositor Interviews

  1. In the past, why have you deposited or attempted to deposit items in IDEALS?

    1. Follow-up questions may relate to: How consistently have you put research outputs in IDEALS? Are there any factors that shape why you include some things and not others? Are there types of outputs you specifically choose to deposit or not deposit in IDEALS?

  2. In depositing items to IDEALS, what parts of the process have been easiest?

  3. In depositing items to [repository], what parts of the process have been more difficult?

    1. Potential follow-up question: What features could make depositing an item more appealing or easier to do? If so, what are they?

  4. What is your understanding of the license you agree to whenever you deposit something into IDEALS?

  5. [The interviewers will share a screenshot of the metadata input page in IDEALS with metadata elements highlighted for the participant.] This screenshot shows the basic information you provide about a file when you deposit it in IDEALS. Do you understand what each of the highlighted fields is asking for? Do any of them seem extraneous? Is there any additional information you would like to provide about something you upload to IDEALS?

  6. Are there ways you would like to see the items you have deposited to IDEALS be displayed or useable by others?

  7. Are there metrics you would like to have available about how your deposited items are used?

  8. Is there anything else about your experiences with IDEALS you’d like to mention?

Consumer Interviews

  1. Can you explain your understanding of what [repository name] is?

  2. [explain what the repository is if necessary] Have you accessed items in IDEALS before? If so, can you talk about your experiences?

    1. Follow-up questions related to what was easiest and hardest, and if they have used the repository content as a dataset.

  3. Using the repository, can you show us how you would find this item for which you have the author and title [example provided to interviewee]?

  4. Using the repository, can you show us how you would find a recent dissertation in chemistry?

    1. Possible follow-up questions about browsing: How do you browse the community/collection structure/navigation? How do you use the browse categories?

  5. When you find an item you want in IDEALS, what kinds of things would you like to be able to do with that item in your browser and what things would you prefer to do by downloading the file?

  6. [The interviewers will share a screenshot of an item-level page view in IDEALS with metadata elements highlighted for the participant.] This screenshot shows the basic information you get about a file in IDEALS. Do you understand what each of the highlighted fields is? Do any of them seem extraneous? Is there any additional information you wish was here?

  7. Is there anything else about your experiences with IDEALS that you would like to share?

Intermediary Focus Groups

  1. What are the most common ways you interact with IDEALS as an administrator or liaison?

  2. What is something you have to do routinely in IDEALS but have problems with?

  3. What are things you would want to be able to do with IDEALS that you cannot do as an administrator?

  4. What routine tasks in IDEALS work well for you currently?

  5. What questions from end-users you work with, if any, are most typical in relation to IDEALS?

  6. Is this a service you tell people about? Why or why not?

  7. Metadata activity: the researchers provide a print-out of all the metadata in the submission screen and another sheet of all the Dublin Core elements the repository supports.

    1. Indicate which information is not relevant on the submission screen and indicate which information should be included on “potential” metadata list. [Participants are given a time to mark up the sheets separately, and then discuss as a group.]

  8. What else would you like to see in IDEALS either from the deposit or access side, and as an administrator?

  9. Is there anything else you’d like to say about your experiences with IDEALS?

Appendix B:  Codes for Interview Transcript Passages

accessibility: accessibility for people with disabilities

account: issues related to logging in and to finding or manipulating user account information

administrative interface: the literal administrative interface, not activities that admins have the privilege to do (see user privileges for the latter)

authority control: issues related to consistency of names of entities in IDEALS

batch processing: issues related to uploading or editing groups of item records at one time

browsing: discovery by navigating the links to established communities or other categories directly provided for exploration, and looking through the assets collected in those categories

community/collection-level display: actual display of collections and communities, not how they are organized conceptually (see information organization for the latter)

community/collection-level metadata: issues related to metadata applied to communities and collections in general or specific communities or collections, regardless of whether the metadata is user-generated or system-generated

deposit agreement: ideas about what the agreement states or what it should state

deposit workflow: the process of submitting an item into [the repository]

document quality: scanning quality, file integrity, and other issues related to quality of files deposited in the repository

information organization: how resources, communities, and collections are conceptually related, and activities involving organizing information (e.g., data modeling, or the action of item-level mapping)

interoperability: issues related to current, imagined, or desired interoperability of the repository with other information systems

item-level display: what users see in the web display of an item record

item-level metadata: issues related to metadata fields applied to items generally, or in the case of specific items, regardless of whether the metadata is user-generated or system-generated

multimedia: issues related to uploading, storying, and displaying non-text media such as video and audio

permanence: issues related to preservation as well as handle link permanence

relationships: any relationships between items in the repository beyond versioning

search process and results: discovery through use of the search bar functionality and navigating the results

sharing: issues related to using IDEALS to share content by depositing content there or sharing links to content there

social and legal issues: includes any such issues not captured by other codes, including issues of law and policy as well expressions of values or interactions between users

statistics and reporting: front- and back-end reports related to metrics on deposited content or on usage of deposited content

training: approaches to teaching people how to use the repository in any role, including via documentation, help text, in-person training, or other methods, as well as expressed needs for such training

user interface: issues related to the public interface not specifically related to item-level or collection/community-level display

user privileges: actions that can be performed by users in particular roles

versioning: issues related to tracking different iterations of an item

Appendix C:  Unique Themes by User Type

Depositors

  • Need to verify screen reader compatibility and ARIA

  • Users not added to system that should be and they need admin assistance

  • Two-factor authentication is annoying

  • Placement of “My Account” information is unintuitive

  • Log-in process is easy

  • External user account creation and authentication as challenge

  • Finding log-in button is non-intuitive

  • Mismatch between back-end editing label for metadata and regular user interface label

  • Template creation helps for collection-level templates and would like to be able to create templates across collections

  • Can only browse collections as lists of items, which is limiting

  • Want to link to specific community/collection

  • Do not know what collection to submit to or why

  • When submitters read the deposit agreement license, they understand what it’s saying

  • Submitters don’t read the deposit agreement license and they mistakenly think that the deposit agreement license signs ownership of the work to the university’s Board of Regents

  • Feel uncertain about standards for completeness and precision for metadata, which makes it hard to deposit

  • Can’t figure out where/how to deposit

  • Unclear that you can upload more than one file to a record

  • Extra process for material with DOI yet to be assigned

  • Lag in workflow, especially when depositing large files

  • Inconsistent calendar interfaces for dates / Date entry not always appropriate as implemented

  • Need to be told identifier after you complete submission

  • Unclear how to submit a website

  • Approval turnaround time for moderated community was long

  • Upload process is cumbersome

  • Didn’t have all metadata prepared that would need

  • Contributor vs. author is unclear

  • Depositors like that the system accepts PDFs

  • Select a collection list is overwhelming for some depositors

  • Some users don’t know what collections are, how to deposit to them, or which ones they’re allowed to deposit into, if any

  • Appreciation for use of digital preservation best practices

  • Need to distinguish types of contributors to allow specification of types of contributions

  • Desire for easier creation and editing of metadata

  • Lack of clarity about who is the publisher for some items, types

  • Depositors want the handle once they’ve clicked submit and don’t want to wait until an item has been approved

  • Want altmetrics and citations

  • Desire for enhanced web analytics

  • Need to be mobile friendly

  • Authors of co-authored papers may feel more empowered to deposit their work into an IR if they are the lead author

  • Enabling automated permissions for users at multiple institutions is difficult due to institutional security protections

  • Some depositors don’t read the deposit agreement, or they can’t recall all the details

  • Repository Team is responsive

  • It can be difficult for coauthors at different universities to decide in which IR to deposit their work

  • It’s helpful for librarians to have experience depositing materials for when they’re advocating the service to users

  • Metadata input may cause anxiety for some users

  • Takes too long to approve submissions

  • Creators use IDEALS as a way to circumvent for-profit ownership and restriction of research products

  • IDEALS can make the document-type research outputs that aren’t papers or formally published available

  • Concern that users will include third-party owned copyrighted materials into their works and deposit them into the repository

Consumers

  • Quality of media plays a role in whether users prefer to view in browser or download files

  • Users don’t understand how the subject browse category is supposed to function.

  • Hard to tell what community/collection item is associated with

  • Lack of links to works by same researcher in other repositories

  • Hard to decide what main file is if there is more than one file on item

  • Format requests

  • Desire to annotate in browser

  • Desire to search by electronic thesis or dissertation committees and advisors

  • Desire to include metadata only records

  • Researchers are creating more complex media types such as 3D modeling

  • Search bar seems like it is in wrong place

  • It is easy to find work by specific people

  • Not sure how to search for combination of metadata fields or specific metadata fields like date, name

  • Doesn’t know advanced search is there

  • Using quotations for exact phrasing seems not to work correctly, either eliminates too much or nothing

  • No ability to search by key fields for dissertations

  • Sometimes creators prefer to share their work as attachments instead of links

  • Make labels, graphs, and charts for statistics more user-friendly

  • Unclear labeling of headers/section names and links

  • Icons low-res/out of date

  • Website copyright dates don’t make sense

  • Users prefer DOIs over handles

  • Non-European names are confusing to some users

  • Undergraduate research that is in the repository has used variable, inconsistent approaches and doesn’t include Advisor information in the metadata

Intermediaries

  • Reduce need to enter same metadata multiple times

  • Want to see stream of new submissions

  • Better admin-level controls and ability to assign minor admin roles without causing problems

  • Cannot find all items in collections easily because page defaults to limited view

  • Admins want stats at every level of their communities

  • Community landing pages should be more informative

  • Unclear what goes in a community/collection description and why it doesn’t have content

  • Content should go in multiple collections but can’t do that as user

  • Need way to track changes in community names for users

  • There are some items that are poorly scanned

  • Administrators find being able to map items to other collections useful but not all admins are aware this functionality exists

  • Graduate electronic theses or dissertations are not mapped to relevant units outside home unit

  • Some administrators want to keep both communities and collections

  • Some users disagree with how administrative units such as Departments and Schools are represented in the list of communities

  • Some users disagree with the which departmental collections electronic theses or dissertations are cataloged under

  • There are communities that are completely empty

  • Collection admins unknown to public—hard to find contact info

  • Desire to create notes to explain metadata creation or values

  • Need training for admins/community managers

  • Need better guidance within the platform on how to do things like metadata correctly

  • Administrator control can help preserve intellectual integrity when users want to make changes that violate policy

  • Depositors may need help determining usage licenses and copyright

  • It’s unclear to users why certain materials are inaccessible or who to contact

References

Arlitsch, K. (2018). My user is a machine: Making information accessible to disabled users by structuring for machine intermediaries. Journal of Library Administration, 58(7), 728–738. https://doi.org/10.1080/01930826.2018.1514834https://doi.org/10.1080/01930826.2018.1514834

Babu, R., & Xie, I. (2017). Haze in the digital library: Design issues hampering accessibility for blind users. The Electronic Library, 35(5), 1052–1065. https://doi.org/10.1108/EL-10-2016-0209https://doi.org/10.1108/EL-10-2016-0209

Bell, S., & Sarr, N. (2010). Case study: Re-engineering an institutional repository to engage users. New Review of Academic Librarianship, 16(sup1), 77–89. https://doi.org/10.1080/13614533.2010.509517https://doi.org/10.1080/13614533.2010.509517

Bernard, J., Daberkow, D., Fellner, D., Fischer, K., Koepler, O., Kohlhammer, J., Runnwerth, M., Ruppert, T., Schreck, T., & Sens, I. (2015). VisInfo: A digital library system for time series research data based on exploratory search—a user-centered design approach. International Journal on Digital Libraries, 16(1), 37–59. https://doi.org/10.1007/s00799-014-0134-yhttps://doi.org/10.1007/s00799-014-0134-y

Betz, S., & Hall, R. (2015). Self-archiving with ease in an institutional repository: Microinteractions and the user experience. Information Technology and Libraries, 34(3). https://doi.org/10.6017/ital.v34i3.5900https://doi.org/10.6017/ital.v34i3.5900

Caccialupi, R., Calvi, L., Cassella, M., & Conte, G. (2009). Usability evaluation of a multimedia archive: B@bele. In Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., & Tsakonas, G. (Eds.), Research and Advanced Technology for Digital Libraries (Vol. 5714, pp. 370–376). Springer. https://doi.org/10.1007/978-3-642-04346-8_36https://doi.org/10.1007/978-3-642-04346-8_36

Chapman, J., Kelly, E. J., Woolcott, L., & Zhang, T. (2015). Surveying the landscape: Use and usability assessment of digital libraries. Open Science Foundation. https://doi.org/10.17605/OSF.IO/9NBQGhttps://doi.org/10.17605/OSF.IO/9NBQG

Cunningham, S. J., Hinze, A., & Nichols, D. M. (2016). Supporting gender-neutral digital library creation: A case study using the GenderMag Toolkit. In Morishima, A., Rauber, A., & Liew, C. L. (Eds.), Digital Libraries: Knowledge, Information, and Data in an Open Access Society (pp. 45–50). Springer. https://doi.org/10.1007/978-3-319-49304-6_6https://doi.org/10.1007/978-3-319-49304-6_6

Fernandez, M. (2018). How accessible is our collection? Performing an e-resources accessibility review. The Serials Librarian, 74(1–4), 81–86. https://doi.org/10.1080/0361526X.2018.1430424https://doi.org/10.1080/0361526X.2018.1430424

Foster, N. F., & Gibbons, S. (2005). Understanding faculty to improve content recruitment for institutional repositories. D-Lib Magazine, 11(1). https://doi.org/10.1045/january2005-fosterhttps://doi.org/10.1045/january2005-foster

Gaona-García, P. A., Martín-Moncunill, D., Gaona-García, E. E., Gómez-Acosta, A., & Monenegro-Marin, C. (2018). Usability of big data resources in visual search interfaces of repositories based on KOS. ICCBDC’18: Proceedings of the 2018 2nd International Conference on Cloud and Big Data Computing (pp. 33–37). Association for Computing Machinery. https://doi.org/10.1145/3264560.3264567https://doi.org/10.1145/3264560.3264567

Gaona-García, P. A., Martin-Moncunill, D., & Montenegro-Marin, C. E. (2017). Trends and challenges of visual search interfaces in digital libraries and repositories. The Electronic Library, 35(1), 69–98. https://doi.org/10.1108/EL-03-2015-0046https://doi.org/10.1108/EL-03-2015-0046

García, P. A. G., Martín-Moncunill, D., Sánchez-Alonso, S., & García, A. F. (2014). A usability study of taxonomy visualisation user interfaces in digital repositories. Online Information Review, 38(2), 284–304. https://doi.org/10.1108/OIR-03-2013-0051https://doi.org/10.1108/OIR-03-2013-0051

González-Pérez, L. I., & Ramírez, M. S. (2019). User experience of an institutional repository in a private university in Mexico: A fundamental component in the Framework of Open Science. Journal of Information Technology Research, 12(4), 67–87. https://doi.org/10.4018/JITR.2019100104https://doi.org/10.4018/JITR.2019100104

Goodale, P., Clough, P. D., Fernando, S., Ford, N., & Stevenson, M. (2014). Cognitive styles within an exploratory search system for digital libraries. Journal of Documentation, 70(6), 970–996. https://doi.org/10.1108/JD-03-2014-0045https://doi.org/10.1108/JD-03-2014-0045

Green, R., Dolphin, I., Awre, C., & Sherratt, R. (2007). The RepoMMan project: Automating workflow and metadata for an institutional repository. OCLC Systems & Services: International Digital Library Perspectives, 23(2), 210–215. https://doi.org/10.1108/10650750710748513https://doi.org/10.1108/10650750710748513

Guedon, J.-C. (2009). It’s a repository, it’s a depository, it’s an archive…: Open access, digital collections and value. Arbor-Ciencia Pensamiento Y Cultura, 185(737), 581–595.

Hee Kim, H., & Ho Kim, Y. (2008). Usability study of digital institutional repositories. The Electronic Library, 26(6), 863–881. https://doi.org/10.1108/02640470810921637https://doi.org/10.1108/02640470810921637

Hochwarter, W. A., Ferris, G. R., Laird, M. D., Treadway, D. C., & Gallagher, V. C. (2010). Nonlinear politics perceptions—work outcomes relationships: A three-study, five-sample investigation. Journal of Management, 36(3), 740–763. https://doi.org/10.1177/0149206308324065https://doi.org/10.1177/0149206308324065

Hoffman, L. (2020). Your coworkers deserve good UX, too. Weave: Journal of Library User Experience, 3(1). https://doi.org/10.3998/weave.12535642.0003.103https://doi.org/10.3998/weave.12535642.0003.103

Koshiyama, D., de Pinho, A. L. S., & Santa Rosa, J. G. (2015). Analysis of usability and information architecture of the UFRN institutional repository. In Marcus, A. (Ed.), Design, User Experience, and Usability: Interactive Experience Design (pp. 197–207). Springer. https://doi.org/10.1007/978-3-319-20889-3_19https://doi.org/10.1007/978-3-319-20889-3_19

Luca, E., & Narayan, B. (2016). Redesigning the open-access institutional repository: A user experience approach. In Morishima, A., Rauber, A., & Liew, C. L. (Eds.), Digital Libraries: Knowledge, Information, and Data in an Open Access Society (pp. 275–281). https://doi.org/10.1007/978-3-319-49304-6_33https://doi.org/10.1007/978-3-319-49304-6_33

Lynch, C. A. (2003). Institutional repositories: Essential infrastructure for scholarship in the digital age. portal: Libraries and the Academy, 3(2), 327–336. https://doi.org/10.1353/pla.2003.0039https://doi.org/10.1353/pla.2003.0039

Maness, J. M., Miaskiewicz, T., & Sumner, T. (2008). Using personas to understand the needs and goals of institutional repository users. D-Lib Magazine, 14(9/10). https://doi.org/10.1045/september2008-manesshttps://doi.org/10.1045/september2008-maness

Matusiak, K. (2017). User navigation in large-scale distributed digital libraries: The case of the Digital Public Library of America. Journal of Web Librarianship, 11(3/4), 157–171. https://doi.org/10.1080/19322909.2017.1356257https://doi.org/10.1080/19322909.2017.1356257

McKay, D. (2007). Institutional repositories and their “other” users: Usability beyond authors. Ariadne, 52, 62–69.

McKay, D., & Burriss, S. (2008). Improving the usability of novel web software: An industrial case study of an institutional repository. In Hartmann, S., Zhou, X., & Kirchberg, M. (Eds.), Web Information Systems Engineering—WISE 2008 Workshops (pp. 102–111). Springer. https://doi.org/10.1007/978-3-540-85200-1_12https://doi.org/10.1007/978-3-540-85200-1_12

Michalak, R., & Rysavy, M. D. T. (2020). Assessing the accessibility of library tools & services when you aren’t an accessibility expert: Part 2. Journal of Library Administration, 60(3), 295–300. https://doi.org/10.1080/01930826.2020.1727283https://doi.org/10.1080/01930826.2020.1727283

NetLingo. (n.d.). Smart tech. In NetLingo The Internet Dictionary: Online Dictionary of Computer and Internet Terms, Acronyms, Text Messaging, Smileys ;-). Retrieved November 22, 2021, from https://www.netlingo.com/word/smart-tech.phphttps://www.netlingo.com/word/smart-tech.php

Next Generation Repositories Working Group. (2018). Next generation repositories: Behaviours and technical recommendations of the COAR Next Generation Repositories Working Group. Confederation of Open Access Repositories. https://www.coar-repositories.org/files/NGR-Final-Formatted-Report-cc.pdfhttps://www.coar-repositories.org/files/NGR-Final-Formatted-Report-cc.pdf

Prom, C. J., Rimkus, K., & Popp, T. (2019). Swimming against the repository tide [PowerPoint]. IDEALS. https://www.ideals.illinois.edu/handle/2142/106026https://www.ideals.illinois.edu/handle/2142/106026

Rieger, O. Y. (2008). Opening up institutional repositories: Social construction of innovation in scholarly communication. Journal of Electronic Publishing, 11(3). https://doi.org/10.3998/3336451.0011.301https://doi.org/10.3998/3336451.0011.301

Rosen, C. C., Ferris, D. L., Brown, D. J., Chen, Y., & Yan, M. (2014). Perceptions of organizational politics: A need satisfaction paradigm. Organization Science, 25(4), 1026–1055. https://doi.org/10.1287/orsc.2013.0857https://doi.org/10.1287/orsc.2013.0857

Russell, R., & Day, M. (2010). Institutional repository interaction with research users: A review of current practice. New Review of Academic Librarianship, 16(sup1), 116–131. https://doi.org/10.1080/13614533.2010.509996https://doi.org/10.1080/13614533.2010.509996

Rysavy, M. D. T., & Michalak, R. (2020). Assessing the accessibility of library tools & services when you aren’t an accessibility expert: Part 1. Journal of Library Administration, 60(1), 71–79. https://doi.org/10.1080/01930826.2019.1685273https://doi.org/10.1080/01930826.2019.1685273

Salo, D. (2008). Innkeeper at the Roach Motel. Library Trends, 57(2), 98–123. https://doi.org/10.1353/lib.0.0031https://doi.org/10.1353/lib.0.0031

Seaman, D. (2011). Discovering the information needs of humanists when planning an institutional repository. D-Lib Magazine, 17(3/4). https://doi.org/10.1045/march2011-seamanhttps://doi.org/10.1045/march2011-seaman

Shen, Y. (2019). Emerging scenarios of data infrastructure and novel concepts of digital libraries in intelligent infrastructure for human-centred communities: A qualitative research. Journal of Information Science, 45(5), 691–704. https://doi.org/10.1177/0165551518811459https://doi.org/10.1177/0165551518811459

Stein, A., Applegate, K. J., & Robbins, S. (2017). Achieving and maintaining metadata quality: Toward a sustainable workflow for the IDEALS Institutional Repository. Cataloging & Classification Quarterly, 55(7–8), 644–666. https://doi.org/10.1080/01639374.2017.1358786https://doi.org/10.1080/01639374.2017.1358786

Stein, A., & Thompson, S. (2015). Taking control: Identifying motivations for migrating library digital asset management systems. D-Lib Magazine, 21(9/10). https://doi.org/10.1045/september2015-steinhttps://doi.org/10.1045/september2015-stein

Stobbs, R., Shiri, A., Farnel, S., Cockney, C., Rathi, D., & Campbell, S. (2018). A community-driven usability evaluation: The case of the Inuvialuit Settlement Region Digital Library. Proceedings of the Annual Conference of CAIS/Actes Du Congrès Annuel de l’ACSI. https://journals.library.ualberta.ca/ojs.cais-acsi.ca/index.php/cais-asci/article/view/1034https://journals.library.ualberta.ca/ojs.cais-acsi.ca/index.php/cais-asci/article/view/1034

Walker, I., & Halvey, M. (2017). On designing an oral history search system. Journal of Documentation, 73(6), 1281–1298. https://doi.org/10.1108/JD-10-2016-0121https://doi.org/10.1108/JD-10-2016-0121

Xie, I., Babu, R., Lee, T. H., Castillo, M. D., You, S., & Hanlon, A. M. (2020). Enhancing usability of digital libraries: Designing help features to support blind and visually impaired users. Information Processing & Management, 57(3), 102–110. https://doi.org/10.1016/j.ipm.2019.102110https://doi.org/10.1016/j.ipm.2019.102110

Zavalina, O. L. (2014). Exploring the role of scale: Comparative analysis of digital library user searching. Proceedings of the Association for Information Science & Technology, 51(1). https://doi.org/10.1002/meet.2014.14505101120https://doi.org/10.1002/meet.2014.14505101120

Zhang, T., Maron, D. J., & Charles, C. C. (2013). Usability evaluation of a research repository and collaboration web site. Journal of Web Librarianship, 7(1), 58–82. https://doi.org/10.1080/19322909.2013.739041https://doi.org/10.1080/19322909.2013.739041