Nevada State College (NSC) is a four-year public institution in Henderson, Nevada, founded in 2002. NSC is a Hispanic Serving Institution, an Asian American Native American Pacific Islander serving institution, and a Minority Serving Institution. There are 7,215 students enrolled with a full-time equivalency of 3,831. The Marydean Martin Library at NSC is the first digital library in the state of Nevada, and in 2020 the library received the ACRL Excellence in Academic Libraries award.

In 2019, we began noticing problems with the library’s link resolver. A link resolver is “software that interprets source OpenURLs, checks holdings in the local knowledge base, and creates links to targets and services.”¹ In other words, it takes users from an indexed citation in one database to the full text in another. In audits of reported electronic resource access issues, link resolver or knowledge base problems have consistently come up as the most frequent or second most frequent type of issue libraries encounter.² Most large-scale audits of electronic resource access problems categorize problems reported to ticketing systems,³ but at NSC we had a very low volume of issues reported. Rather than looking at how many of our reported problems were related to the link resolver, we wanted to analyze how prevalent link resolver problems were across the collection. This led us to design a review based on a random sample of citations to which we should have full text access.

As a small library without a dedicated electronic resources librarian, we were also interested in how we could approach this large-scale review as a team. We were inspired by libraries that had involved student workers in the troubleshooting process⁴ but did not have the staff to train students at a high level. Ultimately, we designed a process that enabled students to identify problems but relied on professional staff to categorize and diagnose them. This process allowed us to involve student workers with minimal training.

The first step in our process was collecting a random sample of citations to which we should have full text access. We started with a random selection of journals drawn from a master list of our journal holdings. We randomized that list in Excel and took the first 383 journals, a statistically significant sample size based on our total journal holdings. Each journal title was then searched in our Journal A-Z list, in Ex Libris Primo, and we then selected the first platform with full-text access to the journal listed in Primo. To randomly generate the citation, we then input number ranges into the number generator on Random.org to determine which year, volume, issue, and article was selected for the citation. During the collection process, we discovered that some journals had dropped out of our holdings, and we replaced those journals with other journals from the randomized list.

We entered the randomly-generated list of citations into a Google Sheet. Students were instructed to check each citation in three places: our discovery layer, Google Scholar, and EBSCO. Each source had a dropdown menu to select whether the citation was indexed in the source, and if so, whether a full text link through the link resolver was present and that link was working to connect to the full text. Students were asked to make notes for anything unusual and to save permalinks to aid in future troubleshooting. In general, student workers were much more likely to identify something as a problem when it was working as expected than the other way around. We did have issues with students indicating citations were not found when a more advanced search strategy could have uncovered them. If we were to do this project again, we would dedicate more time to training each student in advanced search techniques and strategies for finding full text when faced with various usability problems.

Before analyzing our data, we reviewed student work and added some information about each citation. We reviewed any citations students had identified as a problem or left notes on. We also verified indexing in our Primo discovery search for any citations students marked as “Citation Not Found” for discovery. We did not verify indexing for Google Scholar or EBSCO. When there was a citation in Primo, we clicked the permalink and noted the number of clicks required to access the full text. We also made note of information about the journals or publications, such as the full-text platform and database, ISSN if available, the language of publication, and whether the publication was Open Access or peer-reviewed (determined based on metadata in our discovery records or EBSCOhost publication records).

We could not analyze link resolver linking for forty-eight of the citations in our sample—about 9 percent—because they were not indexed in Primo, Google Scholar, or any of our EBSCOhost databases. Table 1 shows the number of citations not indexed in each source by publication type. EBSCOhost had the best overall indexing, mainly because most of the reports in our sample were from EBSCO databases like Business Source Complete. Primo and Google Scholar had lower indexing overall but outperformed EBSCOhost databases in their indexing of journal articles.

When looking at citations that were not indexed in any of the three sources by publication type, we found that journal articles and books were heavily indexed, but newspapers and primary sources were not. Figure 1 shows a breakdown by publication type.

With discovery systems, this is an actionable insight. Most discovery vendors allow libraries some control over what is included in the index. In our case, we are considering adding more newspaper databases to our Central Discovery Index to see if that improves overall coverage.

After eliminating citations that could not be assessed due to lack of indexing, we found 24 percent of the citations in our sample had some barrier to access. For citations that did not link directly to the full text, we organized them into four exclusive categories based on the nature of the problem: invisible, temporary, usability, and critical. Invisible problems were problems with holdings or indexing that prevented full text links from appearing but did not result in a broken link path. Temporary problems were problems that would resolve themselves in time, such as a website outage or recent coverage change. Usability problems were issues where the full text was available, but the path to get there was confusing or difficult to navigate. Critical problems included error messages or hitting a paywall. A majority of the problems (62 percent) were invisible problems students would be unlikely to discover or report. See Figure 2 for a breakdown by problem category.

In addition to the larger problem categories, we assigned more specific and non-exclusive problem types that described more precisely the cause of the citation linking problem. A single citation might have two to three problem types. Table 2 includes an overview of problem types, and how often they occurred in our sample.

When breaking down the issues by problem type, holdings issues were by far the most prevalent. Eighty-two of the citations in our sample had an issue related to holdings, accounting for a total of 21 percent of citations. Of the citations with holdings issues, 87 percent were Open Access publications, a situation that was due to a problem with journal holdings in the Ex Libris Alma Community Zone collection for the Directory of Open Access Journals. Many of the journals in this collection did not include dates of coverage, so our link resolver treated these titles as having no access. As a result, many of these Open Access articles were not discoverable. They did not show up in our Primo discovery search results unless users included results outside our institution, and instead of linking to the full text, they often displayed interlibrary loan request options.

Another prevalent problem type was Indirect Linking. This impacted twenty-eight of the citations in our sample or about 7 percent. In this case, users were not taken directly to the full text via the link resolver. Instead, the link resolver often targeted a journal publication page, requiring users to navigate to the correct volume and issue for full text. Sometimes it linked to the database homepage and required a search for full text. Figure 3 includes a chart of the number of extra clicks required to reach the full text in cases of indirect linking. The average was 2.8 clicks.

In twenty-seven of the citations in our sample, we identified a problem type that we coded as EBSCO Custom Link Error. This is one we easily resolved, demonstrating the advantage of doing a large-scale review that reveals patterns. Our link resolver is accessed from the EBSCOhost platform via a custom link that we had set up to appear on every detailed record page. This decision resulted in occasional errors where the full text would be accessible in EBSCO, but the custom link would result in an error. This generally overlapped with other problem types, such as holdings or metadata. As a simple fix, we edited the logic for our EBSCO Custom Link so it only appeared when the full text was unavailable via EBSCO. This reduced redundant links on our interface and resolved about seven percent of the problems identified.

Twenty-three of the citations in our sample, or about six percent, had error messages. In many cases, these were among the simplest to resolve. An error message signals unintended behavior and the message made clear the platform, vendor, and nature of the problem. With only two citations in our sample, the smallest category of problems we found were metadata issues. These were issues where the index and full text databases might have slightly different metadata for a citation, resulting in a failed link.

In addition to identifying and working to resolve specific problem types, our analysis allowed us to look for patterns based on criteria such as peer review status or Open Access status. Figure 4 shows an analysis of citations that worked properly, were not indexed in any source, or had some type of linking problem by peer review status. Overall, peer-reviewed items were much more heavily indexed. The ratio of working to problematic citations was about the same between peer-reviewed and non-peer-reviewed sources.

Figure 5 shows a similar analysis by Open Access status. As you can see, Open Access citations were much more likely to be indexed. However, we did see more problems than working citations among the open access citations in our sample. The vast majority of these problems were invisible to the user, meaning they were holdings problems that resulted in low discoverability for the citations but did not produce an error, indirect linking path, or similar issue.

Conducting a large-scale review of our link resolver allowed us to identify some problems we could solve in bulk. However, most of the issues we identified were not so easily resolved. The team-based approach to identify problems was helpful and efficient, but a team-based approach to resolve them would require much more training and time to develop. In our case, we found that student workers were identifying problems much faster than we could identify solutions. Despite this, we would still recommend our approach to other libraries. While we did not have the time to resolve every uncovered issue, this approach helped us prioritize and focus on the fixes that would have the most significant impact.

One of the challenges we faced in our review was not having a shared vocabulary for the kinds of issues we might uncover. We had to develop this over time, which required revisiting citations that had already been reviewed as our understanding developed. It would be useful to see what types of problems other libraries uncover if they proceed with similar reviews. As a profession, we could work toward a shared vocabulary that would aid in troubleshooting, training, and analysis.