Regular usability testing in academic libraries is essential to ensure that users can effectively navigate and use the resources available. With the shift toward online resources and the rise of remote and hybrid learning, the modality of these studies must evolve accordingly. Additionally, the growing use of mobile devices to access library resources and services highlights the need for more mobile-specific usability testing. To address these gaps, we compare our experiences conducting usability studies remotely with adapted in-person testing. We outline specific methods and steps for usability practitioners to plan and execute usability testing, regardless of the method. While remote testing proved less effective than traditional methods in certain areas, its lower resource demands make it a viable and scalable alternative when researchers apply it with careful planning and design.
For as long as academic libraries have had an online presence, they have relied on usability studies to uncover problems with their websites, discovery layers, research guides, and other services in order to provide a better user experience (UX) for their populations. As the technology facilitating modern workplaces evolved, so did its application in usability testing. Previously expensive software with specialized functionalities have made their way into professional arenas through clever adoption and affordable, cloud-based apps designed for ease of use. By the late 2010s, internet-based video conferencing was a boon to those interested in exploring remote options for testing. By 2020, these opportunities would prove ideal for academic librarians who were beginning to experience new hurdles in conducting usability research.
Since the start of the COVID-19 pandemic in 2020, many more higher education institutions offer both remote and hybrid options for their course modalities and service venues (
While virtual usability testing is starting to be discussed more often in the library literature, it has for some time been widely used in a variety of similar industries. One of the primary benefits of virtual usability studies is that they are more cost-effective and do not require a lab or any form of travel, eliminating the need for in-person meetings and travel expenses. Removing the need for a physical study space also makes it easier to recruit a more diverse group of participants. Virtual usability studies are also highly flexible and customizable to meet the needs of specific projects or products. For example, some studies may focus on UX and design, while others may examine the functionality and performance of a website. By tailoring the study to the specific needs of the project, researchers can obtain targeted feedback to inform decisions about website development and design.
Although there are many benefits to virtual usability testing, both research participants and research administrators have reported several challenges. Being distracted while participating in the study in a familiar environment is the most common issue participants reported. Administrators, on the other hand, cited under-reporting of issues and missing non-verbal cues—both of which can affect the depth and accuracy of usability findings. We will further explore these and other issues throughout this article.
We provide insight into our experiences conducting remote and in-person testing across institutions. This collaboration originated from the New Jersey Academic Libraries Network, a collective of five academic institutions that implemented the same integrated library system and previously maintained a shared lending network. Other universities in the network are New Jersey Institute of Technology, The College of New Jersey, and Rowan University. Implementing an integrated library system as a group has become a common practice, in part because it provides all participating institutions with additional opportunities for partnerships among the institutions. We were also members of the network’s Discovery/UX Group. Building upon this foundation, we combined our efforts while exploring the burgeoning area of remote usability testing in libraries.
Since we each navigated pandemic conditions to conduct usability studies at our respective libraries in spite of the challenges, we detail our experiences so other academic librarians can benefit from our work and implement similar studies in their libraries. By leveraging the insights gained from these studies, usability researchers can ensure that their library websites are intuitive, efficient, and enjoyable to use, which can lead to increased user satisfaction and foster institutional loyalty—a growing priority in higher education.
To chart the development of modern usability testing in libraries, we examined literature stemming from both the librarian profession and the world of commercial design practice. These sources provided interesting insight on the early history of remote testing, its overall practical advantages, the administrator roles involved in running them, the fundamental similarities and differences between traditional and remote testing, and the common technology and drawbacks associated with this approach.
In conducting our review of the literature, one thing was apparent: conducting usability testing with remote participants is not new. Although conducting virtual usability studies in libraries has become more commonplace, primarily in response to the COVID-19 pandemic and the quick move to virtual services starting in 2020, earlier studies looking at virtual usability in similar populations do exist in the literature. For example, researchers in the 1990s explored remote usability testing as a cost-effective method for testing users in a more natural setting (
The first studies on remote usability in libraries appeared in the library literature a few years later.
The remote testing medium offers many benefits to both administrators and participants, and perhaps the most evident of these is time and cost efficiency for those conducting the study. Remote usability testing allowed the administrators to spend less money and time without compromising the number of participants (
Also, participant recruitment was easier if volunteers could take part remotely (
The role of a testing administrator can vary greatly depending on the type of test being conducted and the conditions under which it is run. In instances of synchronous remote testing, an administrator plays a hands-on role by observing the participants during their sessions and recording notes based on those observations (
In terms of process and procedure, remote usability studies share many similarities with traditional, in-person studies in libraries. For example, most remote studies also use task-completion as their primary method of assessing usability (
As in best practices for traditional usability studies, remote usability research emphasizes careful procedural design.
Usability researchers conducting remote studies also employed many of the same recruitment methods used in traditional studies, such as email (
Remote usability tests also share certain similarities of practice with traditional tests where post-test assessments are concerned. Multiple studies asked participants to self-assess their experience using a five-point Leikert scale (
Among instances of remote testing, there are outlying studies that employed methods unlike many others.
Researchers also must give careful consideration to the software and equipment used in remote studies. Remote usability tests often employ specific platforms and applications, including Loop11 usability research platform (
When it comes to remote usability testing, there are not exclusively net positives; drawbacks do exist for this approach. For example, only remote study participants reported that instances of distraction were an issue during study sessions (
Finally, although remote testing participants tended to identify a higher number of usability problems than traditional participants , they also generally underreported the severity of those problems in assigned ratings (
A final area to consider is the burgeoning field of artificial intelligence. Librarians are actively incorporating these tools in many areas, including enhancing library website usability testing. In this area, artificial intelligence tools are used to develop user personas, map user journeys, create study tasks, define and test navigation, and analyze user feedback (
To further explore both the advantages and drawbacks of this process, we consider the remote usability testing the first author, Linda Salveson, conducted at the Cheng Library of WPUNJ, a mid-sized public university in New Jersey; we will refer to this study as US1. We will compare this study with a usability study the second author, Eric Jeitner, conducted at the Richard E. Bjork Library of Stockton University, another mid-sized New Jersey public university library; we will refer to this study as US2.
Both studies occurred during the pandemic. However, unlike at WPUNJ, the Stockton University researchers conducted their usability study under traditional, in-person conditions while also maintaining masking and social distancing. The two usability tests occurred independent of each other and were not designed and carried out with any intention of comparing them. However, the methodologies and procedures used in both tests were similar enough that it allowed for a straightforward comparison. This similarity provided valuable insights into the strengths and weaknesses of different usability testing formats.
This section of the methods is written from the perspective of the first author, who administered US1, the remote usability study.
Conducted in the 2021–2022 academic year, during the later days of the COVID-19 pandemic, US1 looked at the usability of the Alma LMS discovery tool, Primo VE (
Following typical study protocol, I applied for and received institutional review board approval before the usability study began. This ensured the maintenance of participants privacy and confidentiality of data, including the anonymizing of all data and recordings.
I placed a QR code throughout the library, posted on social media, and sent it to students in campus announcements that linked to a screening survey form. The form recorded the volunteer’s full name, university email address, the frequency with which they used Primo VE on a weekly basis expressed on a Likert scale, and whether they had received virtual and/or in-person library instruction in their university classes. Finally, the form collected date and time availability for the usability testing sessions. I used this information to determine other demographic data about the volunteers, including their university status and major/subject area. Since participants should be as close to a representative sample as possible, I recruited a range of student participants for screening based on both their year and major and scheduled participants for virtual usability testing sessions.
I used Zoom to conduct the real-time portion of US1. The platform’s screen capture functionality recorded both audio and video from the remote participants. During the virtual usability sessions, the primary mode of assessment was the think-aloud protocol paired with task-completion exercises. I gave participants a list of scenarios that required them to interact with the library’s discovery tool. I took care in wording tasks to avoid leading the participants’ thinking or subtly directing them toward successful task completion. The tasks themselves mimicked common activities performed by users of the discovery tool. Outside of the tasks, I asked participants about their library instruction history and provided an opportunity for them to ask debriefing questions pertaining to the tasks.
There were eight total virtual usability sessions for US1 in February and March 2022, based on suggested best practices by
Across all US1 sessions, no user participated more than once to avoid repeated testing, which is a “threat to internal validity” (
For the task-completion portion of the testing sessions in US1, participants shared their screens and started on the homepage of the university’s discovery service, PrimoVE. They completed three to five scenarios.
This section of the methods is written from the perspective of the second author, who administered US2, the traditional usability study.
Conducted in the 2021–2022 academic year, US2 focused on testing the library’s website, not its discovery platform. This study was traditional in design, conducted in-person using a library conference room. A library intern and I jointly administered US2 , and a student assistant performed some data processing. The choice of researchers was a practical consideration: I was responsible for assessment at Stockton University and the intern was interested in taking part in an assessment project.
We gained institutional review board approval for the study, as we intended to publish the findings. As in US1, we anonymized all participant data after collection.
Recruitment for US2 happened through word-of-mouth advertising. I worked with the student assistant to identify eight undergraduate students who were willing to be study volunteers. Acceptance was based on willingness to participate; there was no vetting process beyond that. Participating volunteers received a voucher worth eight dollars redeemable at the campus food court.
US2 sessions happened in a library conference room and used a table, three chairs, and a laptop. We asked each volunteer to complete ten tasks within the library website. The library intern acted as interviewer, reading a script to each volunteer and introducing the study tasks. I served as notetaker, observing the volunteer as they attempted each task and recording handwritten notes. Zoom recorded a screen capture of all desktop mouse movement and any spoken audio from the volunteer. To maintain privacy, we disabled the laptop camera and encouraged volunteers to voice any thoughts, concerns, or frustrations as they worked. The tasks were aligned with high-engagement points of the website, such as building hours, the chat help interface, subject research guides, etc.
In total, US2 consisted of eight sessions held across two weeks during the fall semester. Following the sessions, we analyzed the data extracted from the recordings. Much like US1, we used the screen-capture recordings to determine task success, time to completion, and number of clicks involved in each task. We wrote a report that included an extensive list of recommended changes to website elements based on task completion difficulty and presented it to the library’s technology team.
A second series of eight sessions with new volunteers occurred later during the fall 2022 semester. This was meant to act as a post-assessment method, testing whether the changes to the website improved usability for completing the tasks.
In both US1 and US2, we tracked the following statistical categories for each task: success/failure, time to completion, and number of clicks. We kept notes for each session, including steps taken to complete each task, important processes we observed, and comments from participants. In US1, the first author also considered prior experience with Primo and information literacy sessions to see if it was predictive of how effective our interface would be in supporting participants’ successful task completion.
In both studies, we anonymized the data by assigning a number to each participant and storing the consent forms and numbers separately. In US1, all study data was stored in WPUNJ’s Microsoft OneDrive; in US2, it was stored in Stockton University’s Google Drive. We made these choices because each university already had a respective subscription, but also because it allowed us to share the anonymized data among ourselves and to store session data in the cloud. We deleted all recordings after 365 days, per each university’s policy.
Evaluating the statistical categories helped pinpoint where our patrons were most often running into issues. In US1, the first author distributed a document providing recommendations based on the usability testing data at an internal library meeting and provided it to user education, discovery, and website maintenance teams. The internal library committee decided, as a group, which of the study’s findings they could implement.
In US2, the second author created and shared a similar report of findings and recommendations with the library’s technology team. This internal committee reviewed and approved all recommendations and oversaw updates to the website according to the recommendations.
Librarians who are looking to implement usability testing, whether virtual or traditional, should know that success depends on thoughtful preparation. Testing the usability of a library website in any format is crucial to ensure that users can effectively navigate and use the resources available. Once you make the decision to conduct usability testing, the next step is to determine which modality best suits your goals. Each approach offers unique advantages and challenges, and understanding these will help you more effectively plan and execute your studies. See Table 1 for a summary of these considerations.
Considerations for practice. For each topic, we offer key considerations for the in-person and remote modalities.
Topic |
In-Person Testing |
Remote Testing |
|---|---|---|
Recruitment |
May limit participation to those who can travel, potentially narrowing sample diversity |
May broaden participation by removing geographic and scheduling barriers, leading to greater sample diversity |
Setting |
Allows for richer observation data (e.g., body language, facial expressions), which can be invaluable for interpreting user frustrations or confusion |
Relies on digital platforms that streamline data collection and analysis but require careful attention to privacy |
Costs |
Involves higher costs due to physical space, additional staff, and onsite logistics; incentives required |
Involves lower costs due to reduced staffing and no facility use; flexible setup from any location; incentives required |
Configuring Screen Sharing and Recording |
Works best in controlled environments using specialized equipment or close observation to isolate usability issues |
Works well for observing participants in naturalistic settings, yielding more authentic behavior and reducing participant burden |
Observation and Note taking |
Requires structured note-taking during live sessions to accurately capture nonverbal cues and user behaviors |
Allows for session recording and automated transcripts, which support detailed review but raise privacy considerations |
Recruitment proved to be the most significant challenge in US1, likely due to the largely remote nature of campus operations at the time. While the first author had some initial success with traditional methods of recruitment, such as sending out official university announcements, as the study progressed, it became difficult to solicit enough volunteers to achieve the ideal demographic range of students. Recruitment picked up again after adding five-dollar Amazon gift card incentives to the study; these were reflected in an addendum to the institutional review board application.
More broadly, recruitment methods and study modality can shape the diversity of the participant pool. In-person testing may limit participation to those who are able to travel to a designated location at a specific time, potentially narrowing the range of student experiences represented. By contrast, remote testing can remove geographic and scheduling barriers, making participation more accessible to students with work, caregiving, or commuting constraints. However, as US1 demonstrated, simply offering a remote option does not guarantee broad participation; additional outreach strategies and incentives may still be necessary.
Outside of incentives, the biggest drivers of participation in both studies were personal connections to the library and spreading the call for volunteers via word-of-mouth. In light of this, we would recommend first spreading word among everyone working in the library, including student workers, to assist with word-of-mouth recruitment. However, be careful not to rely on library workers for participants, as they are biased.
Regarding participant status, while US1 was open to all students, the volunteers were mostly undergraduate students. Perhaps researchers should explore additional methods for recruiting graduate students so this user group is properly represented in usability testing efforts.
The fully remote usability testing of US1 yielded many benefits, including streamlining the scheduling and execution of sessions. While other librarians assisted with areas like recruitment, only one librarian was needed to conduct the testing. Conversely, US2 required two researchers in its traditional, in-person method: one interviewer and one notetaker. In addition to staffing differences, the physical setting of US2 allowed for richer observational data, including body language and facial expressions that can help researchers interpret moments of confusion or frustration.
Additionally, by “meeting students where they are,” remote usability testing carries the added benefit of having “ecological validity.” By this, we mean that the physical environment of the study setting—their home, office, study space, etc.—matches the space in which users usually perform research. In contrast, researchers conducting traditional usability studies typically ask participants to travel to testing labs or other controlled environments—settings that differ significantly from familiar spaces where they normally use online library tools and services. Though US2 used a library conference room as testing space for their traditional study, it was complicated by the need for social distancing measures to address pandemic safety concerns. At the same time, digital platforms used in US1 streamlined recording and data capture, simplifying later analysis, though they required careful attention to privacy and recording protocols.
While remote participants have the benefit of not needing to travel, the primary downside of the home testing environment is a higher probability of user distraction. The first author asked participants in US1 to minimize disruptions, if possible, but it was not guaranteed. US2 sessions occurred in a closed environment, isolated from visual and auditory distractions that might affect the participants. For remote studies, we recommend that communication about the importance of minimizing disruptions occurs both in advance of and at the beginning of each session.
The financial implications of modality were notable in both studies. Remote testing substantially reduces personnel and space-related expenses, and then sessions conducted from an office or home setup eliminated the need to reserve physical rooms or coordinate additional staffing for on-site observation. In contrast, US2 required a dedicated library space, which carried hidden costs, including space reservation, equipment setup, and, in this case, safety considerations (e.g., social distancing). Additionally, using two people—interviewer and notetaker—for US2’s in-person sessions increased labor costs. These factors underscore how traditional, in-person usability testing often involves higher logistical and staffing costs.
As noted earlier, participant incentives also played a key role in recruitment for both studies. Although initial recruitment efforts were successful without incentives, additional funding was eventually required to sustain participation. This highlights a recurring cost consideration for practitioners planning usability studies regardless of modality.
In both usability studies, only the screen-sharing portions were recorded, ensuring that we could reference sessions when compiling data. An important factor for usability studies is maintaining participant privacy. During US1, because Zoom displays the participant’s full name, it was important to first “rename” them using either initials or, ideally, a participant number. This ensured we were able to track participants without violating privacy standards. US2 used a library laptop and staff Zoom credentials during all sessions, ensuring privacy. If you are conducting a usability study remotely, privacy can be ensured by asking participants to turn off their cameras before beginning the recording. For reference, the last line of the interviewer’s script before they began to record the session was “Once you turn your camera off and share your screen with me, I’ll start recording.”
Beyond privacy considerations, the configuration of screen sharing and recording also shaped the type of data collected in US1 and US2. US2, conducted in person, benefited from a controlled environment and closer observation of participants’ interactions, allowing usability issues to be more deliberately isolated. In contrast, US1, conducted remotely, allowed participants to engage with the website in their natural environments, which may have yielded more authentic behaviors and reduced participant burden. While this naturalistic context introduced greater variability, it also offered insight into how users navigate library websites under real-world conditions.
This is one area where the testing modalities differ greatly. Since participants in US1 were instructed to turn their cameras off before the recorded testing portion began, the first author could not observe them during the testing sessions. This is one challenge that could not be overcome. As a result, note-taking in US1 relied heavily on screen activity and verbal comments rather than nonverbal cues. Direct observation was also difficult in US2, although the underlying reason differed. Masking and social distancing made it more difficult to observe small changes in facial expressions and, as a result, to guarantee administrator observations were always correct. In traditional in-person usability studies, structured note-taking during live sessions is typically used to capture body language, hesitation, and other behavioral indicators that supplement screen-based findings; however, pandemic precautions limited the extent to which these cues could be accurately interpreted in US2.
This is one major limitation in both studies, since nonverbal communication is what usability researchers typically rely on to supplement analysis. However, revisiting the recordings of sessions for additional analysis enhanced screen observation and note-taking. In the case of US1, session recordings supported detailed review of participant navigation patterns, though these tools required careful attention to informed consent and data security. Finally, always ensure informed consent, data security, and accessibility, regardless of modality.
Much work remains to identify the ideal recruitment procedures for virtual usability studies in academic libraries. For both of our studies, incentives were vital—although that may not be practical at many institutions, especially if testing is done regularly. Successful alternatives have appeared in the literature, such as the concept of using “guerilla usability testing” (
Because we derived great value from our cross-institutional collaboration, we would like to stress the importance of connecting with your peers. The New Jersey Academic Libraries Network provided us with a means of sharing tips, tricks, and best practices, which was especially vital during the isolating times of the COVID-19 pandemic. Through this network, we also connected with other librarians doing great work in this space, including the creator of the recently launched DIY Usability Testing Toolkit for academic librarians (
As remote and hybrid learning continue to expand, virtual usability testing remains a vital method for academic libraries to engage with their increasingly distributed patrons. While this approach offers efficiency and accessibility, traditional in-person testing still provides unique insights that virtual methods may not fully capture. As
Overall, we each gained valuable insights from our respective studies. After sharing these with each other, we formally presented our findings to our respective internal user education, discovery, and website maintenance teams.
We remain committed to fostering a culture of usability testing within our libraries, allowing us to take a more proactive approach to regularly evaluating our web presences through iterative design. This ongoing commitment not only strengthens our digital services but also reinforces the value of user-centered design in academic library environments.
Finally, we hope this article supports other academic librarians in making informed decisions about conducting their own usability testing. By considering the two modalities we have explored—along with their respective conditions, requirements, strengths, and shortcomings—librarians can use this comparative framework to guide their own usability studies with greater clarity and confidence.