Five Second Testing

The method for our first test was inspired by the finding that users develop an impression about a web page in as little as 50 milliseconds (Lindgaard et al., 2006). To study the first impressions of our users, we tried the method known as the five second test, thinking that this method would best capture those first impressions (An Introduction to Five Second Testing, n.d.). We conducted five second testing on a drop-in basis in front of the entrance of the primary student dining hall on campus. We had twenty-one participants who were a convenience sample of current SMCM students recruited through various methods: advertising through the All Student listserv, an announcement on the Library website, flyers posted around campus, and approaching students as they passed the dining hall entrance during the testing. Participation was voluntary, and the only criterion for participation was status as a current SMCM student.

After signing the consent form, we provided participants with a paper packet that included a brief questionnaire to gather basic demographics and information on average use of the library’s LibAnswers FAQs, an instruction sheet, a printout of the library’s LibAnswers FAQ homepage, and a response sheet. The instruction sheet asked participants to view the printout of the LibAnswers FAQ homepage for five seconds (see fig.1), then flip to the final page of the packet—the response sheet—and answer the questions. The response sheet asked participants to recall the overall purpose of the page and as much information as they remembered seeing on the page. We instructed participants to return the testing packet to us when finished.

Figure 1.

Library’s LibAnswers FAQ homepage at the time of testing.

Due to the drop-in nature of this testing and the fact that there were only two of us conducting the research, after participants returned their testing packet, we handed them a written debriefing paragraph in lieu of doing a verbal debrief. Participants received incentives in the form of library buttons and a raffle entry form for a chance to win a library-branded water bottle. The raffle entry form included the option to be contacted about participating in usability testing at a future date.

We identified all testing materials with an alphanumeric code for cross-referencing to ensure participant anonymity. We did not code consent forms and raffle entries for cross-referencing, and we kept them separate from any study material.

Usability Testing

Usability testing involves a series of scenario-based tasks using a think-aloud protocol (Nielsen, 2012). We designed our tasks to reflect information the LibAnswers FAQs are intended to help students find. We provided participants with a series of short tasks to complete using the LibAnswers FAQ homepage, and we asked them to describe aloud their experiences while completing the tasks.

The conventional wisdom for usability testing is that you only need to test five participants, as after five you tend to see the same successes and pain points and thereby get diminishing returns per subsequent test (Krug, 2010). As our goal with this study was to identify recurring issues that could be used to inform a LibAnswers FAQ homepage redesign to improve overall usability, we chose to follow the typical usability research approach.

We recruited a convenience sample of six current SMCM students through the All Student listserv, an announcement on the Library website, flyers posted around campus, and follow-up emails to five second testing participants who indicated interest in usability testing on their raffle entry form. This gave us the standard five participants plus one participant in case someone dropped out. In the end, all six participants showed up for testing. Of our six participants, five lived on campus and one was a commuter. Two of our six participants were transfer students. Among all participants, time spent as an SMCM student ranged from two to eight semesters.

Testing occurred in the college’s Usability Testing Lab to reduce ambient noise. Further measures to ensure a quiet testing space included using wooden chairs to reduce distracting noise produced by chair movement. Another benefit of using the Usability Testing Lab was using the TechSmith Morae usability software licensed for that lab. While Morae has many features, we used the usability testing options of screen recording, designating tasks to identify sections of the recordings easily, annotating recordings at specific timestamps, and creating and analyzing pre- and post-test surveys. Morae is by no means necessary to conduct computer-based testing, but it can streamline analysis of test recordings.

Initially, we planned to use a method inspired by Krug (2010) in which one investigator acts as facilitator and a second investigator acts as observer. Ideally in this scenario, both facilitator and observer are present in the room during testing sessions with the observer making notes about participant behavior to supplement the data captured by the usability software. However, due to an unexpected illness that sidelined one of us during the scheduled testing sessions, we conducted the majority of testing sessions with only a facilitator.

The usability testing setup consisted of a desktop personal computer connected to a laptop, both installed with Morae. The participant completed the tasks on the desktop computer, while the facilitator administered pre- and post-test survey questions and controlled the recording in Morae from the laptop to avoid distracting the participant. When moving between usability tasks, the facilitator selected each task in Morae and stopped and started the recording to help with data analysis. The facilitator sat to the side of the testing participant, but at an angle so as to not crowd the participant and so the laptop screen was not visible to the participant during testing. As facilitator and observer, we put away our phones and requested participants do the same.

After providing consent, participants completed a brief demographic questionnaire as well as a pre-test survey, followed by an icebreaker task—a library homepage tour. We designed this task to help participants get comfortable with the think-aloud protocol in which participants are encouraged to expose their internal thought processes by talking as they complete tasks and interact with the website (Krug, 2010). The facilitator encouraged participants to think aloud, and the observer—when present—took notes on details not captured by screen or audio recording, such as facial or body expressions or other contextual details, while documenting which tasks the notes referred to. Following the series of tasks, participants completed a post-test survey. At the conclusion of the testing session, we debriefed each participant and gave them a library-branded water bottle as incentive.

To test the library’s LibAnswers FAQ homepage, we created four tasks based on common questions to test participants’ abilities to find needed information in the knowledge base. For all tasks, participants received a card with the task wording printed on it while the facilitator read the task aloud. We did not control how participants could look for the information; they were able to browse by question, topic, or use the search function. See Table 1 for the individual tasks and their wording. We did not share the task names with the participants in print or verbally; as researchers we used them to refer to the tasks internally. At the time of the study, the library’s LibAnswers FAQ page was branded Ask Us.

To protect participant anonymity, we assigned screen and audio recordings and questionnaires an alphanumeric code for cross-referencing. We did not code consent forms for cross-referencing, and we kept them separate from any study materials. We redacted any identifying information that came up during the testing from audio recordings and written transcripts.

Five Second Testing

After completing the five second testing, we transcribed the responses to the questions into a spreadsheet and evaluated the responses to the first question, “What do you think the overall purpose of this webpage is?” to determine whether participants did in fact understand the overall purpose of the page. When reviewing the responses to the second question, “What information is available on this webpage? Please list everything you remember seeing on the page,” we identified a few recurring themes such as whether they noticed the search bar, the topics links, the option to submit a new question, and the Library’s contact information. Results of the five second testing reflected the open-ended nature of the question with responses showing great variation in length, verbosity, specificity, and effort.

When asked about the overall purpose of the page, students generally understood that the library’s LibAnswers FAQ homepage is there to provide answers to common questions, with fifteen out of twenty-one responses mentioning some variation on “questions,” “Q&A,” or “FAQs.” Of the other six responses, five thought the purpose was to talk about the resources the library has to offer. Only one response did not seem to understand the question, responding simply with, “Kinda bland.” When asked to list everything they remembered seeing on the page, ten responses identified the field for question submission and three identified topic-level shortcuts and specific FAQs; only two out of twenty-one respondents identified the search bar and Library contact information. This exemplifies one of the major contrasts between participant responses: generality vs. specificity. When asked, “What information is available on this webpage? Please list everything you remember seeing on the page,” some participants identified general ideas of the page whereas others identified specific aspects.

We can only speculate about whether the lack of specifics indicated the information was not encoded or retained by participants, or if it was due to the wording of the question. For example, it is possible that participants who reported general ideas latched onto the first portion of the question “What information is available on this webpage?” whereas participants who reported specific aspects latched onto the second portion of the question “Please list everything you remember seeing on the page.”

While we had hoped students would observe and remember major navigational aspects of the page such as the search bar and topic level shortcuts, that was not the case. This illustrates important lessons we learned as first-time testers: one, there is not always a direct translation from methods in the literature to your specific context; two, our expectations for what we hoped to see in the results may have colored our feelings about the results we did receive; and three, the test design itself may be flawed. The questions we presented to participants could have been more concise, leaving less room for additional interpretations. After some reflection, perhaps a more straightforward question would have been, “What is the purpose of this page?”

Usability Testing

After we completed the usability test with the six participants, we reviewed the screen and audio recordings individually, noting participant statements and behavior we found interesting or important, such as whether they used the search function versus browsing, search terms employed, and anything participants said in response to our prompting to think aloud. From these detailed notes we—again individually—identified between one to five major takeaways for each participant testing session, focusing on pain points the participants experienced during the tests. Once we completed our individual review of the test recordings, we compiled our major takeaways into a list of the top ten issues identified during the tests, ranked by how often they occurred. From this top ten list, we identified our top two takeaways, which both concerned navigability related to language and naming conventions.

With regard to navigability, to our surprise, almost all participants completed every task using the search box. Provided there was no language disconnect between users’ search terms and language used in the specific FAQs, this was generally a successful strategy. However, this strategy was less successful if participants expected to find their specific search keywords reflected exactly in the FAQ results. If participants did not see these keywords replicated in the results, they resorted to trial and error, at times resulting in erratic, seemingly unfocused or random search behavior. This behavior resulted in two of the six participants failing to complete the Liaison Contact task.

We attribute this to the participants latching onto specific keywords from the task and using those to search the knowledge base. For example, in the Liaison Contact task, both participants initially searched for “economics librarian,” which was the example in the task prompt. The top search result for these keywords was the FAQ “How can I set up a meeting for research?” which contained the relevant information on setting up an appointment with a subject liaison. However, as the FAQ title did not match the specific keywords “economics librarian,” these two participants expressed hesitation and ignored this FAQ. The participants then resorted to scanning through the topic links at the bottom of the LibAnswers FAQ homepage, first selecting Research Help then Library Services then Faculty before ultimately giving up without retrieving the task-relevant information.

Separate from the actual usability tasks but again related to language, three of the six participants were unsure of how to navigate from the library homepage to the library’s LibAnswers FAQ homepage at the very beginning of the test. We attribute this to the inconsistency of naming conventions for the service across the organization’s web platforms at the time. Specifically, on the library homepage and the overall library website navigation bar, the LibAnswers FAQ service was referred to as “Ask a Librarian.” In contrast, on the LibAnswers FAQ homepage itself, the service was titled “Ask Us.” These naming conventions led to additional misunderstanding of the purpose of the page. For example, one participant said that they thought the “Ask a Librarian” link led to a chat feature for immediate, direct contact with a librarian, a service that the library did not provide at that time. To resolve this, the service was renamed “FAQs” on both the library and LibAnswers FAQ homepages.

Naming the Service

First, when it comes to naming the service, users in both our study and the UX @ Harvard (2018) study expected names with “Ask” in them—such as “Ask Us” or “Ask A Librarian”— to immediately connect them to chat, not an FAQ site. As FAQ pages are still widely used across the internet, we feel that users understand what it means and what to expect when they see a link to “FAQs” (see fig. 2).

Figure 2.

Library’s updated LibAnswers FAQ homepage after testing.

Using Expected Language

Second, we found that participants were looking for FAQs that matched the words they used in their search. Participants had the most success when the FAQs had the expected language, such as the FAQ “What are the library hours during the summer?” for the Summer Hours task, versus more esoteric language, such as the FAQ “How can I set up a meeting for research?” for the Liaison Contact task.

Describing Topics

Third, topics should be more descriptive and their purpose more clear. For example, in the Liaison Contact task, one participant browsed under the Faculty topic, equating it with information about library faculty and not information for teaching faculty. An easy solution here would be to make the topic more descriptive by calling it Information for Faculty (see fig. 2).

However, what words and phrases are going to be most useful to users may be hard to determine without user input, so we suggest focusing energy into card sorting or other research methods to uncover how users would categorize FAQs. Regardless of whether that is feasible, we suggest mapping FAQs to all topics the questions relate to, so they have greater likelihood of showing up where users are looking.

Additionally, in both our study and the UX @ Harvard (n.d.) study, participants who used the topics did not fully understand their purpose. UX @ Harvard recommends wording like “FAQ Topics” and “Question Search Topics”; we suggest “FAQ Categories” as another possibility (see fig. 2).

Conducting Usability Testing: Recommendations for Researchers

For other researchers interested in performing usability testing to improve their own LibAnswers FAQ pages, we can offer some recommendations. First and foremost, a specialized lab or software such as Morae is not required to run a usability test. We used the lab and Morae software because it was freely available to us on our campus—and we wanted to try it—but in truth, we would have been able to perform every aspect of the usability testing without it.

According to Krug (2010), all that is needed to run a usability test is a computer with internet access, screen recording software, a mouse, and a microphone plus an external monitor, keyboard, and mouse which helps the facilitator set up tasks without reaching into the participants’ space. You can create this basic setup with equipment and software already available in most libraries. Using a specialized setup and software does offer more features, but this also increases the learning curve.

The second recommendation is to select methods wisely, and consider streamlining the number of methods you use. Usability testing is ultimately about making improvements to your products, services, or interfaces, for your user base, based on problems that they are experiencing. While takeaways from others’ usability testing can be informative—and we hope you find ours informative—results are not intended to be widely applicable outside of the context in which they were gathered.

With that in mind, usability studies do not have to be complexly designed with a multi-method approach. One quick round of testing, using one method, will generate results that will provide insight, even if that insight is how to do it better next time. As Krug (2010) puts it, “Doing this kind of testing is enormously valuable if you do it, and people don’t do it because they have the impression that it’s more complicated than it needs to be” (p. 11).

The methods we selected for our study were the ones that we felt would best help us uncover what we wanted to know based on the literature we were reading at the time. We also wanted to try them and gain the experience of running these tests, especially the usability testing. There are myriad methods for UX research, and the ones we present are by no means the only ones. It is important to select methods based on your own research objectives. One resource to get started is a handy chart from Anderson (2020) published on the UX Collective blog.

Five Second Testing Protocols

[P. 1] Testing Instructions

On the next page, you have a paper version of the SMCM Library Ask Us homepage. Take approximately five seconds to view the homepage. Once you’ve viewed the homepage, flip to the final page to complete the task.

[P. 2] Image of LibAnswers FAQ Homepage (see fig.1)

[P. 3] Task Questions

Now that you’ve viewed the Ask Us homepage, please answer the following questions:

• What do you think the overall purpose of this webpage is?

• What information is available on this webpage? Please list everything you remember seeing on the page.

Icebreaker & Homepage Tour Script

(Adapted from Krug (2010) and Hungerford et al. (2010) procedures)

Hi! Thank you for taking part in our research study. My name is __________, and I’m going to be walking you through this session today. If I could ask you at this time to silence and put away any electronic devices during the session. If you have a snack, please set that aside until the session is over. [Observer asks participant to put snack on different table.]

Before we begin, I have some information to share with you, and I’m going to read it to make sure I cover everything. Your participation today will help the Library make student-centered, informed decisions on how to best update the websites under study. Decisions made as a result of the study will be publicized on the Library website and through All Student emails. If you are interested in discussing the research further, please contact [researcher email and phone number].

We’ve asked you to come in today to help us evaluate the Library’s Ask Us web resources. We want to see if the sites work as intended, and we will be asking you to perform several tasks and will observe you while you do them. The session should last about an hour.

First, I want to make it clear that we are testing the site, not you. You can’t do anything wrong here. In fact, this is probably the one place today where you don’t have to worry about making any mistakes.

As we go through the tasks, I’m going to ask you, as much as possible, to try to think out loud: say what you’re looking at, what you’re trying to do, and what you’re thinking. We really want to hear what you honestly think, so please do not worry about hurting our feelings. We’re trying to get as much honest information as possible so we can improve the webpages. Your thoughts and input will be a big help to us.

If you have any questions as we go along, please feel free to ask them. I may not be able to answer them right away, since we’re interested in what people do when they don’t have someone sitting next to them who can help, but I will try to answer any questions you may still have when we are finished. Also, if you need a break at any point, please just let me know.

One last thing: we would like to record what happens on the screen and through the microphone. We want to record this session so we can later review what happened and take better notes. We don’t use our recordings for purposes other than helping us learn about the site, and there will be no identifying information in any of the recordings, so your participation will be anonymous.

If you would, I’m going to ask you to sign a simple consent form that just says that you are willing to take part in this study. After that if you could fill out a brief demographic questionnaire and short survey on how you use the Ask Us. After the test there will be another very short follow-up survey.

(Participant signs consent form)

(Give participant demo questionnaire and pre-test survey)

(Collect and move into homepage tour)

Just to get started, let’s take 5 minutes to take a look around this webpage and practice thinking aloud. Here are some ideas of things to let me know:

• What is your overall impression of this page?

• What do you think the overall purpose of the page is?

• Where do you think you can navigate to from this page?

• What aspects of this page do you notice first?

• What aspects of this page do you find confusing?

• If you could change anything about this page, what would it be?

• Is there anything you find most helpful on this page?

Try to think aloud as much as possible; I might also ask you what you are thinking occasionally. And again, we want as honest feedback as possible. Nothing you say is wrong or will hurt our feelings in any way!

(Encourage participant to think aloud)

(After 5 minutes) Ok great! Now I’m going to present you with a series of specific tasks. I’m going to read each one out loud and give you a printed copy. Again, as much as possible, it will help us if you can try to think aloud as you go along through these tests.

Let’s get started!

(See Ask Us testing scenarios to continue)

Usability Testing Tasks (Facilitator Copy)

NOTE: Participants received a printed copy with only the BOLDED scenario prompt

Task 1: Liaison librarian contact (Facilitator - read the BOLDED text. For this section it is OK if participant uses search box)

(FACILITATOR READ:) You need help finding statistics for an economics paper you’re writing. You wonder if there is an Economics librarian who you could make an appointment with. Use the Ask Us to find out.

(Prompt participant to navigate back to LibAnswers homepage)

Task 2: View previous St. Mary’s Projects (Facilitator - read the BOLDED text. For this section it is OK if participant uses search box)

(FACILITATOR READ:) Your SMP advisor suggested checking out some past SMPs for ideas. Use the Ask Us to find out how you can access previous students’ SMPs.

(Prompt participant to navigate back to LibAnswers homepage)

Task 3: Summer hours (Facilitator - read the BOLDED text. For this section it is OK if participant uses search box)

(FACILITATOR READ:) You’re planning on staying in the area this summer and want to know when the library will be open. Use the Ask Us to find out what the SMCM Library’s summer hours are.

(Prompt participant to navigate back to LibAnswers homepage)

Task 4: Printing (Facilitator - read the BOLDED text. For this section it is OK if participant uses search box)

(FACILITATOR READ:) Arg, GoPrint deducted your print money but your document never actually printed! Use the Ask Us to find out if you can get your money back.

(Prompt participant to navigate back to LibAnswers homepage)