Usability

The International Organization for Standardization defines usability as the “extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (2018, 3.1.1). Originally, companies designed usability questionnaires to analyze the potential benefits of new products or systems before investing time and money into their development (Chung & Sahari, 2015). Most usability questionnaires to assess how effectively users learn to use a system or to evaluate a product’s usability. Usability testing has historically focused on functionality, but researchers are increasingly recognizing the importance of including aesthetics and hedonic value when evaluating perceived usability (Chung & Sahari, 2015). However, Chung and Sahari analyzed and compared eleven usability questionnaires and observed that none considered aesthetics.

Extensive experience with interactive devices has led to heightened user expectations of intuitive interfaces and bold visual designs (Hinderks et al., 2018; Schrepp et al., 2014). Research has increasingly recognized that “user’s needs go beyond usability” and that the entire UX, including visual aesthetics and emotion, should be considered (Moshagen & Thielsch, 2010, p. 691; 2012).

User Experience

The International Organization for Standardization defines UX as a “user’s perceptions and responses that result from the use and/or anticipated use of a system, product or service” (2018, 3.2.3). UX holistically considers user’s reactions during product interactions and combines usability criteria with “hedonic quality criteria, like stimulation, fun-of-use, novelty, emotions or aesthetics” (Schrepp et al., 2014).

Questionnaires are a common, efficient, and inexpensive method for collecting user data (Schrepp et al., 2014; see also Díaz-Oreiro et. al., 2019; Hinderks et al., 2018; Laugwitz et al., 2008). Researchers can distribute questionnaires easily to large groups of users and allow participants to respond at their convenience (Díaz-Oreiro et. al, 2019; Hinderks et al., 2018). Standardized questionnaires are reliable, validated, quantitative measurements that allow for efficient data analysis (Díaz-Oreiro et al., 2019; Hinderks et al., 2018; Laugwitz et al., 2008).

UX questionnaires are a cost-effective way to determine and prioritize design issues (Lund, 2001). As their use has become more widespread, users are less likely to participate when questionnaires are lengthy and time-intensive (Hinderks et al., 2018). Additionally, users’ experiences are subjective and can vary extensively (Chung & Sahari, 2015; Hinderks et al., 2018; Schrepp et al., 2014). Although companies often overlook subjective reactions to product usability, these reactions are strongly correlated with aspects of UX, such as user behaviour and purchase decisions.

Díaz-Oreiro et al. (2019) analyzed 553 academic studies to examine the use of three standardized UX questionnaires: AttrakDiff, meCUE, and UEQ. The AttrakDiff questionnaire was the most frequently used, likely because it was developed first in 2003, followed by the UEQ in 2008 and meCUE in 2013. Although only twenty studies were conducted in North America, seven were recent, having been published in 2017 or 2018. This suggests an increased use of standardized UX questionnaires in North American research. While all three questionnaires saw increased usage, the UEQ rose more rapidly, suggesting that it was becoming the preferred questionnaire for researchers.

User Experience Questionnaire (UEQ)

In 2008, Laugwitz et al. developed the User Experience Questionnaire (UEQ) as a quick and comprehensive assessment tool that allows users “in a very simple and immediate way to express feelings, impressions, and attitudes that arise when experiencing the product under investigation” (p. 64; see also Schrepp et al., 2014). Since its inception, the UEQ has been widely used in many studies, and it has become “an established and frequently used questionnaire for the evaluation of the UX of interactive products” (Hinderks et al., 2018, p. 374). In their review of 553 standardized UX questionnaire studies, Díaz-Oreiro et al. (2019) found that 200 of these studies employed the UEQ.

The UEQ collects immediate and spontaneous feedback, rather than responses requiring “deeper rational analysis,” and is based on a theoretical research framework that “distinguishes between perceived ergonomic quality, perceived hedonic quality and perceived attractiveness of a product” (Laugwitz et al., 2008, p. 65). Ergonomic or pragmatic quality refers to task-oriented facets, while hedonic qualities refer to non-task-oriented facets such as product aesthetics, or whether a product is perceived to be boring or interesting (Díaz-Oreiro et al., 2019; Laugwitz et al., 2008). The UEQ evaluates both hard, pragmatic usability criteria, and soft, hedonic UX criteria, which are of “similar relevance for the end user” and have not been adequately represented in other UX questionnaires (Laugwitz et al., 2008, p. 73; Schrepp et al., 2014).

The UEQ is comprised of six scales (Hinderks et al., 2018, p. 374; see also Schrepp et al., 2014):

• attractiveness, or “the overall impression of a product”

• perspicuity, or “how easy it is to learn or understand”

• efficiency, or whether users are able to “solve their tasks without unnecessary effort”

• dependability, or how predictable or secure the user feels in the interaction

• stimulation, or how “exciting and motivating” it is to use the product

• and novelty, or how innovative or creative the product is

Validated standardized questionnaires often use “Likert scales or semantic differentials” (Díaz-Oreiro et al., 2019, p. 1), or “two terms that describe the opposite ends of a semantic dimension” (Schrepp et al., 2021, p. 1), to gather users’ perceptions of a product’s characteristics and its impact on the user (Laugwitz et al., 2008). The UEQ asks participants to rate two semantic differentials on opposing sides of a seven-point Likert scale (Hinderks et al., 2018; Schrepp et al., 2021). Users can complete this questionnaire in 3–5 minutes, and researchers can analyze the results using the free online handbooks and Excel sheets available through UEQ: User Experience Questionnaire (Schrepp et al., 2014). The questionnaire is available in over thirty languages, and native English speakers translated and tested the English version (Laugwitz et al., 2008).

Schrepp et al. (2014) examined the reliability and validity of the UEQ “in 11 usability tests with a total number of 144 participants and an online survey with 722 participants” (p. 3). The authors stated that the studies showed consistent reliability and good construct validity of the UEQ scales, as measured by Cronbach’s Alpha (Schrepp et al., 2014; Schrepp, 2023). Cronbach's Alpha measures “the consistency of a scale” and “indicates that all items in a scale measure a similar construct” (Schrepp, 2023, p. 10). In 2022, Schankin et al. published a paper evaluating the reliability and validity of the UEQ through two studies, noting that “only a few studies reported about its reliability and validity so far” (p. 2). They found that “the UEQ can measure hedonic and pragmatic quality aspects as well as an overall attraction of a product in a reliable and valid way” (p. 10).

Hinderks et al. (2018) created a short version of the UEQ, known as UEQ-S, to address scenarios in which completing the full questionnaire could be tedious and potentially impact the quality of responses. The perceived attractiveness criteria is not included in the UEQ-S, which is one of the two main item sets in the full version of the UEQ. The UEQ-S consists of eight items that equally measure pragmatic and hedonic qualities, with the total mean value used as a measure of the overall UX. However, researchers should interpret this value carefully, as the importance of pragmatic qualities in relation to hedonic qualities may differ in practice from the ratio used in the UEQ-S overall mean value (Schrepp, 2023). The authors conducted an evaluation study comparing the UEQ-S to the full UEQ to validate the short version (Schrepp et al., 2017). They found that “the consistency of the pragmatic quality and hedonic quality scales was reasonably high” and “seems to approximate the long version expectedly well” (p. 106).

Researchers have combined methods “in innovative ways to further our understanding of user experience” (O’Brien, 2016, p. 27). Díaz-Oreiro et al. (2019) found that 61.5 percent of the studies they reviewed employed more than one evaluation method. Similarly, Laugwitz et al. (2008) stated that “in general, user questionnaires have to be combined with other quality assessment methods to achieve interpretable results” (p. 63). For example, Aufderhaar et al. (2019) used both the UEQ and VisAWI-S questionnaires in their study of potential gender differences in users’ perceptions of three websites.

Visual Aesthetics

Aesthetics, or the importance of beauty, has been perceived and understood since antiquity (Lavie & Tractinsky, 2004). Scholars have debated whether aesthetics is an objective or subjective property, and current theories tend to include both perspectives. Psychologists and researchers suggest that people perceive objects as more beautiful when they are easier to mentally process (Moshagen & Thielsch, 2010). Research on the relationship between visual aesthetics and perceived usability has increased, with findings suggesting that visual aesthetics may improve perceived usability (Lavie & Tractinsky, 2004; Moshagen & Thielsch, 2010). While primarily an aesthetic attribute, simplicity is fundamental to “creating usable systems” and enhances the connection between usability and aesthetics (Lavie & Tractinsky, 2004, p. 277).

Evidence shows that beauty is a significant determinant of website preference, and aesthetics are crucial to user satisfaction (Lavie & Tractinsky, 2004). Although fundamentally important, aesthetic considerations are often overlooked or undervalued in studies of human-computer interaction. However, it is essential that visual aesthetics in these interactions are “adequately assessed” (Moshagen & Thielsch, 2012, p. 1305).

Visual Aesthetics of Website Inventory (VisAWI)

Moshagen and Thielsch (2010) created the VisAWI as a validated instrument for measuring perceived visual aesthetics of websites. They based their questionnaire construction on interviews with experts and users, aiming to make it applicable to both research and industry contexts. They created a model with four facets: simplicity, diversity, colorfulness, and craftsmanship. While these facets collectively represent perceived visual aesthetics, they “are still distinguishable from each other” and independently respond to changes in website design properties (p. 705). Three studies were conducted to determine the “various types of validity” of the VisAWI (p. 693). Overall, the evidence from these studies suggested that the VisAWI is a reliable measure “for a precise assessment of perceived visual aesthetics of websites” (p. 706).

A limitation of the VisAWI is that although it takes less than three minutes to complete, it may still be too lengthy in specific research settings (Moshagen & Thielsch, 2010, 2012). Moshagen and Thielsch (2012) constructed a shortened version of the VisAWI (VisAWI-S) by selecting one item that best represented each of the four facet: simplicity, diversity, colorfulness, and craftsmanship. While the VisAWI-S has lower reliability than the full version, it is “still sufficiently reliable for most purposes” and provides “a very close approximation to perceived visual aesthetics as measured by the full VisAWI” (p. 1310). Although both versions were originally created in German, the English version is considered “grammatically and semantically equivalent” (p. 1307).

Recently, researchers have adapted versions of the VisAWI in Arabic, Farsi, and Indonesian (Abbas et al., 2022; Saremi et al., 2023; Sadita et al., 2022). The Indonesian version examined “graphical interface aesthetics” in video games and found the VisAWI to be reliable and valid in this context (Sadita et al., 2022, p. 382). The VisAWI has also been recognized as a standardized UX questionnaire (Hinderks et al., 2019; Schrepp et al., 2021).

Online Library Instructional Resources

Online information literacy resources have expanded the range of library instruction methods available to students and become increasingly popular (Fernández-Ramos, 2019; Walters et al., 2014). With the growth of online and distance learning in higher education, online information literacy tutorials help to address these student needs (Blummer & Kritskaya, 2009; Bury & Oud, 2005; Fernández-Ramos, 2019; Su & Kuo, 2010; Zhang, 2006; see also Goodsett, 2020; Olsen & Harlow, 2022). Online tutorials are accessible anytime and anywhere, allowing students the flexibility to learn independently at their own pace and revisit the content as needed (Fernández-Ramos, 2019; Su & Kuo, 2010; Walters et al., 2014; Zhang, 2006; see also Bury & Oud, 2005). They can also be updated and revised over time, providing long-lasting value to students (Goodsett, 2020). As librarian capacity to provide in-person instruction is increasingly limited, online tutorials offer an effective time-saving alternative (Fernández-Ramos, 2019; see also Walters et al., 2014).

Online library tutorial interfaces must be well designed in order to sustain user interest and engagement (Zhang, 2006). During tutorial development, creators should identify objectives based on pedagogical principles or standards to ensure a focus on teaching and learning (Blummer & Kritskaya, 2009; Walters et al., 2014; Zhang, 2006). Effective tutorial designs consistently apply website design guidelines for elements such as text, headings, color, and visuals (Walters et al., 2014; Zhang, 2006). Franklin et al. also emphasized the importance of incorporating equity, diversity, and inclusion considerations when designing online learning tools to reflect the varied learning experiences of students (2021).

Text is a fundamental element of online instruction (Zhang, 2006). Designers can use text font, size, and color to emphasize specific words or sections. Research shows that limiting text to short paragraphs and minimizing content per page can improve readability. Applying color consistently reinforces the content’s framework, helping students understand the organization and relationships between content while effectively highlighting information.

Visual images can aid the learning process and transform “a boring, lifeless web tutorial into a set of compelling and interesting online instructions” (Zhang, 2006, p. 299). They can help to break up sections of static text and reduce the cognitive load for users. Images “should be carefully designed to capture students’ attention, orient them for learning, and encourage them to actively participate in the learning process” (p. 297). Using multimedia has become increasingly popular in online library tutorials as a method to engage and motivate students (Su & Kuo, 2010; Zhang, 2006; see also Olsen & Harlow, 2022). While videos can enhance learning, overuse can lead to internet bandwidth constraints for users (Su & Kuo, 2010; Zhang, 2006).

Immediate feedback is essential in online interactive tutorials, as it enables students to assess their progress (Zhang, 2006; see also Blummer & Kritskaya, 2009; Walters et al., 2014). Quizzes are an interactive and engaging method for providing feedback while evaluating students’ understanding of the tutorial content (Blummer & Kritskaya, 2009; Walters et al., 2014; see also Zhang, 2006). Multiple-choice questions are common, with research suggesting that “well-crafted multiple-choice questions can reliably and validly measure higher order thinking skills” (Goodsett, 2020, p. 4).

Practitioners can evaluate the usability of online library tutorials through pilot implementations, usability tests, surveys, and anecdotal observations, with user comments being “one of the easiest methods of evaluation” (Blummer & Kritskaya, 2009, p. 209). Student feedback “is commonly recognized to help instructors improve” and revise online information literacy tutorials (Su & Kuo, 2010, p. 327). A small sample size of users is sufficient for testing, as these users “are likely to encounter all of the most significant problems” (Bury & Oud, 2005, p. 56; Franklin et al., 2021). User feedback is invaluable for tutorial content design and can significantly improve student learning (Bury & Oud, 2005; Franklin et al., 2021).

UEQ and VisAWI Application in Libraries

Few studies have applied the UEQ or VisAWI in academic library research. Yi and Kim (2021) used the UEQ and the VisAWI to evaluate participants’ museum experiences, both in person and with wearable mixed reality technology. Saleh et al. (2022) employed the UEQ to evaluate the experiences of participants from several Jordanian academic institutions with the Moodle learning management system. Kuhar and Merčun (2022) used the UEQ-S, alongside an emotion-related questionnaire, to compare the UX of two digital libraries. They noted that hedonic qualities “have often been overlooked in the evaluation of digital libraries” and found that both questionnaires worked well in the digital libraries context (p. 3). They also observed that perceived aesthetics was “an aspect that was missing in the researcher’s interpretation of the results” (p. 10).

The studies above applied the UEQ, and in one instance the VisAWI, to evaluate the UX of a learning management system, mixed reality in museums, and digital libraries sites. However, there has been little to no use of the UEQ in academic library UX research. While studies of online library instruction tutorials have occurred, they have primarily focused on usability rather than the user’s experience. To date, no studies have employed the UEQ or VisAWI to measure the UX of online library instruction tutorials. Additionally, the UX of library tutorials created with LibWizard, a library-specific software tool, has not been studied in the current literature.

Participants

In the User Testing phase, some students who expressed interest in participating did not complete their assigned tutorials, resulting in an unequal distribution of participants across the tutorials (see Table 1). I excluded participants who did not participate in the second round from the quantitative data analysis. In the Implementation phase, some participants repeated tutorials to improve their grades, so the total number of participants was unknown.

User Testing Phase: User Survey Feedback

Participants gave qualitative feedback on the tutorial design and their experience with the tutorials, including specific errors and issues, through a user survey.

The user survey questions that received relevant responses are:

• What did you like the most about taking this tutorial?

• What did you like the least about taking this tutorial?

• Was there anything you preferred or missed from the previous version of this tutorial?

• Do you have any additional comments you would like to make about this tutorial?

I anticipated that embedding YouTube videos in the tutorials would offer a welcome break from reading the textual content. However, some users found watching multiple videos time consuming and suggested adding closed captions to improve accessibility. In response, I reduced the number of videos and added instructions for enabling closed captions to the remaining videos. One participant in the second round provided positive feedback to the closed captions change.

The first round tutorials contained pop culture references as answer options for multiple choice questions. These inclusions received negative feedback, primarily from faculty members. After removing most of the references, one participant remarked that they missed the amusing pop culture answers.

Although the first version of the Evaluating Resources tutorial was considered informative and interactive, three participants mentioned that some sections were too easy or “not very complicated.” Participants also found the article at the end of the tutorial, to which they applied their evaluating skills, too lengthy. I removed this article and embedded articles throughout instead. Participants found the second version “more enjoyable to complete,” with improved pacing.

The first version of the Searching for Resources tutorial had multiple issues. Several databases could not be embedded into the tutorial, so students needed to open them individually. One participant commented that “the opening and closing of tabs got tiring.” These links sometimes opened in the same tab as the tutorial, forcing participants to restart the tutorial from the beginning. Participants found the instructions complicated and hard to follow, and they struggled with using specific subject filters and finding the correct answers in their searches. In the second version of the tutorial, I minimized the use of database links, clarified instructions, and added more instructional screenshots to demonstrate tasks such as locating subject filters. Participants described the second version as “a nice improvement” and noted that “the changes that were implemented helped a lot.”

Participants in the first round of the Citing Resources tutorial appreciated its step-by-step approach and “good, relatable examples throughout,” but they had difficulty focusing on the “very dense material” and found the amount of reading time-intensive. In the second version, I spread the text across more pages, with each page focusing on a specific citation style. Although feedback still mentioned the dryness of the tutorial content and that “it took quite a bit of time” to complete, participants felt the second version was “definitely an improvement over the old tutorial.” One participant commented, “The information was excellent. I wish I could print it out as a manual.”

User Testing Phase: UEQ-S

I analyzed the results of the UEQ-S using the UEQ handbooks and Excel sheets, which are freely available through the UEQ: User Experience Questionnaire. The UEQ-S item values range from -3, the most negative, to +3, the most positive (Schrepp, 2023). For the UEQ-S data analysis, mean values greater than +0.8 indicate a positive evaluation, values between -0.8 and +0.8 indicate a neutral evaluation, and values less than -0.8 indicate a negative evaluation. In the following tables, the items represent both pragmatic and hedonic qualities of the user’s experience, with overall mean values calculated for each. The overall UEQ-S mean value of the eight items represents the overall UX (Hinderks et al., 2018).

In the User Testing phase, I conducted paired t-tests to determine whether the differences between versions of the same tutorial were statistically significant. I found statistically significant differences for the Searching for Resources and Evaluating Resources tutorials, but no statistically significant differences between the Identifying Resources and Citing Resources tutorial versions.

In the first version of the Searching for Resources tutorial, all of the item mean values fell within the neutral range (see Table 2). The second version showed considerable improvement, with six of the eight items having mean values in the positive range. For the Evaluating Resources tutorial, the first version had only one item, clear, with a mean score in the positive range. In the second version, all items improved except for clear, which stayed in the positive range. Notable improvements occurred in the pragmatic qualities of supportive and easy, as well as in the hedonic qualities of exciting and interesting.

All of the second tutorials had positive overall UEQ-S mean value results above 1 (see Table 3). The Identifying Resources tutorial had the highest overall UEQ-S mean values in each round, with participants rating both versions positively for pragmatic and hedonic qualities. The Citing Resources tutorial also garnered positive overall UEQ-S values for both tutorial versions. The Searching for Resources tutorial showed significant improvement, with its overall UEQ-S value increasing by 1.227. The Evaluating Resources tutorials also improved, with its overall UEQ-S value increasing by 0.727. None of the tutorial versions in the User Testing phase had any item or overall mean values in the negative range.

User Testing Phase: VisAWI-S

The VisAWI-S has four questionnaire items, each representing one of the four facets: simplicity (everything goes together in this tutorial), diversity (the layout is pleasantly varied), craftsmanship (the layout appears professionally designed), and colorfulness (the color composition is attractive). The overall mean value can represent “a general factor of aesthetics,” but since there are only four items, they cannot be used to individually represent the facets (Thielsch & Moshagen, 2015, p. 15). Each item of the VisAWI-S is rated on a seven-point Likert scale, with 1 indicating strongly disagree, and 7 indicating strongly agree (Moshagen & Thielsch, 2012).

According to the VisAWI-S questionnaire, participants in the User Testing phase perceived an improvement in the visual aesthetics of each tutorial (see Table 4). The second versions of all the tutorials had VisAWI-S mean values exceeding 6, indicating that the aesthetics were generally pleasing, with notable improvements in the Searching for Resources and Evaluating Resources tutorials.

Implementation Phase: UEQ-S

In this phase, participants completed the final versions of the tutorials, which I had slightly modified following the completion of the User Testing phase. A total of 124 tutorials were completed, with the numbers for each of the four tutorials shown in Table 1. The items supportive and leading edge received positive mean values in all of the Implementation phase tutorials (see Table 5). Three of the four tutorials had positive mean values for interesting, while the mean values for exciting were neutral across all tutorials. Inventive received the lowest mean values overall, with the Citing Resources tutorial having the only negative item mean value in the study (-1.3).

In the Implementation phase, the UEQ-S overall mean value for the Identifying Resources tutorial was the most positive by a substantial margin (see Table 6). For the Identifying Resources tutorial, participants rated the pragmatic qualities positively, while they rated the hedonic qualities neutrally. The Searching for Resources, Evaluating Resources, and Citing Resources tutorials all received neutral scores for both pragmatic and hedonic qualities. These three tutorials also had neutral overall UEQ-S mean values, suggesting neutral user experiences.

Implementation Phase: VisAWI-S Results

The VisAWI-S overall values for the tutorials in the Implementation phase ranged from 5.78 to 6.22 (see Table 7). The VisAWI-S mean values were above 5.7, indicating that the aesthetics were pleasing overall, with the Identifying Resources tutorial having the highest visual aesthetics value.