Gamification Mechanics

The definition of games or gaming is dependent upon the industry (Pagulayan et al., 2008), and can set the tone of the product. Commercial games for the entertainment industry do not have to balance entertainment with learning objectives, as do educational games. “Games are usually more colorful, wacky, and escapist than productivity applications, with the inclusion of interesting storylines or animated characters” (Pagulayan et al., 2008, p. 742).

Sailer et al. (2017) defined gamification as the implementation of game design elements for non-gaming purposes. Their case study identified seven key elements that are visible to players: points, badges, leaderboards, performance graphs, meaningful stories, avatars, and teammates. Usage of such game design elements “modifies that environment, thereby potentially affecting motivational and psychological user experiences” (p. 374). These elements suggest that gamification may affect motivational outcomes and can be a solution for learning contexts if well-designed.

Elements of interaction, engagement, and immersion are key to a gamified experience (Gunter et al., 2008). The field of human-computer interaction is both user-centered and iterative, using common practices of ethnographic research, prototyping, and testing. In general, a prototype is a representation of an idea, and prototypes can take many forms, from a simple sketch on paper to a fully functioning artifact. Prototypes are quick and inexpensive ways to uncover design issues before significant time and money are put into the development of the final product. Prototype design and playtesting support human-computer interaction in game design by providing users with an early look at the final product, as well as allowing designers the opportunity to identify functional requirements and performance issues early on (Beaudouin-Lafon & Mackay, 2008).

As compared to usability tests or online survey research, playtests are advantageous to game designers because they provide actionable feedback derived from users’ hands-on play experience (Pagulayan, 2008). The primary focus is on perceptions and supports the iterative process key to human-computer interaction and UX design.

Most successful games have a storyline, allowing players to have a sense of engagement. There are two main perspectives: the designer’s story and the player’s story (Rouse, 2001). The designer’s story is the storytelling built into the game, cueing the player as to purpose and content: “Some games have stronger designer’s stories, such as adventure and role-playing games, while others have little to no designer’s story, such as classic arcade games like Pacman and puzzle games like Tetris” (Winn & Heeter, 2007, para. 12). The player’s story is present in every game, and the storyline drives it. Players find a storyline to be most meaningful when it engages their social groups; games can accomplish this objective through ethnographic research-based persona creation. A persona is based on a fictional character and is derived from a profile of social attributes of the target audience; game developers then use these personas to drive storylines and player goals.

Other successful characteristics of game design are replayability (Prensky, 2001) and scoring; these are immersive activities whereby players feel connected to the game. Developers boost replayability through multiple avenues to completing the game’s goals as well as the insertion of random elements into the game, stimulating player interest (Winn & Heeter, 2007). Points or a scoring system can replace the simple count of right versus wrong choices and heighten the feeling of competitive play.

Commercial gaming relies on an iterative process of prototyping and playtesting (Salen & Zimmerman, 2006), which is even more necessary for designing great learning games that present complexity in their pedagogy (Winn & Heeter, 2007). Regardless of game purpose, playtesting helps designers refine their game’s mechanics by moving from the theoretical to actualized successes or failures. Although game-based learning is frequently associated with video games, libraries often prefer to focus on a more general application of both UX methods and digital games (Long, 2017).

UX and Design Thinking in Libraries

UX and design thinking have increasingly entered literature in the context of libraries in the past ten years, and can help libraries address usability concerns and adapt to changes in user behaviors and preferences. For example, the University of Technology Sydney adapted the design thinking process from Stanford’s design school to create a five-step process for improving wayfinding in academic libraries (Luca & Narayan, 2016). Porat and Zinger (2018) discussed their success of improving the library’s discovery tool through usability studies. Library literature also discusses the use of personas in understanding the library user’s experience (Sundt, 2017; Zaugg & Ziegenfuss, 2018) and how the psychology of UX can help us question assumptions about typical library users (Fox, 2017). These are just some of many examples of applying UX methods, including wayfinding, navigation and discovery, to enhance the experiences of library users.

User-centered design, UX, and library games often appear together in the literature. Somerville and Brar (2009) discussed the successful outcome of a user-centered approach to digital library projects, one that emphasized focus groups, ethnographic studies, and participant observation in addition to iterative design and prototyping. The design experience in this study served to acquaint its librarians with systems design and usability testing principles, new roles for these information professionals. Addressing the user perspective can therefore not only help library users but aid in the further professional development of the library staff.

Gamified Library Experiences

Narrowing in on library games yields examples of gamified learning activities focused on the research process (Long, 2017), library instruction (Battles et al., 2011), plagiarism, library anxiety (Reed & Caudle, 2016), information literacy (Guo & Goh, 2016), and library orientation or tours (Smith & Baker, 2011). The literature shows a high use of prototyping in university libraries (Long, 2017; Guo & Goh, 2006; Ward, 2006), however these were primarily for website design or website usability projects and not for gamification of library activities.

Long (2017) described how librarians at Miami University aspired to create an engaging and interactive tutorial that was easy to build and maintain with little learning curve. In order to address the challenge of teaching the research process in an online tutorial format, the library ran pilot tests for two text-based tutorials—with Google Sites and Twine—and a video-based tutorial with Adventur. Although gamifying the research process through these pilots proved fruitful for promoting fun and learning support, Long pointed out the need for future adaptations to address evolving technologies and accessibility concerns.

Librarians at Utah Valley University (UVU) created two self-paced library orientation games (Smith & Baker, 2011). Students played the game LibraryCraft online; it guided them around the library’s website in a type of medieval quest based on World of Warcraft. The authors solicited feedback from library student employees on many factors: artwork, storylines, wording of text, and overall UX. Of the fifty-two survey respondents, 84.5 percent indicated that the game taught them a new library skill, while 56 percent reported difficulty using the game. Interestingly, LibraryCraft was not as popular as the library’s physical game, Get a Clue; the UVU authors attributed this difference to the complexities of digital development and launching an online game during a busy time of the year. Perhaps most impressive about the game development at UVU was their communication plan and small focus groups with students (Baker & Smith, 2012). Additionally, this online game did not require a download in order to play, making it more accessible to library users.

Guo and Goh (2016) employed a user-centered design process to create an information literacy game to counteract the increasing numbers of students unwilling to approach librarians when encountering difficulty searching. While designing Library Escape, the authors facilitated design workshops, created prototypes, and solicited feedback. The design workshops had participants familiarize themselves with two existing game-based learning activities at different institutions; participants then discussed what they liked and disliked about these games, which informed the design of Library Escape. Once completed, Library Escape user testing feedback expressed the need to change presentation formats for this role-playing game, avoiding time-consuming introductory videos to allow the gamer to start play quicker. The game itself could take an hour or more to play, a large time commitment for players. It focused on library instruction rather than library orientation.

University of Kansas students designed a digital game for the university libraries to help address library anxiety among undergraduates; it was implemented in the first-year experience curriculum (Reed & Caudle, 2016). The student developer team used Twine software, wrote the game storyline, and created animated GIFs. Impressively, the game is still active and accessible to users not affiliated with the university. Another noteworthy feature is that in addition to the HTML file used to play the game in a browser, the authors also made a README file available which explains the game in more detail.

The University of Alabama Libraries created the web-based alternate reality game, Project Velius, as an interactive form of traditional library instruction (Battles et al., 2011). The authors cited several resourcing challenges in producing the game, such as librarian lack of experience with online game development, as well as librarian workload—developers had to produce the game in addition to their regular duties.

Although there are several citations in the literature on the use of gamified experiences in libraries, those studies tend to focus on the context of the game—for example, research instruction—and not on the development process. Libraries tend to describe the results once they launch a product or service, and not the process it took to get there. Most significantly, there is a lack of discussion of UX design methods, particularly of iterative prototypes and playtesting prior to final product launch. We seek to fill this gap by documenting the development approach, including the iterative prototyping and testing phases of LibGO.

Empathize

The first phase involves empathizing with users. For libraries this means learning, observing, and understanding how library users feel about a product or service. Literature demonstrates that the “not knowing” feeling is common, especially among users who have never participated in library instruction classes that may provide a brief orientation and tour of the library. As applied to LibGO, we found overwhelming evidence of users not knowing which library resources were available to them. We collected these frustrations over the years through librarian observation and student survey feedback. Library orientations can counteract the library anxiety—including the struggle of how to locate buildings, items, and services—affecting new and returning student and faculty library users (Luca & Narayan, 2016).

Define

The second phase involves defining the problem. With the knowledge gained by empathizing with the library users in the first phase, we reflected on user behaviors, needs, and desires. We defined the problem as: reaching all types of library users with a traditional, in-person library orientation is not possible. How do we reach users beyond first-year students, who are the main recipents of in-person orientation and how do we make the orientation information engaging and relevant?

In this phase, we also focused on the project broadly with game objectives, logistics, event launch, budget, staffing, and documentation of processes and workflows (see appendix B). At this point, there was no game development, but rather a step of defining the processes, workflows, and division of responsibilities needed to achieve the goals of helping the library users. For example, we defined launching and marketing of the product in the workflows. We had to decide whether to help users with the defined problem through an event, an online environment, or even both. These event and online experiences required preparations to target potential users and included tasks such as recruitment emails and marketing drafts, institutional review board approval, and event design.

We also addressed potential staff needed throughout the process in this phase. Getting student workers to help run aspects of the event, as well as to encourage other students to participate in-person, can be a very effective means of delegating the workload. Recruiting participants can be time-consuming as it concerns marketing, dealing with no-shows, offering incentives, and providing physical and web space for testing (Mitchell & West, 2016); thus, it is important to reflect on a project timeline and include potential partners when available.

At this point, the formal and informal student surveys demonstrated that the user population wanted online access to resource knowledge and that they were more likely to use such a product if it were engaging. We discussed possible solutions including trivia games, tutorials, and virtual reality.

Ideate

The third phase, ideate, involves brainstorming solutions to the defined problem. We considered many solutions, including user engagement, grounded in an empathic approach to increasing library user interest and exploration of library resources. This approach sparked the initial concept of a university trivia game as a way to pilot the use of Twine, an open source software for non-linear stories. We selected Twine for its ease of access, affordability, and low learning curve; it uses HTML, CSS, and JavaScript. We used the default Twine theme despite it presenting color contrast challenges with text-based activities.

We designed a small pilot with five questions, each with multiple choice answers and related images for graphical appeal. Three colleagues played the game and gave feedback on the potential use of this software for a library-based engagement activity. The feedback came from people with varying academic backgrounds and roles. Although informal and anecdotal, all feedback demonstrated that the pilot showed promise but that several areas needed improvement: refinement of the storyline and greater visual appeal were necessary for a more enjoyable game experience. This quick and generalized feedback during the ideation phase led to further research and collaborative efforts that informed the game-development process.

Next, the focus transitioned from a trivia game to a more game-like experience, similar to a “choose-your-own-adventure” theme. We drafted a generic storyline and developed personas for the characters needed to create a personalized adventure. We, the game developers, are not programmers, but rather two librarians whose strengths lie in research design and digital libraries. Combining these skills with UX methodologies enabled us to design working prototypes based on our internal workflows (as detailed in appendix B).

We considered a variety of potential users for the game experience. Our institution is experiencing increasing numbers of transfer, graduate, and online students—groups who often have difficulty attending a traditional in-person library orientation tour. Similarly, faculty, staff, and community members are users of the library who do not get the opportunity to join classroom-based library instruction or new-student tours. Therefore, we expanded the intended audience for this game design to include individuals who are not part of the groups typically served in library orientation tours.

With the general storyline, target population of users, and goals of the game in place, the actual game design began by looking at university publications that gave a sense of the student, faculty, and community population. From the data in the Middle Tennessee State University Fact Book (2017), an annual publication available online, we created personas based on the four major groups identified as the intended audience: undergraduate students, graduate students, faculty, and community members (see fig. 2). We identified point-of-interest (POI) destinations, which were locations, services, or resources located throughout the library. We analyzed these POIs for the different information needs of the four personas, resulting in shared and specific POIs in response to the character path the game user selected.

Figure 2.

Library user personas and points of interest by persona type.

Persona development and LibGO points of interest locations based on data from the Middle Tennessee State University Fact Book, library surveys, and observational data from interviews and focus groups.

Prototype

The fourth phase involves creating prototypes of the possible solutions formed during the ideate phase. Rather than making one prototype, this phase included both low-fidelity and high-fidelity prototypes to facilitate the early evaluation of designs.

A prototype can have varying levels of fidelity. Low-fidelity prototypes are often quick sketches on paper, a low or no-cost hypothesis that can easily be tested and modified. High-fidelity prototypes often cost more to produce—both time to create and cost of materials—and reflect artifacts that more closely resemble the final look and/or function of the final prototype or product. Regardless of fidelity level, developers are meant to iteratively test and modify prototypes in order to weed out design issues and evaluate the effectiveness of the product.

By pushing together large tables, we created a surface large enough to hold twenty-four pieces of 8.5 x 11 inch paper in seven rows. As a low-fidelity prototype, each paper represented a screen of a POI in the LibGO journey, with rectangular paper strips cut into varying sizes to show directional pathway options for each prototype screen. The prototype also included print outs of image possibilities for each page, allowing for easy viewing and swapping out of images for testing the look and flow of the paper prototype. Our interactions with this level of prototyping in the early stages of game design led to discussions on the POIs common among the user groups and on how to simplify the design.

The resulting solution set the common POIs as fixed locations—regardless of character selected—and restricted the data flow to only four levels of character advancement opportunities; these decisions required fewer changes to the code. This evolved into a smaller representation of prototypes that were more manageable and are represented in the data flow diagram in Figure 3.

Figure 3.

Example of the data flow diagram for the Community character.

Game developers have many options for creating data flow diagrams:

• LibreOffice Draw (https://www.libreoffice.org), free downloadable presentation software;

• Microsoft PowerPoint, commercial presentation software;

• Lucidchart (https://www.lucidchart.com/pages/), free, in-browser draw software;

• proto.io (https://proto.io), free trial or subscription, in-browser draw software;

• diagrams.net (https://app.diagrams.net/), free, in-browser draw software.

Figure 4.

Sample single screen data.

Example of a single screen of the data flow diagram for the Graduate character. Image on the left is one screen of the diagram workflow with script and semi-code notes in low-fidelity form. Image in the upper right is a screenshot of the corresponding high-fidelity prototype.

We replicated the data flow diagram and low-fidelity prototyping for each character. The arrows in the data flow diagram show the possible directional pathways, color-coded to each page. The format and layout of the data flow was the same for each character; however, the content of the pages may have had differences. The semi-uniformity we describe here enabled simplification of coding, keeping the game development manageable for novice developers.

We applied the changes discussed during the exploration of the paper prototype to the high-fidelity prototype, which had its own data flow diagram within the Twine software (see fig. 5). Prototyping for each character continued at both low- and high-fidelity, as new iterations of construction and testing revealed areas of incremental improvement. To see an example of the high-fidelity prototype from the developer’s view and the user’s view (the user interface), see Figure 6.

Figure 5.

Data flow diagram for the high-fidelity prototype.

Figure 6:

High-fidelity prototype from developer (backend) and user interface (front-end) views.

In the left image of Figure 6, notice the <b>Digital Seed Grants</b>. This uses HTML to bold the words between the tags <b>and</b>. HTML has a lower learning curve than programming languages, and creators can use it to aid in piloting gamified experiences in Twine, as we did with LibGO.

Test

The fifth phase involves an iterative cycle of testing the prototype and making modifications. This testing happens before implementation of the final product, service, or space. Once we had a functional high-fidelity prototype, two rounds of playtesting commenced. Usability testing is ideally done in multiple rounds with a small number of participants, with three to five users as a target number, in order to have time and a budget to test more design iterations (Nielsen, 2012). Accordingly, four to five playtesters participated in each round of LibGO testing. We selected participants with varying backgrounds, skills, and roles from each department in the library—public services, collection development, technology, and administration—and we included student workers representing a range of academic disciplines. Both rounds of playtesting used a pre- and post-playtest survey that asked for participant consent. The playtest, the surveys, and a final post-launch survey received institutional review board approval.

The pre-playtest survey first asked playtesters to list the computer model, browser, and character they would be testing. The survey then asked the participant to identify the average number of hours per week they played games—of any kind—and the start time of their playtest so we could later determine a completion time range. The post-playtest survey asked the playtesters to record the time they finished playing LibGO and had a set of fifteen questions that gathered initial perceptions, detailed accounts, and final reactions, along with an optional open-response question. Given the feedback and observational reactions during round 1 of playtesting, we added one more question to the post-test survey used in round 2.

We adapted playtest questions from Patton (2017), and we wrote them to address the goals of our original research study on gamifying library orientation. A copy of the pre- and post-playtest survey is available in appendix C. To summarize, the sixteen post-playtest questions included:

• five open-response answers to questions about initial perceptions of LibGO;

• six questions on detailed accounts of play, five yes/no and one free-text to document the playtester’s final score;

• four questions on final reactions, two no/maybe/yes options and two Likert-scale;

• one open response field for the playtester to provide additional feedback.

Playtest Analysis

Both rounds of playtesting used surveys to understand the prototypical experiences of the intended LibGO users. Prior to starting the usability and playtesting for each round, we administered a pre-playtest survey which gathered demographic information including the device, browser, and character selected for playtesting by the user, as well as the average number of hours per week the user spent playing games of any kind—board, video, and computer. After completion of this survey, users began playing LibGO, during which we observed their reactions and behaviors. After completion of LibGO, the playtesters took the post-playtest survey. The post-playtest data collection included both quantitative and qualitative measurements, which we report on below.

We used a form of affinity diagramming for both rounds of playtesting in order to formulate common themes from the same prompt and analyze the playtesters’ responses to each qualitative question. See Figure 7 for a screenshot of a sample coding of themes. This type of coding, not to be confused with computer programming, is the process of generating a word or short phrases—the codes—and then assigning them to sections of transcripts derived from interviews or surveys; it is a method of interpreting the meaning of textual content. Codes can be pre-defined or interpretive. Examples of interpretive codes include codes for attitudes toward something or consequences that result from acting on something (Asher & Miller, 2011).

Figure 7:

Excerpt of playtesting round 1 theme-focused analysis.

Color-coded spreadsheet data highlights commonalities among playtesters and design resolutions (see shaded area) based on round 1 post-playtest survey data.

Iterative analysis and open coding of these themes from round 1 and round 2 informed the design changes to the data flow and final Twine coding of LibGO. Color-coding for similar themes is a useful strategy in affinity diagramming and is most typically accomplished with different colored sticky notes. We found that moving this analysis to spreadsheet form not only allowed for similar trends to emerge as they would on paper, but it also enabled open coding practices with helpful Find and Replace abilities to search for related words within the playtesters’ responses. The purpose of both methods was to discover themes and trends in the data (Asher & Miller, 2011).

Round 1

Round 1 playtesting included five playtesters selected from the library, all of whom tested LibGO on a Windows computer in the library. Pre-playtest survey results in Table 1 indicate the browser playtesters used to play LibGO, average number of playtesters’ weekly gaming hours, and their selected character. Note that the character selection does not necessarily represent the role of the individual playtester, as this group was comprised of faculty, staff, and student workers from the library.

Table 1:

Playtesting round 1: pre-playtest.

Round 1 Playtester	Player 1	Player 2	Player 3	Player 4	Player 5
Browser	Chrome	Mozilla	Chrome	Mozilla	Chrome
Weekly Game Hours	0	5–7	1–2	1–2	0
Character Selected	Graduate	Undergraduate	Faculty	Faculty	Graduate

Findings from Round 1 Playtesting Feedback

Below are responses to the first post-playtesting question, “What is the most frustrating aspect of what you just played?”

• Player 1: A lot of reading. I don’t mind reading, but I wouldn’t be able to use this for quick information.

• Player 2: I wish there was either more interaction or some animation to keep me interested. Most of it was reading.

• Player 3: Mostly just reading and what if I wanted to explore other options.

• Player 4: Weirdly enough, I think it was the points. After finishing the game I knew I had missed a few sections, so I wanted to retrace my steps. Leaving off options felt like there was a specific order I had to take-when I wanted to explore more freely.

• Player 5: The text to image ratio may be a little high-not really frustrating, per se.

Four out of five playtesters in round 1 mentioned that the game required too much reading. In the Implementation Challenges section below we discuss how developers must balance instructional content with entertainment in educational game design. We were not surprised to see players label this issue as a frustrating aspect, and we attempted to address this point in later iterations.

Below are responses to the second question of the post-playtest survey, “What was your favorite part/aspect?”

• Player 1: It followed the routes I would logically take and answered questions I might not have thought to ask (ie: ILL and group study rooms.)

• Player 2: The detailed description of where everything is. I also liked choosing what path I could take.

• Player 3: The pictures and there was an attempt to be an interactive game environment

• Player 4: wonderful information. I really liked the 1st floor walkthrough-it was welcoming and informative. I kinda wished the other floors had that same location-specific narrative feel. Even though its more service-based and space shouldn’t matter, I liked the whole “you’re on this neat journey, let’s explore” feel.

• Player 5: I like the “make you own adventure” aspect.

Answers to this question helped us identify what we were doing correctly, as well as what to continue doing in future iterations. Three of five playtesters specifically mentioned liking the ability to choose their own adventure, path, or journey.

The remaining three open-response questions on initial perceptions covered wishes, changes, and description. As a result of this user feedback, we:

• Updated character selection to advance via both text and image selection.

• Made the back and forward buttons more visible in the user interface.

• Created a webpage listing links to resources—POI locations—so users could revisit information and learn more about the library.

Although there were mixed answers to the question, “If you could change, add, or remove anything from the experience, what would it be?” the user feedback directed the us to make the following changes:

• Made background colors for the Undergraduate and Faculty characters lighter.

• Added a point tracker in the sidebar menu area, so users could see their points add up in a consistent location, even while scrolling.

• Confirmed the faculty services content was already a part of the POIs; the playtester must not have visited that part of the game. This issue reinforced the need to create a resources page, so users could visit all POIs and links in LibGO after game play.

The final question of the initial perceptions section of the post-playtest survey asked the playtesters how they would describe LibGO to someone else. Two of five playtesters used the term walkthrough of the library’s resources, while game, knowledge-base, and tour each had single uses.

The detailed accounts section asked five questions regarding specific details of the playtester’s experience. The results of this section indicated that all five playtesters finished the game, did not want to quit the game early, and received an award during gameplay. Asked if the playtester learned anything new about the library while playing LibGO, three out of five said yes. This is interesting because all five playtesters worked in the library at some level—faculty, staff, or student worker—yet three out of five learned something new.

Noteworthy findings of the final reactions section pertained to user enjoyment and engagement. Four of the five playtesters responsed to these questions. Asked to rate their enjoyment of the play experience on a scale of 1 (boring) to 5 (enjoyable), playtesters gave ratings yielding an average of 3.5 for enjoyment. We also asked the playtesters to rate how engaged they felt while playing, and they gave an average rating of 3.75 for engagement. These findings suggested that the playtesters were likely to play again and had moderate enjoyment and feelings of engagement.

The final question of the post-playtest survey was an open response area, and four out of five playtesters gave responses. Among the compliments and suggestions for additions or edits, common themes emerged: less text, more media, more options to explore, and links to resources. One playtester found a spelling error in the game’s text, which we corrected.

Round 1 Observational Notes

Although the playtesting was an individual activity, we conducted it in a group setting. We were in proximity to hear and observe playtesting behavior. At times playtesters made verbal comments while playing, such as, “These characters are so cute!” Either during or after the game, more than one playtester asked for clarification of the highest score as well as tips on how to achieve that score. This information regarding user questions, preferences, and expectations was exactly the type of feedback we wanted. As a result, the idea of an About page—later renamed Rules—emerged. This section provided users an overview on how to play the game, its scoring system, and some basic information regarding the development of LibGO. See Figure 8.

Figure 8:

Rules page explaining how to play LibGO and scoring levels.

Round 2

Round 2 of playtesting included four playtesters selected from the library, all of whom tested LibGO on a Windows computer in the library. The pre-playtest survey results in Table 4 indicate the browser playtesters used to play LibGO, their average number of weekly gaming hours, and their selected character. As was the case with round 1, the character selection does not necessarily represent the role of the individual playtester, as this group was comprised of faculty, staff, and student workers from the library.

Table 4:

Playtesting round 2: pre-playtest.

Round 2 Playtester	Player 1	Player 2	Player 3	Player 4
Browser	Chrome	Mozilla	Chrome	Chrome
Weekly Game Hours	1–2	1–2	0	1–2
Character Selected	Faculty	Community	Faculty	Undergraduate

After taking the pre-playtest survey and playing LibGO, the round 2 playtesters took a post-playtest survey. This survey was identical to the post-playtest survey administered to the round 1 playtesters, with the exception of one added question as we mentioned in appendix A. This yes/no question asked whether the participant visited the About page in the sidebar.

Findings from Round 2 Playtesting Feedback

Initial perceptions in round 2 revealed different experiences for the new round of playtesters. Three out of four playtesters in round 2 mentioned a concern about scoring. They stated a desire to attain a high score, for example asking if choosing “silly answers” would cost them points, and stating that they “really wanted that gold star.” Relatedly, two out of four playtesters’ favorite aspect of the game was the “tricky” or “gotcha” choices—for example, taking a nap instead of pursuing research. The other two listed the freedom to explore or learn on their own as their favorite aspect. One playtester found the possibility of losing points for funny decisions as both their most frustrating and favorite aspect of LibGO.

The remaining three open response questions regarding initial perceptions covered wishes, changes, and description. In terms of something a user wished their character could do but could not, one user wished for the ability to go backwards in gameplay. Although this ability existed, either the playtester did not attempt to go back or did not recognize how to do so. As a result of this feedback, we reaffirmed the decision to keep the functionality giving users the ability to go-back or undo previous choices in the game.

There were mixed answers to the question, “If you could change, add, or remove anything from the experience, what would it be?” But because of user feedback, we:

• Created and provided a list of all POIs at the end of the game.

• Added the list of POIs to the About page—later renamed Rules—so the resources were accessible at any point of gameplay.

• Enabled a trial and error method of choices in the game by giving users the option to start over or try again for a new score; this included adding Undo/Back arrows.

• Updated instructions and rules, so users knew there was only one route to hit all POIs in game, which is required to obtain a perfect score.

The final question of the initial perceptions section of the post-playtest survey asked the playtesters how they would describe LibGO to someone else. Three out of four playtesters responded with overview, tour, and educational game.

The detailed accounts section of the round 2 post-playtest survey asked six questions regarding the playtester’s experience. The results indicated that all four playtesters finished the game, did not want to quit the game early, and received an award during gameplay. Half of participants said they learned something new about the library while playing LibGO, and half said they visited the About page in the sidebar menu. Based on user feedback, we:

• Renamed About to Rules, in order to help users identify where to go if they needed help before or during play.

• Added details on scoring to Rules.

Like round 1, the round 2 playtesting attempted to discern user enjoyment and engagement. Asked to rate their enjoyment of the play experience on a scale of 1 (boring) to 5 (enjoyable), playtesters gave an average rating of 4 for enjoyment. This was a higher average than round 1, perhaps indicating that the changes made from round 1 results increased the enjoyment of LibGO. The second Likert-scale question asked the playtesters to rate how engaged they felt while playing. The playtesters gave an average rating of 4.25 for engagement; again, this was a higher average compared to round 1.

The final question of the post-play survey was an open response area asking for any feedback, and the responses provided helpful information. One playtester admitted to playing again, under a different character, in an attempt to maximize their score. We have referenced this desire to try for the maximum score, and it is important to note. A second playtester found LibGO to be “an excellent way for users to fill in the knowledge gaps privately and on their own time.” Some playtesters gave specific suggestions to add content that was pertinent to specific users of the library, for example, how to purchase library materials. One playtester admitted being confused at first, “but as I went on, I picked it up and [understood].” This type of feedback was important for understanding the UX and the on-boarding process of a new website experience. Playtesters provided suggestions on improving the perceived boxy-storyline as well as affirmation of LibGO’s purpose, with one playtester stating, “Wish I had this as freshman, would have been extremely useful.”

Round 2 Observational Notes

As with round 1, the playtesting was an individual activity done in a group setting. We were in proximity to hear and observe playtesting behavior. Two of the four playtesters in round 2 made verbal comments while playing, including, “I wanted a high score, how do I get a high score?!” and, “I can’t get it, I want to keep trying. Ugh, how do you get a high score?” Although these comments were not directed at anyone, we were interested to see how this gamified aspect of orientation motivated the users to keep exploring.

Discussion of Iterative Testing Results

To begin analyzing the playtests, we decided to put the collected data into a form that would allow patterns to visually emerge. We used spreadsheets for this project, but use whatever works best for the person analyzing the data, including outlining programs or mind-mapping software (Rubin & Chisnell, 2008). Next, we tried to identify the tasks with which users had the most difficulty. Tools that can help determine this factor include making lists, tallies, matrices, structure models, and flow diagrams, among others (Rubin & Chisnell, 2008). In the case of LibGO, we kept a running list of errors, issues, and suggestions based on playtest results as well as our own usability tests, which included debugging (see fig. 9). From this list, we tried to identify if user error stemming from either a player misunderstanding or a game attribute caused the misunderstanding. Finally, this list prioritized the problems by criticality. This allowed us to “structure and prioritize the work that is required to improve the product” (Rubin & Chisnell, 2008, p. 262).

Figure 9:

Debugging the HTML and Twee code in Twine.

Descriptive statistics gleaned from the data collection also helped the playtest analysis. For example, play times ranged from 4 to 15 minutes, with an average of 7 minutes and 45 seconds. It was interesting that the average time played in round 1 was 8 minutes and 28 seconds whereas the average time played in round 2 decreased to 7 minutes and 15 seconds (see Table 5). Possible factors for this change, such as LibGO improvements from the iterative design, could be grounds for further analysis in future versions of LibGO.

Initial perceptions data demonstrated several differences between round 1 and round 2 playtesters. Round 1 playtesters were most frustrated by the use of a text-heavy game format. By contrast, round 2 playtesters were most frustrated by not knowing how to achieve a high score. It was very interesting that no one in round 2 mentioned LibGO’s heavy reading load. There are many possible explanations for this difference: for example, the iterative design process took round 1’s usability feedback forward into improvements made before round 2’s playtesting. It is also possible that participants in round 2 were game enthusiasts and did not mind the text load. In future prototype testing, follow-up interviews with the playtesters may shed light on this difference.

When asked how to describe LibGO, there were a variety of answers in both rounds of playtesting. The terms game, walkthrough, and tour each had two uses while knowledge-base and overview each had one use. Despite the differences in their description of LibGO, all four participants in round 2 said they would play again, while only one of five participants in round 1 said they would play again and three said they might play again. Just over half of the total participants—five out of nine—said they learned something new about the library while playing LibGO, which is interesting because all nine playtesters work at the library. Therefore, these results indicated that this was a learning tool even for those who work in the library, giving us a sense of being on the right track for using gamification as an alternative form of learning.

The iterative design process was critical to improving many structural issues of the game discovered during playtesting; specifically, several factors within LibGO discouraged replayability. If a player chose to replay LibGO, the score continued from the previous game—the points did not reset. Similarly, if players used the in-game Undo/Back arrows, the score did not recalculate. Understanding the critical value of replayability, we chose to fix this scoring aspect in the later iteration of the game.

Round 2 playtesters mentioned wanting links to the About screen during game play. We had purposely avoided in-game links in order to keep users playing the game—thereby not potentially losing them before the game’s completion. This design strategy, however, was not helpful to users who wanted more information about a resource they discovered during the game. Therefore, we decided to provide a list of linkable resources in the About page, so users could connect to content post-game.

The playtesting, meant to be a silent activity, also allowed for an impromptu think-aloud test as playtesters vocalized concerns while interacting with or navigating through LibGO. Whether the remarks were said unknowingly or not, they ranged from processing or questioning oneself, congratulatory remarks when advancing through play, or frustration when the user did not get a high score. The latter left some shouting, “How do I get a high score?” or, “I want a high score, can I play again?” This think-aloud process was not planned, but it ultimately spurred improvements to iterations of the prototype.

Another finding from the first round of playtesting was that the audio content was not ideal for an event in a computer classroom with restricted sound capabilities on the computers. Therefore, although the playtesters suggested additional multimedia content, we decided to limit content to one video as one intended audience would be playing the game in an environment without sound. A second intended audience, online players, would be able to control the audio of their personal computing device.

Although gameplay involved text-heavy content at times and required players to answer questions, the playtesting proved that the pedagogy surrounding game mechanics—such as achieving a top score—was more complicated than a trivia game or walking tour. This issue of engagement was an early concern for us. We wondered whether the gamified library experience would have enough game mechanics to leave players feeling the experience was interactive, engaging, and immersive.

The playtesting also showed that coding these themes and trends iteratively from round 1 to round 2, and also collectively for a larger sample, informed the design changes to the data flow and final game coding of LibGO. One example of this is the changes made to the start screen as seen in Figure 10, which shows how LibGO’s start screen looked at the beginning of round 1 playtesting (left) and after round 2 playtesting (right). It is important to note that triangulating think-aloud data and survey data from the playtests was highly effective in resolving user errors and design issues. The use of both qualitative and quantitative data can help evaluate the efficiency or usability of a user interface and learning objectives for the content illustrated in the user interface.

Figure 10:

LibGO start screen before (left) and after (right) iterative rounds of playtesting.