Skip to main content
Concurrent Session

Values-Driven Collection Assessment for Collaborative and Ethical Decision-Making

Authors
  • Anna Rubinowski orcid logo (Deakin University)
  • Wendy Campbell (Deakin University)

Abstract

Deakin Library redeveloped our approach to collection evaluation to ensure that our resources meet the evolving needs of our communities while aligning with and advancing the University’s strategic direction and vision of creating a vibrant, rich, and inclusive ideas ecosystem. The new Collection Governance Framework uses a values-based model which was developed in close collaboration with all library teams, ensuring inclusivity, transparency, and giving teams ownership of the process. Supported by practical principles, a holistic and comprehensive collection evaluation rubric, and reinforced with extensive documentation and work instructions, the collection evaluation process is facilitated through Jira as an agile project management tool. The information gathered through the evaluation process on Jira integrates both quantitative and qualitative insights and is analyzed and visualized through a Tableau dashboard, allowing for benchmarking against various measures and identifying collection strengths and gaps.

Keywords: collection assessment, collection analysis, collection evaluation, collection lifecycle and management

How to Cite:

Rubinowski, A. & Campbell, W., (2026) “Values-Driven Collection Assessment for Collaborative and Ethical Decision-Making”, NASIG Proceedings 40: 3. doi: https://doi.org/10.3998/nasig.11070

Published on
2026-10-08

License

CC-BY-NC 4.0

Development of the Collection Governance Framework

Deakin Library’s collections contain over 650,000 physical items across four Australian campuses and a small collection at our offshore campus in India. Our users have access to over 1,000,000 e-books, over 220,000 e-journals, almost 600 databases, over 360,000 streaming media sources, as well as 16 Read and Publish agreements which are negotiated through our national consortium, the Council of Australasian University Librarians (CAUL).

After major organizational change in late 2022, Deakin Library’s established collection governance structures and decision-making processes based on hierarchical committee structures were discontinued. Additionally, no comprehensive collection governance review or collection assessment and maintenance program had been undertaken in several years despite significant changes in the higher education and academic library sectors, making a comprehensive collection review a high priority.

The strategic Deakin Library Plan 2022–2025 provided guidance throughout the process and acted as major catalyst for the development of Deakin Library’s new Collection Governance Framework starting in early 2023.1 The main aims for the development and implementation of the framework and collection strategy were:

  • Values-driven: Align with Deakin Library’s human-centered strategy through a values-driven approach. Our goal was to focus on ethical and proactive decision-making, not only to ensure the best fit for purpose collections for our Deakin communities but also to challenge the status quo of established systems and business models.

  • Efficient: Replace the old collection governance structure, which was heavily dependent on meetings with rigid and exhaustive documentation requirements that often required significant workloads and resulted in delayed decision-making. Develop an innovative, agile, and efficient approach and replace manual workflows with automation as much as feasible.

  • Transparent and collaborative: Build a comprehensive collection evaluation process that incorporates transparency of responsibilities and information throughout, and provides proactive, robust, and efficient collection development and management practices while empowering staff across the library to contribute to collection governance and maintenance.

  • Fit for purpose: Ensure that our resources are fit for purpose to support Deakin communities’ learning, research, innovation, and a sense of community and belonging. Ensure we avoid resource acquisition and retention for “just in case” or preservation and aim for our collection to be hardworking, demonstrating a strong and consistent use of resources.

  • Open: Prioritize open research and open infrastructure where feasible.

  • Diverse and inclusive: Embed diversity and inclusivity not only through the subjects and content acquired, but also in our metadata and discovery strategies, improved accessibility through a wide range of resource formats and support services, and Indigenous Knowledges programs.

To achieve this goal, we collaborated closely not only with our library colleagues at Deakin, but also more broadly across the university and the broader higher education and scholarly communication sectors on both a national and international level. In the early stages of the project, we conducted an extensive literature review and environmental scan. The key takeaway from this research was that there was no perfect or ready-made evaluation model that would work in the Deakin context. The majority of institutions conducting collection evaluation and assessment relied heavily on cost per use (CPU) analysis, often in reaction to sudden budget constraints that necessitated cancellations. Most cases with a more proactive approach utilized a scorecard model, which was also our initial model to measure collection evaluation criteria in the framework.

Deakin Library’s Collection Governance Framework is based on three distinct layers:

  • Collection Values form the foundation for the Framework.

  • Collection Principles outline in practical terms how the collection values are applied practically to both manage our collections and provide collection services more broadly.

  • Collection Evaluation Criteria measure resources against the values and principles.

Figure 1 provides a summary overview of the framework.

At the top level, spanning the width of the whole figure, is “Library Policy & Library Plan”; underneath this, also spanning the whole figure, is “Collection Governance Framework.” The three layers of the framework are then represented, each taking up an equal third of the image, with specifics listed underneath each. The following are Collection Values: Open based on FAIR and CARE principles; Accessible; Inclusive and Diverse; Equitable; and Engaging and Collaborative. The following are Collection Principles: Providing relevant and current resources; Enabling user-friendly access and availability; Protecting user privacy; Conducting evidence-based and informed collection review; Advancing Deakin Strategy; Ensuring fit for purpose resources; Enabling discovery and accessibility; ensuring financial sustainability; Upholding intellectual freedom and limiting censorship; and Staying accountable and compliant. The following are Collection Evaluation Criteria: Alignment with Deakin Strategy, Vision, and Values; Licensing and Compliance; Cost and Usage; Usability and Accessibility; Disciplinary Field; Administration; Metadata and Discovery; and Publishing.
Figure 1. Deakin Library Collection Governance Framework Showing the Collection Values, Collection Principles, and Collection Evaluation Criteria

Collection Evaluation Rubric

Our original intention in building a scorecard for collection evaluation based on our collection evaluation criteria was quickly superseded by a rubric model. As a well-established assessment and evaluation tool in higher education, a rubric would be familiar to our key stakeholders and offer the advantage of being more flexible and nuanced than a scorecard and allow us to more easily combine qualitative with quantitative assessment.

The narrative-based structure of the rubric facilitated input from subject matter experts using their expertise and own language to define the criteria in their relevant sections and foster strong collaboration between teams and ownership of the process. This also enabled an element of subjectivity in the evaluation while still ensuring consistency in the assessment process, allowing for what was previously considered a gut feeling to be expressed in a more formalized and evidence-based manner. Through this process we were able to introduce discipline-specific nuance within the broader evaluation criteria, balancing quantitative criteria like CPU against qualitative criteria like relevance and quality of the resource for the relevant disciplines. This balance is deeply embedded and maintained in the rubric through scoring and weighing each of the criteria against each other with different importance assigned to different criteria depending on the type of resource.

To ensure the rubric reflects the various perspectives and needs of various stakeholders, the team ran a program of workshops with all relevant library teams to capture the expertise and opinions of all teams who would be contributing to resource evaluation.

It took over six months of workshops, drafting, revising, reviewing, and testing to complete the rubric in its current form. While the framework is very comprehensive, resources are not assessed against every single criterion in the rubric. As Table 1 below shows, resources are only assessed against criteria considered relevant, and depending on the cost of a resource, the criteria are more or less strictly scrutinized. There are also a number of criteria designated as critical for decision-making; if any of these criteria rates below the acceptable level, an in-depth review is initiated to consider the impact of the low rating for the overall purpose of the resource for the university.

Table 1. Deakin Library Collection Evaluation Rubric Overview of Evaluation Criteria

Rubric

Collection Performance Criteria

Initiate in-depth review if rating is low

Physical Collections

Electronic Resources

Research Tools & Resources

Publishing Agreements

Library Working Tools (internal use)

High-value Donations

Digitization Projects

Sector Memberships and Support

Alignment with Deakin Strategy, Vision and Values

Strategic Alignment

✸

✔

✔

✔

✔

✔

✔

✔

✔

Vendor Conduct and Commercial Integrity

✸

✕

✔

✔

✔

✔

✔

✔

✔

Membership Benefits

✕

✕

✕

✕

✕

✕

✕

✔

Disciplinary Field

Supports Research & Teaching

✸

✔

✔

✔

✔

✕

✔

✔

✕

Reputation

✔

✔

✔

✔

✕

✔

✔

✕

Collection Strength and Relevance to Deakin University History

✔

✔

✔

✔

✕

✔

✔

✕

Licensing and Compliance

Compliance

✸

✕

✔

✔

✔

✔

✔

✔

✔

Authorized Users

✸

✕

✔

✔

✔

✔

✕

✕

✕

Terms and Conditions of Use

✸

✕

✔

✔

✔

✔

✔

✕

✕

Privacy

✸

✕

✔

✔

✔

✔

✔

✕

✔

Liability

✸

✕

✔

✔

✔

✕

✔

✕

✕

Administration

Resourcing

✔

✔

✔

✔

✔

✔

✔

✔

Customer and Technical Support

✕

✔

✔

✔

✔

✕

✔

✔

Agreement Administration

✕

✔

✔

✔

✔

✔

✔

✕

Cost and Usage

Cost Benefit

✸

✔

✔

✔

✔

✔

✔

✔

✔

Reporting

✕

✔

✔

✔

✔

✕

✕

✕

Metadata and Discovery

Authentication

✸

✕

✔

✔

✔

✔

✕

✕

✕

Access

✕

✔

✔

✔

✕

✔

✕

✕

Discoverability

✔

✔

✔

✔

✕

✔

✔

✕

Usability and Accessibility

Usability

✔

✔

✔

✔

✔

✔

✔

✕

Accessibility

✕

✔

✔

✔

✕

✕

✕

✕

Publishing

Impact on Deakin’s Open Access Agenda

✸

✕

✕

✕

✔

✕

✕

✕

✕

Publishing Journey

✕

✕

✕

✔

✕

✕

✕

✕

Rights Retention

✕

✕

✕

✔

✕

✕

✕

✕

Implementing and Structuring the Evaluation Framework in Jira

In operationalizing the rubric, it was essential to align with the values underpinning the framework and find a software solution that would complement existing workflows and improve efficiency where possible. The chosen platform was Jira, a project management tool from Atlassian. Jira is widely used in the software industry to support various agile methodologies, including Scrum and Kanban, and is highly customizable to accommodate diverse project workflows. Key features such as agile boards, backlogs, roadmaps, and reporting enable efficient and collaborative project management.

For this implementation, a shared digital workspace was required that allowed for collaboration across multiple teams and individuals with the flexibility to involve contributors on an as-needed basis. Transparency was also a critical consideration; it was important that evaluations be visible across teams to ensure that decision-making moved beyond closed committee processes and became more accessible to all library staff. Finally, a single source of truth was required: a stable, central location in which all discussions, evaluations, and decisions could be recorded and accessed year after year.

To implement the rubric in Jira, a Parent-Child issue hierarchy was implemented. The Parent issue provides a central overview of the resource, while each Child issue corresponds to a specific rubric section where criteria are assessed. This structure enables distributed evaluation, allowing teams to focus on areas aligned with their expertise.

To support transparency and usability, each Parent issue includes several key components: general instructions, a resource description, and essential metadata such as publisher, vendor, purchase history, discipline, and other administrative details. Custom fields and controlled vocabulary are used where possible to ensure consistency in data entry. A prominent Approval Progress section provides a concise summary of the review status, including approvals, decision-makers, associated costs, and planned review dates. All associated Child issues are listed along with hyperlinks for ease of navigation.

Child issues inherit resource metadata through Jira automation and updates made to the Parent issue cascade down to the Child issues to ensure consistency. The primary evaluation activity occurs within the rubric criteria section, where each item is assessed using a drop-down list aligned with a six-point scale from unacceptable to ideal. A comments section allows evaluators to provide contextual notes or justifications for their ratings. Jira also supports ongoing contributions beyond formal review cycles where users can add comments or revise ratings in response to new information, such as user feedback, vendor announcements, or curriculum changes.

Jira Workflows and Automations

Customization of the Jira project and the integration of automation have enabled streamlined and manageable workflows to support the evaluation process. The workflows align with three main streams of activity: work completed by the Collections Team, automation-driven processes, and contributions from other library teams involved in resource evaluation.

The full workflow begins with the creation of a Parent issue, followed by the automated generation of relevant Child issues based on resource type. Metadata is inherited, and non-applicable criteria are automatically flagged. The Collections Team then assigns Child issues to specialist teams, initiating the evaluation phase.

Automation supports key transitions throughout the process including triggering evaluation notifications, monitoring completion, reallocating issues for analysis, and alerting stakeholders at critical points such as final approval. Manual quality checks, final decisions, and actions such as renewal or cancellation are completed by the Collections Team. A final automation resets the Jira issues in preparation for the next review cycle. Figure 2 shows the workflows between the Collections Team, the Jira automations, and the contributions by other library teams.

The workflow is represented as a unidirectional flow chart between the three areas of Collections, Automations, and Library Teams. The workflow begins in Collections with “Create Parent issue.” Next, in Automations, are the steps “Create Child issues,” “Inherit fields from Parent,” and “Pre-populate criteria.” This then returns to Collections for “Assign Child issues to Library teams,” followed by an Automations step, “Email to Library teams.” Library Teams then performs “Resource evaluation” which triggers Automations to “Email to Parent assignee when all Children are complete.” Collections follows with “Transition Parent to analysis stage” which returns to Automations for “Re-assign issues,” “Status update,” and “Email to Analyst.” Next, Collections “Visualize recommendation” and “Recommend and approve,” which returns to Automations for “Email Parent assignee as approval fields are updated.” Collections then performs “Action decision” and “Close Parent,” which is concluded by Automations responding with “Un-assign all issues,” “Remove due dates,” and “Remove teams responsible.”
Figure 2. Jira Workflows: Automations and Human Touchpoints

Integration with Confluence and Visualizing Evaluation Outcomes

Progress and status are monitored using Jira’s visualization tools, including timelines and dashboards, which can be filtered by various controlled vocabulary fields. An integration with Atlassian’s Confluence product provides each library team with a dedicated dashboard displaying their outstanding Child issues. These dashboards are generated using a simple saved search embedded in Confluence, allowing teams to track and manage their tasks, focusing on their assigned issues in isolation, improving usability and reducing cognitive load.

Tableau is used to analyze and visualize evaluation outcomes at the individual issue level. Figure 3 shows a dashboard example with an overview of evaluation progress, approval status, and outcomes.

The dashboard includes three sections: Overview, Section ratings, and Resource details. Example data is represented in each section in a generalized way. In Overview, Approval status is listed as “Watchlist,” and Evaluation type is listed as “Renewal.” The Preliminary outcome is “proposed recommendation based on criteria ratings and subsequent weightings according to resource type.” Under this, Watchlist is defined as “proceed for single year and monitor for changes in ratings.” Low rating alert is flagged, which is defined as “criteria below were rated unacceptable, review comments in Jira.” The Section ratings ranks criteria on a scale from unacceptable to ideal. The criteria Strategic Alignment, Administration, and Publishing all rank low in the acceptable range. Licensing & Compliance and Usability & Accessibility both rank on the higher end of the acceptable range. Disciplinary Field, Cost & Usage, and Metadata & Discovery all rank about halfway between acceptable and ideal. The Resource details section has a dropdown menu to select a resource; an anonymized read and publish agreement is selected. Under this are details for Resource, which includes the name, publisher, vendor/agent, Caul offer (yes/no), resource type, faculty (by discipline), and evaluation type. Following this are details for Sierra, which includes the resource record and order record numbers. The next subsection lists Access information by population type, which includes alumni, Deakin College, GIFT City, and TNE programs. The last section that is visible is History, which lists acquisition year and original requestor.
Figure 3. Visualization of Resource Evaluation

Resource details are displayed in a format consistent with Jira to support decision-making. A preliminary outcome is generated based on the rubric six-point scale and weighting by resource type, providing an initial recommendation. A low rating alert highlights any unacceptable criteria, prompting review of supporting comments.

Section-level results are visualized using a red-to-green gradient aligned with the rating scale. This traffic light system offers a quick, intuitive view of evaluation outcomes, with the same color logic applied to the recommendation, and text is provided to ensure accessibility for all users.

Impact, Learnings, and Insights

The impact of the implementation of the new Collection Governance Framework is a more structured and consistent approach to collection assessment with a single source of information in Jira. This has resulted in more immediate communication and consultation between teams, leading to a more efficient and timely decision- making process.

Efforts are ongoing to further streamline the framework and simplify the associated workflows. Following the completion of a full evaluation cycle, there will be an opportunity to undertake benchmarking such as comparing resources by discipline, format, or cost bracket. We are also actively investigating opportunities for automation through the use of APIs to facilitate data integration.

While the initial setup has been quite time-consuming, we have already seen significant time-savings as staff can add comments and contribute to the evaluation process at any time, cutting down on e-mails and meetings. As this process matures even further, evaluations will simply need to be reviewed during the normal review cycles and only updated if significant changes have occurred, saving even more time.

Setting up Jira required substantial self-directed learning, particularly in building a new project. In hindsight, mapping out fields, workflows, and automation logic in advance would have reduced time spent on revisions during configuration.

Despite early consultation with teams, a delay between planning and implementation led to knowledge gaps. This underscored the need for ongoing support and training, especially for staff not involved in the initial development phase.

The framework and the completed evaluations have also enabled us to form much more robust and informed negotiation strategies, leading to the cancellation and rejection of purchases not only due to unsustainable and unjustified price increases, but also due to a misalignment with our values and principles and ethical concerns.

Acknowledgments

We would like to acknowledge the contributions of our Deakin University colleague:

  • Lauren Halcomb-Smith, Lecturer, Open Education, at Deakin University for the collaborative development of the Deakin Library Collection Governance Framework.

  • Danny Kingsley (Director, Library Services (Information)), Catherine King, (Manager, Licensing and Subscriptions), and Skye Ryan (Manager, Monographs and Course Resources) for their support in preparing this session.

Contributor Notes

Anna Rubinowski is Senior Manager, Collections, Deakin University, Geelong, Victoria, Australia.

Wendy Campbell is Collection Performance Librarian, Deakin University, Geelong, Victoria, Australia.

Note

  1. Deakin Library, Deakin Library Plan 2022–2025 (Geelong: Deakin University, 2024), https://www.deakin.edu.au/library/about/library-plan. ⮭