Metadata for Everyone: Identifying Metadata Quality Issues Across Cultures

Introduction

Treating metadata records as informational objects in their own right, the Metadata for Everyone project examined metadata for scholarly publications to identify issues related to cultural identities and meanings and their associated implications. This paper begins by introducing the project goals and summarizing our scope and methods. We then discuss the issues and categories identified. We conclude with key takeaways, limitations, and next steps for our project.

Context

Metadata is generally understood to provide key bibliographic and domain information and to connect that resource to the larger knowledge ecosystem. At the same time, we may think about metadata as “contributing to a story we are telling about ourselves as individuals, as organizations, and as a community.”¹ In this framing, metadata is inextricably tied to the people who create it and the people, places, and things that it describes. Through description and context, metadata functions as a narrative device to assert the social contexts and genealogies of our knowledge. In this sense, metadata is storytelling and identity building.

In both respects, metadata has the potential to make local research, communities, and stories visible and accessible for wider audiences in our global scholarly ecosystem. The key phrase here is “has the potential,” because creating metadata of sufficient quality to enhance discoverability and access and to represent all our stories as we want them to be represented takes time and labor. For some metadata creators, this investment is much larger than for others.

This challenge is, in part, because many of our technical systems and standards are built around the English language and Western ideas of knowledge and scholarly practice.^2,3,4,5,6,7 This can look like English-language metadata requirements for indexing, geographic lists based on standardized lists that do not include Indigenous nations and contested regions, or user interfaces that struggle with non-Latin scripts and diacritics.^8,9,10,11

Whose stories do our current metadata systems and standards permit? Who is allowed to contribute to and shape the stories that are told about themselves and their knowledge?

Metadata for Everyone

As organizations that provide systems for managing scholarly metadata, the Public Knowledge Project (PKP) and Crossref sought to understand the ways in which individuals and communities actively seek to convey meaning and express identity through metadata. We were interested in identifying who is left out of metadata, even when standards are perfectly applied; where metadata systems and standards are falling short; and the unique ways that individuals have used metadata to assert or retain their identity in response to these shortcomings. We also wanted to determine how well our current metadata systems reflected our global communities of users.

The project involved three stages, beginning with an initial discovery phase between PKP and Crossref to establish scope and priorities and identify data sources. The Crossref application programming interface (API) was queried to retrieve a sample of records from the identified sources, and the 427 records pulled were analyzed to surface and categorize culturally-related issues in the first phase. The categories derived in this phase continued to inform work in the second phase to programmatically measure metadata quality based on the presence of these issues on a much larger scale.

Identifying Cultural Quality Issues

Common definitions of metadata quality in the literature attend to dimensions like the informational completeness and accuracy of the record and its conformance to standards and controlled vocabularies for use by humans and machines alike.^12,13,14 Such issues are referred to from here on as those of “general quality.” By contrast, the cultural quality of metadata is measured against the possibility that an aspect of the metadata could cause harm or disservice to a person or group and their work.¹⁵

Specifically, cultural issues are defined as issues that impact or have the potential to impact the representation of identities, roles, intentions, and other factors specific to social, regional, or research cultures. This definition is intentionally broad because “culture” can be interpreted in numerous ways and, with each interpretation, it inflects and impacts how knowledge is understood and research is conducted differently.

To identify issues of cultural quality, we embarked on a qualitative study of the sampled records. Where “a comparison between the surrogate and the original item is absolutely necessary” when evaluating metadata, each record was close read alongside the published item it described—usually a PDF for a journal article, the webpage for accessing that item, and contextual information about the larger work.¹⁶ Close reading involved noting which fields or values were present or absent, as well as examining the content and form of values provided. Differences in content and form between the record, item, webpage, and work were also recorded.

We initially reviewed sixty-one records to identify fields more likely to relate to culture and identity. We found nine relevant fields: 1) abstract, 2) item title, 3) given name, 4) family name, 5) institutional affiliation, 6) publisher name, 7) title of work, 8) language/s accepted in the larger work, and 9) the subject headings applied to the larger work. Although we did not actively review for issues of general quality, these were also noted when found.

Findings

Across the 427 records and nine fields of focus, 4,859 issues of both general and cultural quality were found. Of these, 90 percent impact or have the potential to impact cultural meanings and identity. Among the cultural issues, thirty-two unique issues were identified and were organized into five main forms (see Table 1).

Issues were not equally common, with eight unique types comprising 75 percent of all general and cultural issues found; among the cultural issues specifically, they amounted to 83 percent. Of these eight, seven were of the “value absent” form, indicating a glaring absence in this sample that has cultural implications. Due to the non-random nature of the sample, the frequency of each unique issue is less significant than the categories found. Counts are noted for transparency.

To better understand what is at stake along cultural lines, we classified the thirty-two issues to derive five key categories related to cultural meanings and identities:

• Language: Issues relate to the languages and scripts of values and/or the way in which they are identified using language and style attributes.

• Contribution: Issues relate to the acknowledgment of contributors to the creation and publication of the item and its contents.

• Naming: Issues relate to the recording of individual and organizational names in accordance with linguistic and cultural conventions.

• Status: Issues relate to stylistic and content-based interventions to capture the status, seniority, or prestige of individuals or institutions.

• Geography: Issues are caused by the absence or partial representation of physical location and its social and cultural associations.

These categories are conceptual and issues often fit into one or more categories depending on their context. Table 2 provides annotated examples of issues for each category.

Discussion

With issues across five categories of language, naming, status, contribution, and geography, and cutting across questions of consistency, completeness, and quality, it is apparent that metadata is not for everyone. The five key types, thirty-two unique issues, and five main forms represent just some of the ways that our metadata systems and standards are failing users.

Although many of the identified issues may in fact result from poor metadata practice, we should not pre-emptively assume that all issues are ones of general quality. It is just as possible that they result from deliberate acts to assert or retain cultural meanings and aspects of identity because the current systems and standards are failing to reflect user realities and require significant resources to generate rich and inclusive metadata.

As well, deviations from metadata standards and best practices can affect how cultural meanings and identities are represented. Certain issues may have more substantive impacts than others. In all cases, though, there is potential for confusion and, taken together, increased wariness of how well metadata can convey meanings and identities.

At local and regional levels, more conversations are needed with scholarly, publishing, technical, and metadata communities to understand and address the ways that metadata renders identities and meanings invisible in relation to the five categories identified in this review. If metadata contributes to stories that individuals and communities seek to tell about themselves, we must ask what stories our current systems and standards are neglecting and how this neglect can be addressed.

This review was not intended to surface all issues in the sample. Those found are specific to a single interpretation of these 427 records. The project also did not reveal every possible issue of cultural quality that could exist in metadata. The ability to recognize issues often depends on one’s familiarity with particular social, regional, and research cultures.

Future Developments

The next phase of our project focuses on detecting the identified cases in a sample of 100,000 records to provide a better sense of their prevalence in the scholarly record. With over 5,600 publishers and thirty-seven languages in this sample, this work also looks at intersections between metadata quality, language use, and publisher size to explore who is most affected by issues related to language.

Preliminary analyses suggest that the issues discussed above appear more frequently in multilingual and non-English monolingual records. While instances are found in records created by publishers of all sizes, these records are most often created by the smallest publishers. The smallest publishers working with multilingual and non-English monolingual publications thus bear the highest burden for issues related to language, which often relate to the four other types.

Conclusion

The work of libraries, publishing houses, and technical organizations revolves heavily around the management of information. We nurture information resources in their creation, development, distribution, acquisition, discovery, access, use, and preservation in the scholarly record for the long term. With so many touch points in the resource life cycle, it is important to ask how our metadata systems and standards contribute to definitions of who or what can be in the scholarly record, and who or what can subsequently be discovered and how.

In the larger scholarly ecosystem, we also hope these findings can support and contribute to broader conversations and struggles around equitable participation and homogenizing standards. By critically reflecting on our current systems and standards in this way, we can tease out such biases and start to unpack the tensions that they can create. From there, we can weave principles of equity into our best practices for metadata creation, journal publishing, and indexing, as well as our development and evaluation processes for technical systems and standards.