Interwoven is a digital humanities (DH) platform from the Museum of Art and Photography (MAP) in Bengaluru, India, for exploring textiles from fourteen museums through an advanced linked open database (LOD). This LOD is powered by Microsoft's computer vision (CV) and AI text analytics. The undertaking is an ambitious one that seeks to bring together collections from across the world into one digital visual space, offering opportunities to see and make connections between objects that are only possible through comparative analyses. In concept, Interwoven is seeking to accomplish two objectives. First, the platform empowers users to explore connected databases visually and to envision stylistic, historical, and other links between objects from diverse collections. Second, the platform takes on the task of executing functional AI-based metadata analytics and image analysis using Microsoft's Azure custom vision across its LOD to produce a corpus of connected objects from which users can “curate” their object list. While Interwoven is a welcome innovative case study on how digital methodologies can expand ways of engaging with art and how scholars carry out research, the limitations of existing museum metadata and the platform's back-end system pose certain challenges to these objectives.
Computer vision and data scientists have worked on refining image-retrieval systems and the application of both text- and image-based analysis to art collections for almost two decades, with varying levels of success.1 The most successful art-history-oriented projects that work with corpora of images are ones that are not centered on image analysis, but use metadata associated with images as their dataset. Since this method relies on the accuracy and efficacy of existing metadata, often the dataset is generated as part of the project. The interactive diagram produced to accompany the exhibition Inventing Abstraction, 1910–1925 (Museum of Modern Art, New York, December 23, 2012–April 15, 2013) is a great example of this type of digital art history (DAH). But when databases work with extracted metadata from museums and archival collections without cleaning up the data (i.e., working with “tidy data”), these database-based DH projects run into critical problems.
At the other end of the spectrum of DH methods, when image analysis through CV or in other ways is carried out, larger projects have been limited to identifying form-based parts of images (e.g., identifying animals or faces within paintings) or aiding image retrieval (e.g., gathering replicas, duplicates, or images similar to a hero image). Projects in the former category include large-scale CV projects such as those that enable Google Lens. More specific but smaller art-history-CV projects that focus on object detection and form analysis are carried out prominently in EU and UK institutions, including at Ludwig-Maximilians-University Munich. In the latter category of projects that focus on using machine learning for image retrieval is John Resig's early work in this field that created an LOD for ukiyo-e block prints. Art historians have, to some extent, been critical of these interventions as they do not always afford or include the types of critical and contextual analyses that are at the core of the discipline today.2
While it is not immediately clear from available documentation how Interwoven uses metadata and images, I have estimated from a close practical study of the platform that the project has used both metadata and image analysis options available through Microsoft's Azure AI, possibly in combination with additional digital tools. In so doing, Interwoven runs up against the same set of problems that previous medium-to-large digital visual-analysis projects have encountered. In the next few paragraphs, I will detail some of these issues and the potential benefits for the field of art history and museum studies in resolving these issues.
So How Does Interwoven Work?
When you first open the Interwoven interface, Microsoft AI sources entries from the databases of MAP and the other partnering institutions. Once the full database is loaded, you are provided with 1,733 artworks categorized as textiles from MAP's collection. These appear as a gallery on the main field of the interface. You can browse these artworks by scrolling or use the filtering function from the navigation bar on the left to narrow your search. The platform provides seven filters: type, date, pattern, color, medium, region, and function. You can then pick one of the artworks and start your “custom journey.” When you select an artwork to start your journey, the platform retrieves artworks from all partnering databases to provide you with the type of image interface shown in figure 1.
For me, this visual interface is not the most helpful as the categories that are “threaded” to the main object are sometimes vague and confusing. For example, I chose a nineteenth-century Kalamkari prayer mat as my first object (MAP, TXT.00214). My objective with this test case was to create a curated journey for Kalamkari textiles that relate to one another stylistically. The interface provided me with fifty artworks connected to this prayer mat within six categories: trade (twenty-five); cotton, painted, stitched (five); cotton, painted, woven (five); textiles, wall hangings (one); textile (one); ritual and sacred, trade (five); and trade, ritual and sacred (five). (I am still trying to account for the three artworks that do not appear to be relevant to these categories.) Since I was using Interwoven in this test case for a visual comparison of artworks, choosing the “grid” view (video 1) instead of the “network” view was far more helpful.
The video below shows how I used the grid view to select the second artwork for my “custom journey.”
The grid view allowed me to easily browse through available artworks and identify five Kalamkari prayer mats and one wall hanging that resembled a prayer mat from the fifty query results that I was interested in adding to the custom journey. I made these identifications based on a rudimentary visual analysis, choosing textiles with an architectural structure resembling a mihrab. For an art historian working on finding objects that are similar to each other to build a corpus or conduct comparative visual analyses, the grid feature and the interface in general are great tools. However, there are multiple problems with the way the search query functions as well as with the functionality of the custom journey option, which is the only interactive offering that the platform provides at present.
The main issue with the custom journey option is that it only allows the selection of one artwork at a time from which to build a journey. For my first selection for a custom journey, I selected Kalamkari prayer mat (MAP, TXT.00214). This provided me with the set of fifty artworks shown in figures 1 and video 1. For the second artwork of the journey, I selected from my query return a late-nineteenth-century Kalamkari prayer mat (MAP, TXT.00039). In the screen recording (video 1), an “add to custom journey” button appears in the pop-up box on the main field when I click on the chosen textile.
Once I pick the second object, however, I have no way to return to the first object, with which I started my journey, or to select the other five Kalamkari textiles I had identified from the initial network of fifty objects. For the third object, I can only select from the network connected to the second artwork in my journey. In the screen recording in video 2, when I try to click on the third textile from the left in the top row (from the network of my first selected textile), the pop-up box does not provide me with the “add to custom journey” option. To add the third textile that I wanted to include in my custom journey—a twentieth-century Kalamkari prayer mat (MAP, TXT.00596)—I had to find it within the network of the second textile I chose. This greatly reduces the functional efficiency of this program since the unidirectionality disables the purported ability to link artworks from across the LOD.
By continuing with the unidirectional curation, I curated in total seven Kalamkari textiles, all having an arch as the central motif in the main field, with five of them being Kalamkari prayer mats from the nineteenth and twentieth centuries, all from MAP's collection; another twentieth- to twenty-first-century Kalamkari (MAP, TXT.01437) that was not a prayer mat, also from MAP; and one nineteenth-century Iranian Kalamkari textile titled “chintz door curtain” from the Cleveland Museum of Art collection (inv. no. 171386).
Scrolling through the fifty artworks from the first object network in my custom journey, the relevance or connections of the artworks become minimal, as seen in figures 2 and 3, and practically nonexistent in the case of the last few artworks (fig. 4).
Without access to the code for the platform, it is difficult to pinpoint why these less relevant connections, or rather connection discrepancies, are present, but one can make some educated guesses through analyzing the custom-journey query results. I posit that the diminishing return of accuracy of relevant objects in the search query is a function of two continual problems in DH projects that include artworks. The first is the problem of “messy” museum metadata, and the second is the nature of object detection presently possible through most image AI software that is not built for analyzing artworks. I will briefly elaborate on both these problems and how they produce challenges for the Interwoven project.
Interwoven's LOD relies on open-access museum metadata from MAP and a host of museums in the US, the UK, and India. Museum metadata provides two challenges to DH projects: first, museum metadata is prepared for human readability, not for machine readability. Its use therefore poses the classic DH problem of messy data and requires a huge amount of labor to clean up that data and make it ready for computational analysis. Second, metadata language across museums is not standardized. While specific types of categories are used—title, description, period, medium, technique, and so on—museum entry formats for these categories vary greatly. For example, some museums enter the time period as “19th century,” while others may enter it as “1800–1899,” and yet others may enter it as “circa 19th century” or “c. 1850–75.” In the case of certain categories, such as medium and technique, some museums combine both of these into the single category of “medium and technique.” Terms such as probably, circa, may be, and approximately—terms that are anathema to computational DH that relies on binaries—abound in museum metadata. Interwoven queries fourteen disparate museum databases with metadata content that varies in quality, and most are not set up for machine readability.
Second, metadata from museums where there are a large number of curators and researchers employed offers greater detail than in museums that do not have the resources to generate research on their collections. Categories such as description, subject headings, and tags are also more of an art than a science, and careful construction of content in these categories requires specialized expertise including those of subject-matter experts, archivists, and data managers.
The issues addressed above directly effect the quality of search returns on the Interwoven platform. For example, in figure 4 the Kalamkari prayer mat at the top right is linked to a second-century Jewish coin from the Harvard Art Museums (HAM) collection. Since there are no immediate stylistic or formal analytical commonalities between the textile and the coin, I sleuthed through the metadata to solve the mysterious connection between these two items. I found that the coin had been erroneously categorized in HAM's metadata within “wall hangings,” and this error along with the description and tags for the coin that mention trees, flowers, and fruits—things also mentioned in the Kalamkari's description—may be the contributing factor to the “connection” between these two artworks.
These challenges to accurately connect artworks are further exacerbated by the CV component of Microsoft AI. For example, in figure 5 Kalamkari prayer mat (MAP, TXT.00040) is “connected” to a nineteenth-century photograph of the iron pillar at Qutb Minar, Delhi, from the Weavers Studio Museum collection (inv. no. PHY.00045). This connection, I presume, is generated both through metadata overlap as well as problems with AI object detection within images. Here, the architectural structure of the Qutb Minar in the background resembles the Mihrab design of the prayer mat. Since most image AI software is trained for object detection with photographs, it is quite possible that Microsoft AI identifies the textile's design as an architectural feature and provides a connection between these two images.
There could be other related human errors creating issues with search queries such as inaccurate pull requests or ineffective flowcharts that direct search queries. Without access to the back end of the platform and the code, it is impossible to say whether these deductions are accurate. But the issues raised above also point out the need for research institutions to spearhead cutting-edge and innovative art historical research, digital or otherwise, rather than leaving this to private entities.
Author Biography
Deepthi Murali, PhD (University of Illinois at Chicago), is research assistant professor at the Roy Rosenzweig Center for History and New Media, George Mason University. She is an art historian of South Asia specializing in eighteenth- and nineteenth-century decorative arts of littoral South Asia with a particular interest in wood and ivory objects, and cotton textiles. Her current major digital humanities project, Connecting Threads: Subaltern Histories of Global Indian Textiles, traces the history of production of checked and striped cotton textiles in southern India and their use by communities of color in the Greater Caribbean Region. She is also the co-principal investigator for the Smithsonian History and Culture Access Consortium project and an experimental digital art history project, Digital Chintz, that is currently using the corpus of Palampore textiles in Western museum collections to develop non-Western art-history-specific digital methodologies. E-mail: dmurali2@gmu.edu
Notes
- Sabine Lang and Björn Ommer, “Transforming Information into Knowledge: How Computational Methods Reshape Art History,” Digital Humanities Quarterly 15.3 (2021). ⮭
- Raphael Rosenberg, “Bridging Art History, Computer Science and Cognitive Science: A Call for Interdisciplinary Collaboration,” Zeitschrift für Kunstgeschichte 79.3 (2016): 305–14; and Claire Bishop, “Against Digital Art History,” Humanities Futures, n.d., accessed October 23, 2023. ⮭