A Task-Level Bias Taxonomy for Historical Image Retrieval

Omkar Gavali, Ratan Sebastian, Christoph Rippe, Ralph Ewerth, Anett HoppearXiv 2026 · arXiv

TL;DR

This paper introduces a task-level taxonomy for identifying and mitigating biases in AI-supported historical image retrieval. By grounding general machine learning fairness concepts in the specific challenges of colonial-era archives, the authors reveal why standard "off-the-shelf" mitigation techniques often fail when applied to degraded, monochrome, and historically skewed cultural heritage data.

What Problem Does This Paper Address?

Galleries, libraries, archives, and museums (GLAM institutions) are sitting on mountains of digitized historical imagery, much of it lacking descriptive metadata. To make these collections searchable, researchers are increasingly turning to machine learning (ML) for automated tagging and captioning.

However, historical archives are not neutral. They reflect the power dynamics and prejudices of the era in which they were created—particularly in colonial contexts where "archival silences" mean colonizers are named and documented while the colonized remain anonymous or described through racist stereotypes. When modern AI models, trained on contemporary web data, are applied to these archives, they don't just preserve these biases; they risk amplifying them. The paper addresses the gap between general ML fairness research and the messy, socio-technical reality of historical image retrieval.

Methods

The authors adopt a socio-technical framing, bridging the gap between humanities-based critiques of archives and technical ML fairness taxonomies. Their primary methodological contribution is a systematic mapping of bias categories onto six specific retrieval tasks:

  1. Entity Linking: Resolving mentions of people/places to a knowledge base.
  2. Multi-Label Image Classification: Predicting content keywords from a controlled vocabulary.
  3. VLM Captioning: Using Vision-Language Models to generate descriptive text.
  4. Text-Based Image Retrieval: Matching free-text queries to images (e.g., using CLIP).
  5. Near-Duplicate Linking: Identifying copies or variations of the same image.
  6. Query-by-Example: Finding visually similar images.

The authors use the VABiKo project—a collection of over 50,000 German colonial-era photographs—as a concrete case study to test how these biases manifest in practice.

Fig. 1: Types of Bias, adapted from [24]

Datasets

The paper centers on the VABiKo collection, held by the Universitätsbibliothek J. C. Senckenberg. This dataset includes ~62,519 records of photographs from 1887–1943.

  • Characteristics: Predominantly monochrome, often physically degraded (scanned from glass-plate negatives), and featuring sparse metadata.
  • Metadata Disparity: Only 3.5% of records ascribed as "African" carry a named individual, compared to 22% for those ascribed as "European."

The authors also reference standard fairness benchmarks like FairFace and FACET, noting that these contemporary datasets are often inadequate for evaluating historical imagery.

Key Insights and Ideas

  • The Failure of Standard Mitigations: Common data augmentation techniques like "color jittering" (randomly changing colors to make a model robust) are useless for monochrome historical photos. Similarly, "adversarial debiasing" developed for modern data can actually decrease accuracy for marginalized groups when applied to historical contexts.
  • Measurement Bias from "Image-Only" Prediction: Many ML tasks try to predict tags from the pixels alone. However, in historical archives, the "truth" often lies in the context (date, location, photographer) that isn't visible in the image.
  • Representation Bias in Pre-training: Models like CLIP are trained on modern web data, which over-represents Western/contemporary aesthetics. This creates a "domain gap" when the model encounters 100-year-old, degraded, black-and-white imagery.
  • Curatorial vs. Algorithmic Intervention: The authors argue that some biases are best handled by curators (e.g., adding context to offensive terms) rather than algorithms (e.g., deleting them), as the original descriptions serve as historical evidence.
Metadata challenges: In this example, the tag 'sisal agave' is the only way to identify an indistinct plant, illustrating how models must rely on sparse, expert-provided labels.

Key Results

The paper provides a comprehensive Bias Taxonomy Table (Table 2) that links specific tasks to bias types. For example:

  • Entity Linking is plagued by Popularity Bias, where models favor well-documented colonial figures over under-represented colonized individuals.
  • Visual Similarity Search suffers from Content Production Bias because near-duplicates in archives often arise from different physical reproduction methods (lantern slides vs. prints) rather than the camera-angle variations assumed by modern benchmarks.
  • VLM Captioning was found to amplify racial disparities already present in training data, sometimes varying the sentiment of a caption based solely on the subject's skin color.

Limitations and Future Work

The authors highlight a critical "evaluation gap": there is currently a lack of domain-specific evaluation data for historical imagery. While benchmarks exist for modern faces, there are no comparable resources for measuring bias in colonial-era archival retrieval.

Future work should focus on:

  1. Developing historical-specific fairness benchmarks.
  2. Exploring transductive self-supervised fine-tuning on target collections to handle near-duplicate detection without needing external labels.
  3. Improving hierarchical multi-label learning to help models handle the "long tail" of rare historical categories.

Summary updated

Abstract

In recent decades, galleries, libraries, archives, and museums (GLAM institutions) have digitised vast amounts of cultural heritage material, including historical imagery. A major challenge in making these collections publicly accessible is enabling effective search and discovery. Many historical images contain incomplete, sparse, or no descriptive metadata. Consequently, machine learning (ML) methods are increasingly being used to generate or enrich metadata to improve the searchability of such collections. However, artificial intelligence (AI)-based systems can reproduce or amplify existing biases, particularly when applied to historical imagery. Factors such as differences between the data used to train the models and the historical images on which they are applied, degraded image quality, monochrome photographs, and historically biased or incomplete descriptions can reduce model performance and introduce bias into search and retrieval results. In this paper, we examine different forms of bias in historical image collections and review current methods to mitigate them. We investigate how these biases arise and how they influence AI-supported search and retrieval systems.
BibTeX
@misc{gavali2026task,
  author = {Omkar Gavali and Ratan Sebastian and Christoph Rippe and Ralph Ewerth and Anett Hoppe},
  title = {{A Task-Level Bias Taxonomy for Historical Image Retrieval}},
  year = {2026},
  eprint = {2611.26019},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26019},
}
Figure