TL;DR
This paper introduces a task-level taxonomy for identifying and mitigating biases in AI-supported historical image retrieval. By grounding general machine learning fairness concepts in the specific challenges of colonial-era archives, the authors reveal why standard "off-the-shelf" mitigation techniques often fail when applied to degraded, monochrome, and historically skewed cultural heritage data.
What Problem Does This Paper Address?
Galleries, libraries, archives, and museums (GLAM institutions) are sitting on mountains of digitized historical imagery, much of it lacking descriptive metadata. To make these collections searchable, researchers are increasingly turning to machine learning (ML) for automated tagging and captioning.
However, historical archives are not neutral. They reflect the power dynamics and prejudices of the era in which they were created—particularly in colonial contexts where "archival silences" mean colonizers are named and documented while the colonized remain anonymous or described through racist stereotypes. When modern AI models, trained on contemporary web data, are applied to these archives, they don't just preserve these biases; they risk amplifying them. The paper addresses the gap between general ML fairness research and the messy, socio-technical reality of historical image retrieval.
Methods
The authors adopt a socio-technical framing, bridging the gap between humanities-based critiques of archives and technical ML fairness taxonomies. Their primary methodological contribution is a systematic mapping of bias categories onto six specific retrieval tasks:
- Entity Linking: Resolving mentions of people/places to a knowledge base.
- Multi-Label Image Classification: Predicting content keywords from a controlled vocabulary.
- VLM Captioning: Using Vision-Language Models to generate descriptive text.
- Text-Based Image Retrieval: Matching free-text queries to images (e.g., using CLIP).
- Near-Duplicate Linking: Identifying copies or variations of the same image.
- Query-by-Example: Finding visually similar images.
The authors use the VABiKo project—a collection of over 50,000 German colonial-era photographs—as a concrete case study to test how these biases manifest in practice.
Datasets
The paper centers on the VABiKo collection, held by the Universitätsbibliothek J. C. Senckenberg. This dataset includes ~62,519 records of photographs from 1887–1943.
- Characteristics: Predominantly monochrome, often physically degraded (scanned from glass-plate negatives), and featuring sparse metadata.
- Metadata Disparity: Only 3.5% of records ascribed as "African" carry a named individual, compared to 22% for those ascribed as "European."
The authors also reference standard fairness benchmarks like FairFace and FACET, noting that these contemporary datasets are often inadequate for evaluating historical imagery.
Key Insights and Ideas
- The Failure of Standard Mitigations: Common data augmentation techniques like "color jittering" (randomly changing colors to make a model robust) are useless for monochrome historical photos. Similarly, "adversarial debiasing" developed for modern data can actually decrease accuracy for marginalized groups when applied to historical contexts.
- Measurement Bias from "Image-Only" Prediction: Many ML tasks try to predict tags from the pixels alone. However, in historical archives, the "truth" often lies in the context (date, location, photographer) that isn't visible in the image.
- Representation Bias in Pre-training: Models like CLIP are trained on modern web data, which over-represents Western/contemporary aesthetics. This creates a "domain gap" when the model encounters 100-year-old, degraded, black-and-white imagery.
- Curatorial vs. Algorithmic Intervention: The authors argue that some biases are best handled by curators (e.g., adding context to offensive terms) rather than algorithms (e.g., deleting them), as the original descriptions serve as historical evidence.
Key Results
The paper provides a comprehensive Bias Taxonomy Table (Table 2) that links specific tasks to bias types. For example:
- Entity Linking is plagued by Popularity Bias, where models favor well-documented colonial figures over under-represented colonized individuals.
- Visual Similarity Search suffers from Content Production Bias because near-duplicates in archives often arise from different physical reproduction methods (lantern slides vs. prints) rather than the camera-angle variations assumed by modern benchmarks.
- VLM Captioning was found to amplify racial disparities already present in training data, sometimes varying the sentiment of a caption based solely on the subject's skin color.
Limitations and Future Work
The authors highlight a critical "evaluation gap": there is currently a lack of domain-specific evaluation data for historical imagery. While benchmarks exist for modern faces, there are no comparable resources for measuring bias in colonial-era archival retrieval.
Future work should focus on:
- Developing historical-specific fairness benchmarks.
- Exploring transductive self-supervised fine-tuning on target collections to handle near-duplicate detection without needing external labels.
- Improving hierarchical multi-label learning to help models handle the "long tail" of rare historical categories.
Summary updated