WISMIR3: A Multi-Modal Dataset to Challenge Text-Image Retrieval Approaches

Florian Schneider, Chris BiemannACL 2024

TL;DR

This paper introduces WISMIR3, a new, large-scale dataset of Wikipedia text-image pairs designed to challenge current text-image retrieval models. WISMIR3 exhibits greater linguistic complexity and diversity than existing benchmarks like COCO and Flickr30k, and the authors demonstrate that state-of-the-art models perform significantly worse on it, highlighting a gap in current model generalization.

What Problem Does This Paper Address?

Text-image retrieval, the task of finding relevant images given a text query (or vice-versa), has seen remarkable progress, with models achieving over 90% recall on popular datasets. However, this paper argues that this success might be an overestimation due to the simplicity and similarity of commonly used training and evaluation datasets like COCO and Flickr30k. These datasets primarily feature short, simple captions created by crowdsourcing workers for everyday scenes. This raises a crucial question: can current models truly understand and retrieve images based on more complex, diverse, and real-world descriptions, or are they overfitting to the specific characteristics of existing benchmarks? The authors aim to address this by creating a dataset that better reflects the complexity of real-world text-image understanding.

Methods

The authors developed a sophisticated automatic ETL (Extract, Transform, Load) pipeline to construct the WISMIR3 dataset. This pipeline started with the WikiCaps dataset, which contains Wikipedia image IDs and their associated English captions. The process involved:

  • Corpus Statistics Collection: Extensive linguistic features were extracted for each caption using spaCy, including token and sentence counts, part-of-speech tags, named entities, and dependency relations.
  • Filtering: Captions were filtered based on length, sentence structure, and the presence of non-depictable words (e.g., "URL," "Sarcasm") to ensure relevance and avoid noise. Language filtering was also applied to retain only English captions.
  • Duplicate Removal: Captions were de-duplicated, with each caption describing at most five different images (a reversal of the common practice in COCO/Flickr30k).
  • Image Processing: Images were downloaded, filtered for size and format errors, and then transformed into RGB, resized while maintaining aspect ratio, compressed, and saved as PNGs.

This pipeline resulted in a clean dataset of approximately 300K text-image pairs. Additionally, the authors pre-computed and released image features from a Faster-R-CNN model and image/text embeddings from CLIP models to facilitate dataset usage.

Datasets

  • WISMIR3: The primary contribution, a new multi-modal dataset comprising ~300K text-image pairs derived from Wikipedia. It is designed to be more linguistically complex and diverse than existing datasets.
  • WikiCaps: The source dataset for WISMIR3, containing ~3.8M text-image pairs from Wikipedia.
  • COCO (Common Objects in Context): A widely used dataset for various computer vision tasks, including text-image retrieval. It contains ~123K images with 5 descriptive captions each.
  • Flickr30k: Another popular dataset for text-image retrieval, featuring ~30K images with 5 descriptive captions each, often depicting everyday activities.

WISMIR3 is presented as a benchmark to challenge models trained or evaluated on COCO and Flickr30k, highlighting potential generalization issues.

Key Insights and Ideas

  • Dataset Bias is Real: The paper strongly suggests that current high performance on COCO and Flickr30k might be inflated due to the datasets' simplicity and similarity to training data, rather than true generalization.
  • Wikipedia as a Richer Source: Leveraging Wikipedia content for text-image pairs introduces greater linguistic complexity, a wider range of topics, and more descriptive captions, making it a more challenging benchmark.
  • Named Entities Matter: WISMIR3 has a significantly higher ratio of named entities in captions compared to COCO/Flickr30k. The authors argue this is crucial for real-world retrieval systems where users often search for specific entities.
  • Readability Gap: WISMIR3 captions are linguistically more complex, requiring a college-level reading comprehension, whereas COCO/Flickr30k captions are understandable by an average 4th-6th grader.

Key Results

  • Performance Drop: When evaluating state-of-the-art text-image retrieval models (like CLIP and UNITER) on WISMIR3, their performance (e.g., Recall@5) is significantly lower compared to their performance on COCO and Flickr30k. For instance, CLIP achieves ~72.4% Recall@5 on WISMIR3, compared to ~81.5% on COCO and ~90.6% on Flickr30k.
  • CLIP's Relative Strength: Despite the overall performance drop, the CLIP model (trained on a massive, diverse dataset) still performs notably better on WISMIR3 than UNITER or TERAN, suggesting its broader pre-training helps it generalize better to more complex data.
  • Linguistic Differences Quantified: The paper provides detailed statistical comparisons showing WISMIR3 has more tokens per caption, a higher ratio of nouns and named entities, and higher readability scores than COCO and Flickr30k.

Limitations and Future Work

  • Dataset Size: While ~300K pairs is substantial, it's smaller than some other large-scale datasets like LAION-5B.
  • Focus on English: The current WISMIR3 dataset is primarily in English, limiting its applicability for multilingual research.
  • Future Work: The authors suggest that WISMIR3 can be used to develop and evaluate more robust text-image retrieval models that can generalize better to real-world scenarios. They also implicitly encourage the creation of even more diverse and complex datasets.

Summary updated

BibTeX
@inproceedings{schneider2024wismir,
  author = {Florian Schneider and Chris Biemann},
  title = {{WISMIR3: A Multi-Modal Dataset to Challenge Text-Image Retrieval Approaches}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure