TL;DR
This paper introduces "WikiScenes with Descriptions," a novel dataset for aligning text and images within complex Wikipedia articles. They demonstrate that current state-of-the-art vision-language models struggle with this task, highlighting the need for models that can better understand descriptive language in longer, naturally occurring documents.
What Problem Does This Paper Address?
The field of Vision and Language (V&L) research heavily relies on datasets where text and images are explicitly aligned, such as image captioning or Visual Question Answering (VQA) datasets. However, real-world multimodal documents, like Wikipedia articles, present a much more complex challenge. In these articles, image-text relationships are often implicit, and alignments are not explicitly marked. This makes it difficult to assess how well V&L models can ground language in visual content within longer, more naturalistic texts. Existing work often relies on unsupervised methods due to a lack of suitable evaluation data. This paper aims to bridge this gap by creating a dataset that provides ground-truth annotations for image-text alignments within multi-paragraph, multi-image Wikipedia articles, enabling a more robust evaluation of V&L models on this complex task.
Methods
The authors developed a two-step annotation procedure to create their dataset. First, annotators performed a "preselection" of paragraphs and images that were likely to have an image-text alignment. This involved a quick judgment of whether a paragraph contained references to visual elements in an image, while excluding non-photographic images and focusing on visible content.
The second step involved "fine-grained image-paragraph annotations." Here, crowdworkers were presented with image-paragraph pairs and instructed to highlight specific text spans within the main article text that described visual content present in the image. This process aimed to capture descriptive sentences and word-level alignments.
For evaluation, the paper explores two main approaches:
- Sentence-Caption Alignments: They use unimodal text-only sentence embeddings (SBERT) and multimodal sentence embeddings (MCSE) to measure semantic similarity between article sentences and image captions. This allows them to assess how well captions represent the descriptive content of the main text.
- Sentence-Image Alignments: They employ state-of-the-art multimodal models like CLIP and VILT to directly align sentences with images. This tests the models' ability to ground textual descriptions in visual content.
The paper also investigates the linguistic differences between descriptive sentences in the main text and the original image captions, using POS and NER tag distributions.
Datasets
The primary dataset introduced in this paper is WikiScenes with Descriptions. This dataset is an extension of the existing WikiScenes corpus (Wu et al., 2021), which contains multimodal Wikipedia articles about buildings. The authors enriched this corpus with detailed annotations of image-text alignments at the sentence and word level.
They also generated captions for the images using two captioning models:
- ClipCap: A lightweight caption generation model based on CLIP.
- IBM-MAX: A model inspired by Vinyals et al. (2017).
These generated captions are compared against the original Wikipedia captions.
Key Insights and Ideas
- Implicit Alignment is Hard: The core finding is that aligning text and images in naturally occurring, complex documents like Wikipedia articles is significantly more challenging for current V&L models than in curated datasets.
- Descriptive Sentences Differ from Captions: The study reveals systematic linguistic differences between descriptive sentences found in the main body of Wikipedia articles and the captions that accompany images. This suggests that V&L research, often focused on captions, might be missing out on a richer understanding of visual language grounding.
- Intra-document Retrieval is a Bottleneck: Simple similarity-based retrieval using captions as a proxy for image-text alignment works reasonably well at the paragraph level but drastically degrades at the article level. This indicates that models struggle to navigate the broader context of an entire document.
- Multimodal Models Don't Always Outperform Unimodal: Surprisingly, in some intra-document retrieval scenarios (especially at the article level), unimodal sentence embedding models (SBERT, MCSE) performed comparably or even better than multimodal models like CLIP and VILT. This suggests that current multimodal models might lack a strong understanding of "depictability" – the ability to detect language that is visually grounded.
Key Results
- Retrieval Accuracy: Top-1 retrieval accuracy for sentence-to-caption/image and caption/image-to-sentence tasks was generally "very poor," even in the simpler "match" condition. The highest top-1 accuracy achieved was 0.66 for paragraph-level sentence-to-image retrieval using CLIP. Article and corpus-level accuracies were drastically lower.
- Linguistic Analysis: POS tag distributions showed no significant difference between matched and unmatched sentences, suggesting annotators weren't biased by specific grammatical structures. However, main text sentences differed significantly from captions. NER tag analysis indicated that human-written wiki captions primarily featured "PERSON" entities, while generated captions avoided them.
- Embedding Space Visualization: A TSNE plot of SBERT embeddings showed that IBM-MAX captions clustered together and were further from the main text and ClipCap captions, suggesting distinct linguistic styles.
Limitations and Future Work
Limitations: * The dataset is relatively small and domain-specific (buildings), meaning results might not generalize to other domains. * The models used for retrieval were not fine-tuned on this specific dataset, which could impact performance.
Future Work: * Developing models that can jointly exploit sentence-image and sentence-caption alignments. * Creating multimodal models that can handle text with non-descriptive language. * Exploring how to better integrate multimodal models for fine-grained paragraph-level distinctions and improve their understanding of "depictability" for longer documents.
Summary updated