Introduction
While humans effortlessly synthesize information from images and text to understand news, AI models often struggle with multi-modal reasoning. Most Visual Question Answering (VQA) datasets are restricted to multiple-choice formats or require only visual analysis. MuMuQA introduces a benchmark for multi-hop question answering that necessitates "cross-media grounding"—identifying an object in an image based on visual attributes and using that identity to extract an answer from associated news text.
The MuMuQA Task
The MuMuQA task requires a model to process an image-caption pair alongside a long-form news article to extract a specific text span as the answer. This process is structured in two "hops":
- Hop 1 (Image Entity Grounding): The model must link a visual attribute in the question (e.g., "blue tie") to a "bridge item" (an entity name like "Benjamin Netanyahu") found in the image-caption pair.
- Hop 2 (Textual Reasoning): The model uses this bridge item to locate the final answer within the news body.
Dataset and Generation
The MuMuQA benchmark consists of 1,384 human-curated questions based on Voice of America news articles (2009–2017). To address the high cost of human annotation, the authors developed a multimedia data augmentation framework to generate a "silver-standard" training set of 20,000 examples. This framework uses Information Extraction (IE) to build entity graphs, a fine-tuned BART model for question generation, and visual attribute extraction to replace entity names with visual traits, forcing the model to utilize the image.
Experimental Results
The researchers evaluated three primary approaches: a multi-hop text-only baseline, an end-to-end multimedia model (OSCAR), and a pipeline-based multimedia system. Results indicate a significant performance gap between AI and humans:
| Model | Dev F1 (%) | Test F1 (%) |
|---|---|---|
| Multi-hop Text-only | 25.6 | 24.8 |
| End-to-End Multimedia (OSCAR) | 12.1 | 11.5 |
| Pipeline-based Multimedia | 37.3 | 32.6 |
| Human Baseline | - | 66.5 |
The pipeline-based model outperformed the end-to-end OSCAR model, suggesting that current vision-language models struggle with reasoning over long-form text. The "bridge item" remains the primary bottleneck; grounding systems correctly identified the bridge entity in only 45% of cases.
Conclusion
MuMuQA highlights the complexity of cross-media reasoning. Future work should focus on improving visual-textual coreference, handling more complex visual attributes (actions, emotions), and expanding to other modalities like video and audio.
Summary updated