Introduction
While Question Answering (QA) has seen massive leaps in performance, most research remains siloed. Text-only QA (like SQuAD) ignores visual context, and standard Visual QA (VQA) often relies on multiple-choice shortcuts or simple image-to-label mapping. In the real world—specifically in news media—images and text are deeply intertwined. An image might show a person, but the text provides the "who, what, and why."
The authors identify a significant gap: current models struggle with cross-media grounding. They can't easily link a visual description in a question to a named entity in a caption and then perform a "second hop" of reasoning into the body of an article to find a specific answer. MuMuQA (Multimedia Multi-hop QA) is designed to force models to perform this complex, multi-step reasoning.
The MuMuQA Task and Methodology
The MuMuQA task is formulated as an extractive QA problem. Given an image-caption pair and a news article, the model must answer a question by extracting a span of text from the article. To succeed, the model must perform two distinct "hops":
- Hop 1: Identify the "bridge item" (the entity) by grounding visual attributes mentioned in the question (e.g., "the person on the left") to the image and caption.
- Hop 2: Use that bridge item to navigate the news body text and find the final answer.
Silver-Standard Data Augmentation
Because human annotation is expensive, the authors developed a pipeline to automatically generate 20,000 "silver-standard" training examples. This pipeline uses Multimedia Information Extraction (IE) to ground entities to bounding boxes and Visual Attribute Extraction to identify colors, positions, and accessories. A fine-tuned BART model generates questions based on the text, which are then edited to replace named entities with visual attributes, ensuring the image is necessary for disambiguation.
Datasets and Benchmark
The primary contribution is the MuMuQA Benchmark, a human-annotated evaluation set containing 1,384 questions (263 for development, 1,121 for testing) based on news articles from Voice of America (VOA). The source data consists of over 100,000 news articles from 2009–2017.
The dataset was specifically filtered to ensure that a text-only model couldn't "cheat" by guessing the answer from the text alone. The image is required for disambiguation, such as when two different people are mentioned in the text, but only the image identifies which one is the subject of the question.
Key Results and Insights
The results highlight a significant gap between human-level multimedia reasoning and current AI capabilities:
- Human Performance: Humans achieved an F1 score of 66.5%.
- Pipeline-based Model: The top-performing AI reached an F1 of 32.6%.
- End-to-End Model (OSCAR): Performed poorly with only an 11.5% F1 score.
The pipeline model only identified the correct "bridge" entity about 31.4% of the time, confirming that cross-media grounding is the primary hurdle. Interestingly, the end-to-end model (OSCAR) struggled significantly. While OSCAR excels at image-caption alignment, it is not designed to reason over the much longer sequences found in news body text.
Conclusion and Future Work
MuMuQA reveals that identifying the correct entity from an image—the "bridge item"—remains a major bottleneck. Future work must address the noise in automated IE systems and develop architectures capable of long-form reasoning across modalities. Expanding these benchmarks to include video and audio will be essential for creating truly robust multimedia QA systems.
Summary updated