MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding

Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil, Shih-Fu Chang, Alexander G. Schwing, Heng Ji

Introduction

While humans effortlessly synthesize information from images and text to understand news, AI models often struggle with multi-modal reasoning. Most Visual Question Answering (VQA) datasets are restricted to multiple-choice formats or require only visual analysis. MuMuQA introduces a benchmark for multi-hop question answering that necessitates "cross-media grounding"—identifying an object in an image based on visual attributes and using that identity to extract an answer from associated news text.

The MuMuQA Task

The MuMuQA task requires a model to process an image-caption pair alongside a long-form news article to extract a specific text span as the answer. This process is structured in two "hops":

  1. Hop 1 (Image Entity Grounding): The model must link a visual attribute in the question (e.g., "blue tie") to a "bridge item" (an entity name like "Benjamin Netanyahu") found in the image-caption pair.
  2. Hop 2 (Textual Reasoning): The model uses this bridge item to locate the final answer within the news body.
Examples of the MuMuQA task requiring multi-hop reasoning across images, captions, and news text.

Dataset and Generation

The MuMuQA benchmark consists of 1,384 human-curated questions based on Voice of America news articles (2009–2017). To address the high cost of human annotation, the authors developed a multimedia data augmentation framework to generate a "silver-standard" training set of 20,000 examples. This framework uses Information Extraction (IE) to build entity graphs, a fine-tuned BART model for question generation, and visual attribute extraction to replace entity names with visual traits, forcing the model to utilize the image.

Distribution of question types in the MuMuQA dataset.

Experimental Results

The researchers evaluated three primary approaches: a multi-hop text-only baseline, an end-to-end multimedia model (OSCAR), and a pipeline-based multimedia system. Results indicate a significant performance gap between AI and humans:

Model Dev F1 (%) Test F1 (%)
Multi-hop Text-only 25.6 24.8
End-to-End Multimedia (OSCAR) 12.1 11.5
Pipeline-based Multimedia 37.3 32.6
Human Baseline - 66.5

The pipeline-based model outperformed the end-to-end OSCAR model, suggesting that current vision-language models struggle with reasoning over long-form text. The "bridge item" remains the primary bottleneck; grounding systems correctly identified the bridge entity in only 45% of cases.

Conclusion

MuMuQA highlights the complexity of cross-media reasoning. Future work should focus on improving visual-textual coreference, handling more complex visual attributes (actions, emotions), and expanding to other modalities like video and audio.

Summary updated

Abstract

Recently, there has been an increasing interest in building question answering (QA) models that reason across multiple modalities, such as text and images. However, QA using images is often limited to just picking the answer from a pre-defined set of options. In addition, images in the real world, especially in news, have objects that are co-referential to the text, with complementary information from both modalities. In this paper, we present a new QA evaluation benchmark with 1,384 questions over news articles that require cross-media grounding of objects in images onto text. Specifically, the task involves multi-hop questions that require reasoning over image-caption pairs to identify the grounded visual object being referred to and then predicting a span from the news body text to answer the question. In addition, we introduce a novel multimedia data augmentation framework, based on cross-media knowledge extraction and synthetic question-answer generation, to automatically augment data that can provide weak supervision for this task. We evaluate both pipeline-based and end-to-end pretraining-based multimedia QA models on our benchmark, and show that they achieve promising performance, while considerably lagging behind human performance hence leaving large room for future work on this challenging new task.
BibTeX
@inproceedings{reddy2022mumuqa,
  author = {Revanth Gangi Reddy and Xilin Rui and Manling Li and Xudong Lin and Haoyang Wen and Jaemin Cho and Lifu Huang and Mohit Bansal and Avirup Sil and Shih-Fu Chang and Alexander G. Schwing and Heng Ji},
  title = {{MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21370},
}
Figure