MIMOQA: Multimodal Input Multimodal Output Question Answering

Hrituraj Singh, Anshul Nasery, Denil Mehta, Aishwarya Agarwal, Jatin Lamba, Balaji Vasan SrinivasanNAACL 2021 · OpenReview

Abstract

Hrituraj Singh, Anshul Nasery, Denil Mehta, Aishwarya Agarwal, Jatin Lamba, Balaji Vasan Srinivasan. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2021.

Introduction

Traditional Multimodal Question Answering (QA) research has primarily focused on the input side—interpreting images and text to generate a response. However, the output has remained largely unimodal, typically restricted to a text string. This "unimodal bottleneck" limits the effectiveness of answers for queries where visual context is essential. For instance, explaining what a "Shillelagh" is becomes significantly more intuitive when a textual description is paired with a relevant image. This paper introduces MIMOQA, a paradigm and framework for generating multimodal answers (text + images) to enhance human understanding and grounding.

The MExBERT Framework

To address the challenges of joint text extraction and image retrieval, the authors propose MExBERT (Multimodal Extractive BERT). Unlike standard models that treat these tasks independently, MExBERT employs a dual-stream architecture designed for cross-modal interaction.

The MExBERT architecture featuring dual textual and visual streams with cross-modal attention.

The framework consists of two primary components:

  • Textual Stream: An augmented BERT-QA model incorporating cross-attention layers. These layers allow textual tokens to attend to visual features, refining the selection of the answer span based on available imagery.
  • Visual Stream: This stream processes global features (extracted via VGG-19) from a pool of candidate images. Through cross-attention to the textual stream, the model identifies which image most effectively complements the specific text being extracted.

By utilizing joint attention, MExBERT ensures that the selected image is not just relevant to the query in general, but specifically grounds the nuances of the extracted text span.

Dataset and Proxy Supervision

A significant hurdle in training MIMO models is the lack of datasets with labeled "best" images for specific answers. The researchers curated the MIMOQA Dataset by extending MS-MARCO and Natural Questions (NQ), scraping associated images from Wikipedia. This resulted in over 56,000 QA pairs and a pool of 400,000 images.

To train the model without explicit ground-truth image labels, the authors developed a proxy supervision heuristic. This system scores candidate images based on their proximity to the answer text within the source article and the TF-IDF similarity between the image's caption and the query/answer.

Experimental Results

MExBERT was evaluated against an "Extract & Match" (E&M) baseline, which first extracts text and then retrieves an image using a separate model (e.g., UNITER). MExBERT consistently outperformed this baseline in both automated and human evaluations.

Qualitative comparison between the Extract & Match (E&M) baseline and MExBERT, demonstrating superior grounding in complex queries.

Key findings include:

  • Improved Text Extraction: MExBERT achieved a BLEU score of 28.50, surpassing the baseline's 25.17. This suggests that visual context helps the model identify more accurate text spans.
  • Superior Image Selection: In human trials, MExBERT reached a Precision@1 of 0.386, compared to 0.284 for the E&M baseline.
  • Enhanced Human Understanding: Human experiments revealed that even when a text-only answer was deemed "sufficient," 87.5% of participants felt that the addition of a relevant image significantly enhanced their comprehension.

Conclusion

MIMOQA shifts the focus of QA research toward richer, multimodal outputs. By breaking the unimodal bottleneck with MExBERT, the authors demonstrate that joint attention between text and vision leads to more grounded and effective communication. Future work may explore generative visual outputs and expanding the framework to less structured data sources beyond Wikipedia.

Summary updated

BibTeX
@inproceedings{singh2021mimoqa,
  author = {Hrituraj Singh and Anshul Nasery and Denil Mehta and Aishwarya Agarwal and Jatin Lamba and Balaji Vasan Srinivasan},
  title = {{MIMOQA: Multimodal Input Multimodal Output Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=qEHZzGjohiI},
}
Figure