Progressive Evidence Refinement for Open-domain Multimodal Retrieval Question Answering

Shuwen Yang, Anran Wu, Xingjiao Wu, Luwei Xiao, Tianlong Ma, Cheng Jin, Liang HearXiv 2023 · DOI · OpenReview

TL;DR

This paper introduces PERQA, a novel two-stage framework for open-domain multimodal retrieval question answering. PERQA tackles the challenge of information loss in evidence processing and the gap between question and evidence feature extraction by employing a progressive evidence refinement strategy for retrieval and a multi-turn dialogue approach for answering, achieving state-of-the-art results on benchmark datasets.

What Problem Does This Paper Address?

The paper addresses a critical challenge in multimodal retrieval question answering (MMRQA): how to effectively extract and utilize information from diverse sources (text, images, tables) to answer complex questions. Existing models often struggle with two main issues:

  1. Information Loss: Compressing multimodal evidence into fixed-size features can lead to the loss of crucial fine-grained details.
  2. Feature Mismatch: A disconnect between how evidence and questions are processed can hinder the model's ability to identify the most relevant information for answering.

These limitations make it difficult for models to accurately answer questions that require reasoning over multiple pieces of evidence, especially when some evidence might be subtly related or contain distractors.

Methods

PERQA employs a two-stage approach:

  1. Progressive Evidence Refinement (for Retrieval): This stage aims to precisely identify the most relevant evidence.
  2. Evidence Initial Screening Module (EISM): A first pass filters out irrelevant sources by encoding questions and evidence (converting images to text for initial processing) and calculating cosine similarity.
  3. Iterative Evidence Retrieval Strategy (IER): This module refines the initial selection by iteratively considering candidate sources. It uses a regression module built on BERT to compute matching scores, progressively adding the most relevant sources to a set of "key evidence." This iterative process helps uncover logical relationships between pieces of evidence.
  4. Negative Sample Semi-Supervised Contrastive Learning (NSCL): To improve the robustness of the initial screening, this strategy uses distractor evidence as "negative samples" during training. This helps the model learn to distinguish truly relevant evidence from irrelevant but similar information.
  5. Multi-Cue Question Answering: This stage generates the answer using the refined evidence.
  6. Multi-Turn Dialogue Strategy: PERQA treats multimodal evidence as a historical dialogue, incorporating it alongside the question. This allows the model to leverage context from multiple pieces of evidence.
  7. Cross-Modal Attention: A cross-attention mechanism is used to capture the intricate relationships between different modalities of evidence and the question.
  8. Model Architecture: The answering model combines ViT (for image encoding), LLaMA (for text/question encoding), and LORA (for efficient fine-tuning).

Datasets

The paper evaluates PERQA on two widely used multimodal question-answering datasets:

  • WebQA: Contains multi-hop, multimodal question-answer pairs, often requiring 1-2 images or text snippets. Answers are free-form sentences.
  • MultimodalQA: Features multimodal questions from various sources (images, text, tables) with 16 question types, 13 of which require cross-modal retrieval. Answers are typically phrases.

Key Insights and Ideas

  • Progressive Refinement is Key: Simply retrieving a large set of evidence and then trying to filter it is less effective than a staged, iterative refinement process that builds upon previous selections. This mimics how humans might search for information.
  • Leveraging Distractors: Treating irrelevant "distractor" evidence as negative examples during training is crucial for building robust models that can truly distinguish signal from noise in complex multimodal datasets.
  • Multi-Turn Dialogue for Context: Framing multimodal evidence as a "conversation history" allows the model to better understand the relationships and dependencies between different pieces of information, leading to more accurate answers.
  • Image-to-Text for Initial Screening: Converting images to text for the initial evidence screening stage is an efficient way to bridge the modality gap without requiring complex, separate visual encoders at that early stage.

Key Results

PERQA achieves state-of-the-art (SOTA) performance on both the WebQA and MultimodalQA datasets for both retrieval and question-answering tasks.

  • On WebQA, PERQA surpasses previous SOTA by significant margins, including a 1.4% lead in retrieval F1 and a 6.7% lead in the crucial QA metric. It also shows strong performance in QA fluency and keyword accuracy.
  • On MultimodalQA, PERQA demonstrates substantial improvements over SOTA in both multi-modal (EM: 2.2%, F1: 2.9%) and single-modal inference (EM: 3.6%, F1: 4.4%).
  • A case study highlights PERQA's ability to correctly retrieve crucial intermediate evidence (E1) and then the final key evidence (E2) by leveraging logical order relationships, a task where a strong baseline (SKURG) failed.

Limitations and Future Work

  • Image-to-Text Conversion: While effective for initial screening, the paper notes that in the question-answering stage, they directly use the initial image information. This suggests that the image-to-text conversion might still involve some information loss compared to direct visual processing.
  • TableQ Performance: The authors mention that their model's performance on table-based questions might not be as strong as on image or text questions, potentially due to less specific pre-training on table data.
  • Future Work: The paper's success in multi-turn dialogue and progressive refinement suggests future work could explore more sophisticated dialogue management and even more granular evidence refinement strategies. The authors also look forward to providing valuable insights for future research in this domain.

Summary updated

Abstract

Pre-trained multimodal models have achieved significant success in retrieval-based question answering. However, current multimodal retrieval question-answering models face two main challenges. Firstly, utilizing compressed evidence features as input to the model results in the loss of fine-grained information within the evidence. Secondly, a gap exists between the feature extraction of evidence and the question, which hinders the model from effectively extracting critical features from the evidence based on the given question. We propose a two-stage framework for evidence retrieval and question-answering to alleviate these issues. First and foremost, we propose a progressive evidence refinement strategy for selecting crucial evidence. This strategy employs an iterative evidence retrieval approach to uncover the logical sequence among the evidence pieces. It incorporates two rounds of filtering to optimize the solution space, thus further ensuring temporal efficiency. Subsequently, we introduce a semi-supervised contrastive learning training strategy based on negative samples to expand the scope of the question domain, allowing for a more thorough exploration of latent knowledge within known samples. Finally, in order to mitigate the loss of fine-grained information, we devise a multi-turn retrieval and question-answering strategy to handle multimodal inputs. This strategy involves incorporating multimodal evidence directly into the model as part of the historical dialogue and question. Meanwhile, we leverage a cross-modal attention mechanism to capture the underlying connections between the evidence and the question, and the answer is generated through a decoding generation approach. We validate the model's effectiveness through extensive experiments, achieving outstanding performance on WebQA and MultimodelQA benchmark tests.
BibTeX
@misc{yang2023progressive,
  author = {Shuwen Yang and Anran Wu and Xingjiao Wu and Luwei Xiao and Tianlong Ma and Cheng Jin and Liang He},
  title = {{Progressive Evidence Refinement for Open-domain Multimodal Retrieval Question Answering}},
  year = {2023},
  eprint = {2310.09696},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.09696},
}
Figure