TL;DR
This paper introduces PERQA, a novel two-stage framework for open-domain multimodal retrieval question answering. PERQA tackles the challenge of information loss in evidence processing and the gap between question and evidence feature extraction by employing a progressive evidence refinement strategy for retrieval and a multi-turn dialogue approach for answering, achieving state-of-the-art results on benchmark datasets.
What Problem Does This Paper Address?
The paper addresses a critical challenge in multimodal retrieval question answering (MMRQA): how to effectively extract and utilize information from diverse sources (text, images, tables) to answer complex questions. Existing models often struggle with two main issues:
- Information Loss: Compressing multimodal evidence into fixed-size features can lead to the loss of crucial fine-grained details.
- Feature Mismatch: A disconnect between how evidence and questions are processed can hinder the model's ability to identify the most relevant information for answering.
These limitations make it difficult for models to accurately answer questions that require reasoning over multiple pieces of evidence, especially when some evidence might be subtly related or contain distractors.
Methods
PERQA employs a two-stage approach:
- Progressive Evidence Refinement (for Retrieval): This stage aims to precisely identify the most relevant evidence.
- Evidence Initial Screening Module (EISM): A first pass filters out irrelevant sources by encoding questions and evidence (converting images to text for initial processing) and calculating cosine similarity.
- Iterative Evidence Retrieval Strategy (IER): This module refines the initial selection by iteratively considering candidate sources. It uses a regression module built on BERT to compute matching scores, progressively adding the most relevant sources to a set of "key evidence." This iterative process helps uncover logical relationships between pieces of evidence.
- Negative Sample Semi-Supervised Contrastive Learning (NSCL): To improve the robustness of the initial screening, this strategy uses distractor evidence as "negative samples" during training. This helps the model learn to distinguish truly relevant evidence from irrelevant but similar information.
- Multi-Cue Question Answering: This stage generates the answer using the refined evidence.
- Multi-Turn Dialogue Strategy: PERQA treats multimodal evidence as a historical dialogue, incorporating it alongside the question. This allows the model to leverage context from multiple pieces of evidence.
- Cross-Modal Attention: A cross-attention mechanism is used to capture the intricate relationships between different modalities of evidence and the question.
- Model Architecture: The answering model combines ViT (for image encoding), LLaMA (for text/question encoding), and LORA (for efficient fine-tuning).
Datasets
The paper evaluates PERQA on two widely used multimodal question-answering datasets:
- WebQA: Contains multi-hop, multimodal question-answer pairs, often requiring 1-2 images or text snippets. Answers are free-form sentences.
- MultimodalQA: Features multimodal questions from various sources (images, text, tables) with 16 question types, 13 of which require cross-modal retrieval. Answers are typically phrases.
Key Insights and Ideas
- Progressive Refinement is Key: Simply retrieving a large set of evidence and then trying to filter it is less effective than a staged, iterative refinement process that builds upon previous selections. This mimics how humans might search for information.
- Leveraging Distractors: Treating irrelevant "distractor" evidence as negative examples during training is crucial for building robust models that can truly distinguish signal from noise in complex multimodal datasets.
- Multi-Turn Dialogue for Context: Framing multimodal evidence as a "conversation history" allows the model to better understand the relationships and dependencies between different pieces of information, leading to more accurate answers.
- Image-to-Text for Initial Screening: Converting images to text for the initial evidence screening stage is an efficient way to bridge the modality gap without requiring complex, separate visual encoders at that early stage.
Key Results
PERQA achieves state-of-the-art (SOTA) performance on both the WebQA and MultimodalQA datasets for both retrieval and question-answering tasks.
- On WebQA, PERQA surpasses previous SOTA by significant margins, including a 1.4% lead in retrieval F1 and a 6.7% lead in the crucial QA metric. It also shows strong performance in QA fluency and keyword accuracy.
- On MultimodalQA, PERQA demonstrates substantial improvements over SOTA in both multi-modal (EM: 2.2%, F1: 2.9%) and single-modal inference (EM: 3.6%, F1: 4.4%).
- A case study highlights PERQA's ability to correctly retrieve crucial intermediate evidence (E1) and then the final key evidence (E2) by leveraging logical order relationships, a task where a strong baseline (SKURG) failed.
Limitations and Future Work
- Image-to-Text Conversion: While effective for initial screening, the paper notes that in the question-answering stage, they directly use the initial image information. This suggests that the image-to-text conversion might still involve some information loss compared to direct visual processing.
- TableQ Performance: The authors mention that their model's performance on table-based questions might not be as strong as on image or text questions, potentially due to less specific pre-training on table data.
- Future Work: The paper's success in multi-turn dialogue and progressive refinement suggests future work could explore more sophisticated dialogue management and even more granular evidence refinement strategies. The authors also look forward to providing valuable insights for future research in this domain.
Summary updated