TL;DR
This paper introduces Multi-Granularity Guided Fusion-in-Decoder (MG-FiD), a novel approach to improve open-domain question answering (ODQA) by better identifying and utilizing relevant evidence. MG-FiD tackles the problem of "spurious passages" that can mislead ODQA models by incorporating both passage-level and sentence-level evidence discernment, leading to significant accuracy improvements.
What Problem Does This Paper Address?
Open-domain question answering (ODQA) systems aim to find answers to questions within a vast collection of documents. A common architecture for this is retrieval-augmented generation (RAG), where a retriever finds relevant passages, and a reader then generates an answer based on these passages. A popular reader architecture is "Fusion-in-Decoder" (FiD), which concatenates multiple retrieved passages and feeds them into a decoder. However, FiD struggles with "spurious passages" – documents that might contain keywords from the question but don't actually provide the correct evidence, leading to incorrect answers. This paper addresses the challenge of making ODQA models more robust to these misleading passages by developing a more nuanced way to identify and leverage evidence at different levels of detail.
Methods
The core of MG-FiD is its multi-task learning approach, which trains the model to perform both answer generation and evidence discernment simultaneously.
- Multi-Granularity Contexts: MG-FiD operates at two levels:
- Passage Re-ranking: It learns to score passages based on their evidentiality. This is done using a listwise loss function that considers the relative importance of passages.
- Sentence Classification: For finer-grained evidence, it classifies individual sentences within passages as supportive or not. This helps to pinpoint the exact information needed.
- Anchor Vector: To guide the decoder, MG-FiD creates an "anchor vector." This vector is derived from the embeddings of positively predicted sentences and is added to the decoder's initial "[BOS]" token embedding. This injects fine-grained evidence signals directly into the generation process.
- Passage Pruning: To improve efficiency, MG-FiD uses the passage re-ranking scores to prune less relevant passages before they are fed to the decoder. This reduces the computational burden without significantly sacrificing accuracy.
- LLM-based Pseudo-labeling: To overcome the challenge of obtaining high-quality labels for passage re-ranking, the authors leverage Large Language Models (LLMs) like ChatGPT and MythoMax to generate pseudo-labels for evidentiality.
Datasets
The paper evaluates MG-FiD on two benchmark datasets:
- Natural Questions (NQ): This dataset consists of actual Google search queries.
- TriviaQA (TQA): This dataset is sourced from trivia and quiz-league websites.
The authors also mention using LLMs for generating pseudo-labels, which is a novel aspect of their evidence labeling strategy.
Key Insights and Ideas
- Beyond Answer Spans: Relying solely on the presence of an answer span within a passage is insufficient for determining evidence. A passage can contain the answer span but still be misleading or irrelevant to the question's context.
- Sentence-Level Nuance is Crucial: While passage-level evidence is important, fine-grained sentence-level analysis is critical for distinguishing truly supportive information from distracting details.
- LLMs as Labelers: Using LLMs to generate pseudo-labels for passage evidentiality is an effective and scalable way to overcome the scarcity of human-annotated data in ODQA.
- Anchor Vector for Direct Guidance: Injecting a summarized signal of supportive sentences (the anchor vector) directly into the decoder's input provides a more focused and effective way to guide answer generation compared to just relying on the decoder's internal attention mechanisms.
- Pruning for Efficiency: Dynamically pruning passages based on their evidentiality scores offers a significant efficiency boost without a substantial drop in performance, making ODQA systems more practical.
Key Results
MG-FiD demonstrates strong performance on both NQ and TQA datasets:
- Accuracy Improvement: MG-FiD achieves significant improvements in Exact Match (EM) scores: 3.5% on NQ and 1.0% on TQA compared to the baseline FiD-KD model.
- Competitive Performance: It outperforms other baselines like EvidentialityQA and RFiD, especially when integrating fine-grained sentence-level evidence.
- Efficiency Gains: Passage pruning effectively reduces the number of passages used by the decoder by 76% on NQ and 61.5% on TQA, while maintaining high performance. For example, with a pruning threshold of 0.05, the number of passages drops to an average of 4.8 on NQ.
- Evidence Labeling Effectiveness: LLM-generated labels (MythoMax and ChatGPT) lead to better performance than using simple answer span presence for passage re-ranking.
Limitations and Future Work
- LLM Filtering Limitations: The LLM-based label filtering is currently limited to extractive QA settings.
- Need for More Passage Validation: Further validation on more passages might be beneficial.
- Marginal TQA Improvement: The improvement on the TQA dataset is noted as marginal, possibly due to its numerous answer candidates and the inherent ease of finding evidence.
- Computational Cost: Reporting results with a large number of passages (e.g., 100) and larger backbone models (e.g., T5-large) was limited by computational resources.
- Seed Variability: The standard deviation on the NQ dataset is large, suggesting the need for validation with additional seeds.
- Future Work: The authors suggest exploring more detailed analysis of why multi-task learning is most effective with fewer documents (K=10) and investigating the impact of larger models and more passages.
Summary updated