TL;DR
This paper introduces the Adaptive Passage Encoder (APE), a novel method that makes state-of-the-art generative Open-Domain Question Answering (ODQA) models more computationally efficient without requiring extensive retraining. APE achieves this by learning a policy to dynamically adjust computation across encoder layers for different passages, leading to improved accuracy and efficiency compared to previous adaptive computation techniques.
What Problem Does This Paper Address?
Open-Domain Question Answering (ODQA) systems aim to find answers to questions within a large collection of documents. Modern ODQA models, particularly generative ones, have achieved impressive results by processing multiple passages. However, these models are computationally expensive to train and run, often requiring significant resources that are inaccessible to many researchers. Existing adaptive computation (AC) methods, while promising for efficiency, typically require fine-tuning all model parameters, making them impractical for these large, state-of-the-art models. This paper addresses the challenge of making these powerful generative ODQA models more efficient and accessible without sacrificing performance.
Methods
The core innovation is the Adaptive Passage Encoder (APE). APE is designed to be added to an existing generative ODQA model, specifically the encoder component. Instead of processing every passage through every encoder layer uniformly, APE introduces an adaptive computation policy. This policy is learned by two new components:
- HasAnswer Model: This model predicts the probability that a given passage contains an answer to the question, based on its encoded representation.
- Scheduler: This component uses the HasAnswer model's predictions and passage information to learn a scheduling policy. It dynamically allocates encoder layers to passages, allowing "spurious" passages to exit computation early while dedicating more layers to promising ones.
Crucially, APE keeps the parameters of the base ODQA model fixed. This means it can be trained efficiently, even on a single GPU, and doesn't require the computationally intensive fine-tuning of the entire large model. The scheduler is trained using the REINFORCE algorithm to maximize a reward signal that encourages the selection of relevant passages.
Datasets
The paper evaluates its method on two standard ODQA datasets:
- NaturalQuestions: A large-scale, distantly supervised dataset for question answering research.
- TriviaQA: Another widely used dataset for reading comprehension and question answering.
The paper uses the retrieval results from a dense retriever (Izacard and Grave, 2020b) as input to the ODQA system.
Key Insights and Ideas
- Efficient Adaptation: The most significant insight is that adaptive computation can be applied to large generative ODQA models without fine-tuning the base model's parameters. This dramatically lowers the barrier to entry for using AC techniques.
- Passage Quality Matters: APE not only optimizes computation but also demonstrably improves the quality of the selected top-k passages. This enhanced passage selection directly contributes to better overall ODQA performance.
- Outperforming Previous AC: APE achieves better accuracy than a previous adaptive computation method (Wu et al., 2020) on ODQA tasks, attributed to the stronger base ODQA model it's applied to.
- Robustness: Unlike some prior AC methods that degraded performance in unrestricted settings, APE maintains or improves performance even when not constrained by a strict computational budget.
Key Results
- APE improves the Exact Match (EM) score of the state-of-the-art FiD model on both NaturalQuestions and TriviaQA, especially under restricted top-k settings.
- On TriviaQA with k=10 passages, APE achieves a statistically significant improvement of 12.4% EM over the previous AC method.
- APE allows FiD to significantly outperform RAG on NaturalQuestions for k={10, 20}.
- The method is computationally feasible, allowing training on a single GPU, which was not possible with prior AC methods for large models.
- Table 3 shows that APE significantly improves the top-k retrieval accuracy of selected passages compared to strong baselines like BM25 and DPR.
Limitations and Future Work
- Computational Budget: While APE improves efficiency, the paper focuses on optimizing computation within a given budget (number of passages and layers). Further work could explore more dynamic budget allocation.
- Base Model Dependency: The performance gains are dependent on the quality of the underlying ODQA model. Applying APE to weaker base models might yield different results.
- Future Work: The authors suggest that their approach can be extended to other sequence-to-sequence models and potentially other NLP tasks where adaptive computation could be beneficial. They also highlight the potential for further improvements by exploring different scheduling policies or reward functions.
Summary updated