TL;DR
Generative Open-Domain Question Answering (ODQA) models achieve state-of-the-art accuracy by encoding dozens of passages, but training and fine-tuning them requires massive compute. The Adaptive Passage Encoder (APE) introduces a lightweight dynamic computation policy that freezes the underlying reader model and trains on a single GPU in under 15 hours. By allocating encoder layers dynamically to promising passages and dropping irrelevant ones early, APE consistently improves answer accuracy and retrieval quality under tight computational budgets.
What Problem Does This Paper Address?
Modern ODQA architectures typically follow a retriever-reader setup: a retriever pulls candidate passages from an external corpus (e.g., Wikipedia), and a reader generates the final answer. State-of-the-art generative readers like FiD (Fusion-in-Decoder) process large numbers of passages through deep Transformer encoders, making them computationally expensive at inference and inaccessible for researchers with limited hardware.
Prior Adaptive Computation (AC) approaches dynamically drop unpromising passages or early-exit at intermediate layers, but they require full fine-tuning of all reader parameters. Attempting this on large generative architectures causes out-of-memory (OOM) errors even on 16GB GPUs. This paper addresses that bottleneck: How can we apply adaptive computation to large generative ODQA models under strict hardware constraints without retraining the backbone?
Methods
The authors propose the Adaptive Passage Encoder (APE), an add-on module requiring less than 4% extra parameters while keeping the base generative model (e.g., FiD-base or FiD-large) completely frozen. APE overrides the standard layer-by-layer encoder computation using two core components:
HasAnswerModule: Given a passage's intermediate representation at layer $l$, this module max-pools the token representations and feeds them into a multi-layer perceptron (MLP) to output the probability $p_n^l$ that the passage contains the answer. It is trained first using standard cross-entropy.- Dynamic Scheduler: The scheduler assigns a dynamic priority score $q_n$ to each passage: $$q_n = \sigma(g(p_n^{l_n}, n, l_n)) p_n^{l_n} + f(p_n^{l_n}, n, l_n)$$ where $n$ is the initial retriever rank, $l_n$ is the current layer depth, and $g, f$ are small MLPs learning dynamic weights and biases.
Computation Flow and Training
- Inference Loop: Starting at layer 0, the scheduler selects the passage with the highest priority $q_n$, advances it by one encoder layer, updates its hidden representation, and recomputes its priority. This repeats until a layer budget $B = L \cdot k$ is exhausted (where $L$ is the number of encoder layers and $k$ is the target number of passages). Only the $k$ passages with the most computed layers are passed to the decoder.
- RL Training: The scheduler is trained with the REINFORCE policy gradient algorithm to maximize reward (giving $+ (1-c)$ for selecting an answer-bearing passage, and $-c$ penalty otherwise). Because the weights and biases are learned dynamically via RL, APE bypasses the complex confidence calibration required by earlier methods.
- Compute Footprint: APE trains in 8–15 hours on a single 16GB Nvidia V100 GPU.
Datasets
The authors evaluate APE on two widely used open-domain QA benchmarks:
- NaturalQuestions (NQ): 79,168 train, 8,757 validation, and 3,610 test questions.
- TriviaQA: 78,785 train, 8,837 validation, and 11,313 test questions.
No new datasets were introduced; both evaluations use standard dense retriever outputs from prior work.
Key Insights and Ideas
- Encoder Reranking without Retraining: Instead of running all passages through every encoder layer, the encoder itself acts as a dynamic re-ranker. Spurious passages exit after only a few layers, preserving compute for passages with high answer likelihood.
- No Backbone Gradient Updates Needed: Because the base Transformer representations are rich, frozen intermediate hidden states are informative enough for an auxiliary MLP to detect answerability early in the network.
- No Unrestricted Degradation: Previous adaptive computation techniques often suffered from performance drops or plateaus when allowed unrestricted computation. APE retains the full performance of the base model in unrestricted settings while boosting constrained settings.
Key Results
Experiments evaluate Exact Match (EM) accuracy across computational limits $k \in \{5, 10, 20\}$ retained passages:
- Outperforming Base Generative Models:
- On TriviaQA, APE+FiD-base achieves 55.4% ($k=5$), 59.0% ($k=10$), and 62.0% ($k=20$) EM—consistently and statistically outperforming vanilla FiD-base (53.9%, 57.9%, 60.7%).
- APE+FiD-large hits 61.4% at $k=10$ and 64.1% at $k=20$, both improving upon baseline FiD-large.
- Massive Gains Over Prior Adaptive Baselines: At $k=10$ on NaturalQuestions, APE+FiD-large outperforms the previous AC baseline (SkylineBuilder) by 12.4% EM (46.6% vs. 34.2%) due to the combination of adaptive scheduling and a frozen, generative backbone.
- Improved Passage Quality: Under identical budgets, APE significantly improves top-$k$ retrieval accuracy compared to DPR and FiD alone (e.g., reaching 75.4% Top-10 retrieval accuracy on NQ vs. FiD's 73.9%), explaining the downstream question-answering gains.
Limitations and Future Work
- Fixed Layer Budget: The computation budget $B = L \cdot k$ is predetermined per query; the framework does not yet dynamically decide to terminate early on exceptionally easy questions.
- Encoder-Only Adaptation: APE optimizes computation in the passage encoder. The generative decoder still generates output tokens autoregressively without adaptive computation.
- Granularity Overhead: Making sequential, single-layer routing decisions for individual passages introduces scheduling logic that may offset some wall-clock throughput benefits on specific parallel hardware setups.
Summary updated