Training Adaptive Computation for Open-Domain Question Answering with Computational Constraints

Yuxiang Wu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel

TL;DR

Generative Open-Domain Question Answering (ODQA) models achieve state-of-the-art accuracy by encoding dozens of passages, but training and fine-tuning them requires massive compute. The Adaptive Passage Encoder (APE) introduces a lightweight dynamic computation policy that freezes the underlying reader model and trains on a single GPU in under 15 hours. By allocating encoder layers dynamically to promising passages and dropping irrelevant ones early, APE consistently improves answer accuracy and retrieval quality under tight computational budgets.

What Problem Does This Paper Address?

Modern ODQA architectures typically follow a retriever-reader setup: a retriever pulls candidate passages from an external corpus (e.g., Wikipedia), and a reader generates the final answer. State-of-the-art generative readers like FiD (Fusion-in-Decoder) process large numbers of passages through deep Transformer encoders, making them computationally expensive at inference and inaccessible for researchers with limited hardware.

Prior Adaptive Computation (AC) approaches dynamically drop unpromising passages or early-exit at intermediate layers, but they require full fine-tuning of all reader parameters. Attempting this on large generative architectures causes out-of-memory (OOM) errors even on 16GB GPUs. This paper addresses that bottleneck: How can we apply adaptive computation to large generative ODQA models under strict hardware constraints without retraining the backbone?

Methods

The authors propose the Adaptive Passage Encoder (APE), an add-on module requiring less than 4% extra parameters while keeping the base generative model (e.g., FiD-base or FiD-large) completely frozen. APE overrides the standard layer-by-layer encoder computation using two core components:

  1. HasAnswer Module: Given a passage's intermediate representation at layer $l$, this module max-pools the token representations and feeds them into a multi-layer perceptron (MLP) to output the probability $p_n^l$ that the passage contains the answer. It is trained first using standard cross-entropy.
  2. Dynamic Scheduler: The scheduler assigns a dynamic priority score $q_n$ to each passage: $$q_n = \sigma(g(p_n^{l_n}, n, l_n)) p_n^{l_n} + f(p_n^{l_n}, n, l_n)$$ where $n$ is the initial retriever rank, $l_n$ is the current layer depth, and $g, f$ are small MLPs learning dynamic weights and biases.

Computation Flow and Training

  • Inference Loop: Starting at layer 0, the scheduler selects the passage with the highest priority $q_n$, advances it by one encoder layer, updates its hidden representation, and recomputes its priority. This repeats until a layer budget $B = L \cdot k$ is exhausted (where $L$ is the number of encoder layers and $k$ is the target number of passages). Only the $k$ passages with the most computed layers are passed to the decoder.
  • RL Training: The scheduler is trained with the REINFORCE policy gradient algorithm to maximize reward (giving $+ (1-c)$ for selecting an answer-bearing passage, and $-c$ penalty otherwise). Because the weights and biases are learned dynamically via RL, APE bypasses the complex confidence calibration required by earlier methods.
  • Compute Footprint: APE trains in 8–15 hours on a single 16GB Nvidia V100 GPU.

Datasets

The authors evaluate APE on two widely used open-domain QA benchmarks:

  • NaturalQuestions (NQ): 79,168 train, 8,757 validation, and 3,610 test questions.
  • TriviaQA: 78,785 train, 8,837 validation, and 11,313 test questions.

No new datasets were introduced; both evaluations use standard dense retriever outputs from prior work.

Key Insights and Ideas

  • Encoder Reranking without Retraining: Instead of running all passages through every encoder layer, the encoder itself acts as a dynamic re-ranker. Spurious passages exit after only a few layers, preserving compute for passages with high answer likelihood.
  • No Backbone Gradient Updates Needed: Because the base Transformer representations are rich, frozen intermediate hidden states are informative enough for an auxiliary MLP to detect answerability early in the network.
  • No Unrestricted Degradation: Previous adaptive computation techniques often suffered from performance drops or plateaus when allowed unrestricted computation. APE retains the full performance of the base model in unrestricted settings while boosting constrained settings.

Key Results

Experiments evaluate Exact Match (EM) accuracy across computational limits $k \in \{5, 10, 20\}$ retained passages:

  • Outperforming Base Generative Models:
  • On TriviaQA, APE+FiD-base achieves 55.4% ($k=5$), 59.0% ($k=10$), and 62.0% ($k=20$) EM—consistently and statistically outperforming vanilla FiD-base (53.9%, 57.9%, 60.7%).
  • APE+FiD-large hits 61.4% at $k=10$ and 64.1% at $k=20$, both improving upon baseline FiD-large.
  • Massive Gains Over Prior Adaptive Baselines: At $k=10$ on NaturalQuestions, APE+FiD-large outperforms the previous AC baseline (SkylineBuilder) by 12.4% EM (46.6% vs. 34.2%) due to the combination of adaptive scheduling and a frozen, generative backbone.
  • Improved Passage Quality: Under identical budgets, APE significantly improves top-$k$ retrieval accuracy compared to DPR and FiD alone (e.g., reaching 75.4% Top-10 retrieval accuracy on NQ vs. FiD's 73.9%), explaining the downstream question-answering gains.

Limitations and Future Work

  • Fixed Layer Budget: The computation budget $B = L \cdot k$ is predetermined per query; the framework does not yet dynamically decide to terminate early on exceptionally easy questions.
  • Encoder-Only Adaptation: APE optimizes computation in the passage encoder. The generative decoder still generates output tokens autoregressively without adaptive computation.
  • Granularity Overhead: Making sequential, single-layer routing decisions for individual passages introduces scheduling logic that may offset some wall-clock throughput benefits on specific parallel hardware setups.

Summary updated

Abstract

Adaptive Computation (AC) has been shown to be effective in improving the efficiency of Open-Domain Question Answering (ODQA) systems. However, the current AC approaches require tuning of all model parameters, and training state-of-the-art ODQA models requires significant computational resources that may not be available for most researchers. We propose Adaptive Passage Encoder, an AC method that can be applied to an existing ODQA model and can be trained efficiently on a single GPU. It keeps the parameters of the base ODQA model fixed, but it overrides the default layer-by-layer computation of the encoder with an AC policy that is trained to optimise the computational efficiency of the model. Our experimental results show that our method improves upon a state-of-the-art model on two datasets, and is also more accurate than previous AC methods due to the stronger base ODQA model. All source code and datasets are available at https://github.com/uclnlp/APE.
BibTeX
@inproceedings{wu2021training,
  author = {Yuxiang Wu and Pasquale Minervini and Pontus Stenetorp and Sebastian Riedel},
  title = {{Training Adaptive Computation for Open-Domain Question Answering with Computational Constraints}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=64TS6DMhsd},
}

Loading related content...

Figure