FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection

Yufei Huang, Xu Han, Maosong Sun

Abstract

Open Domain Question Answering (ODQA) has been advancing rapidly in recent times, driven by significant developments in dense passage retrieval and pretrained language models. Current models typically incorporate the FiD framework, which is composed by a neural retriever alongside an encoder-decoder neural reader. In the answer generation process, the retriever will retrieve numerous passages (around 100 for instance), each of which is then individually encoded by the encoder. Subsequently, the decoder makes predictions based on these encoded passages. Nevertheless, this framework can be relatively time-consuming, particularly due to the extensive length of the gathered passages. To address this, we introduce FastFiD in this paper, a novel approach that executes sentence selection on the encoded passages. This aids in retaining valuable sentences while reducing the context length required for generating answers. Experiments on three commonly used datasets (Natural Questions, TriviaQA and ASQA) demonstrate that our method can enhance the inference speed by 2.3X-5.7X, while simultaneously maintaining the model's performance. Moreover, an in-depth analysis of the model's attention reveals that the selected sentences indeed hold a substantial contribution towards the final answer.

Introduction: The Efficiency Bottleneck in ODQA

Open Domain Question Answering (ODQA) systems typically follow a "Retriever-Reader" architecture: a retriever finds relevant documents from a massive corpus (like Wikipedia), and a reader generates the final answer. The current state-of-the-art reader is the Fusion-in-Decoder (FiD) model.

While FiD is highly accurate because it can process many passages (often 100+) simultaneously, it has a major efficiency bottleneck. The decoder must perform cross-attention over the concatenated text of all these passages. As the number of passages grows, the context length explodes, leading to high latency and computational costs—making it difficult to deploy in real-time applications.

Methods: Two-Stage Sentence Selection

To address this, the authors introduce FastFiD, which adds a sentence-level selection mechanism to the standard FiD pipeline. The core innovation is a two-stage training procedure:

The FastFiD training and inference framework, highlighting the two-stage training process.
  1. 1st-Stage: Multi-Task Training: The model is trained to perform two tasks simultaneously: generating the correct answer and predicting the start/end positions of "key sentences" (sentences containing the answer) within the retrieved passages. This teaches the encoder how to identify "signal" vs. "noise."
  2. 2nd-Stage: Select-Generation Training: Simply selecting sentences at inference time can cause a performance drop because the decoder is not used to such short, dense contexts. In this stage, the model is fine-tuned to generate answers solely based on the sentences selected by the encoder in the first stage.

At inference time, the encoder processes all passages, the selection head picks the top $K$ sentences based on global probability, and only those sentences are passed to the decoder.

Experimental Results

The researchers evaluated FastFiD on three widely-used benchmarks: Natural Questions (NQ), TriviaQA, and ASQA (focused on long-form question answering). FastFiD achieved significant improvements in inference speed while maintaining competitive performance.

Main results comparing FiD, FiD-Light, HybridFiD, and FastFiD across NQ, TriviaQA, and ASQA datasets.
  • Speedup: FastFiD achieved a 3.5x speedup on Natural Questions and a massive 5.7x speedup on the long-form ASQA dataset compared to the standard FiD baseline.
  • Efficiency: On the NQ dataset with 100 retrieved passages, FastFiD reduced the context length from 38,400 tokens to just 1,008 tokens (a 38x reduction) while actually increasing the Exact Match (EM) score slightly (50.06 to 50.17).
  • Superiority to Reranking: FastFiD consistently outperformed "RerankFiD" (a baseline that selects top passages instead of sentences), proving that fine-grained sentence selection is a more effective compression strategy.

Key Insights and Analysis

  • Sentences over Passages: A 100-word passage often contains only one relevant sentence. By selecting sentences rather than reranking entire passages, the model achieves a much higher "information density," allowing the decoder to work faster without losing context.
  • Cross-Attention Validation: To prove their selection method worked, the authors analyzed the decoder's attention maps. They found that the sentences FastFiD selected naturally had significantly higher cross-attention scores in the original FiD model, confirming that the model was successfully identifying the most "useful" parts of the text.
  • Bridging the Training-Inference Gap: A key finding was that a model trained on long contexts fails when suddenly given short ones. The second stage of training is crucial for "re-aligning" the decoder to the new, compressed input format.
  • Scalability to LLMs: The authors demonstrated that this technique isn't just for T5-style models; it also works for decoder-only Large Language Models (like Llama2-7B), providing a nearly 2x speedup.

Conclusion

FastFiD demonstrates that sentence-level selection is a powerful tool for improving the efficiency of ODQA systems. By drastically reducing the input length for the decoder, it enables faster inference without sacrificing accuracy, making high-performance ODQA more viable for real-world deployment.

Summary updated

Abstract

Open Domain Question Answering (ODQA) has been advancing rapidly in recent times, driven by significant developments in dense passage retrieval and pretrained language models. Current models typically incorporate the FiD framework, which is composed by a neural retriever alongside an encoder-decoder neural reader. In the answer generation process, the retriever will retrieve numerous passages (around 100 for instance), each of which is then individually encoded by the encoder. Subsequently, the decoder makes predictions based on these encoded passages. Nevertheless, this framework can be relatively time-consuming, particularly due to the extensive length of the gathered passages. To address this, we introduce FastFiD in this paper, a novel approach that executes sentence selection on the encoded passages. This aids in retaining valuable sentences while reducing the context length required for generating answers. Experiments on three commonly used datasets (Natural Questions, TriviaQA and ASQA) demonstrate that our method can enhance the inference speed by 2.3X-5.7X, while simultaneously maintaining the model's performance. Moreover, an in-depth analysis of the model's attention reveals that the selected sentences indeed hold a substantial contribution towards the final answer. The codes are publicly available at https://github.com/thunlp/FastFiD.
BibTeX
@inproceedings{huang2024fastfid,
  author = {Yufei Huang and Xu Han and Maosong Sun},
  title = {{FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=z8RScUgvqH},
}
Figure