Abstract
Open Domain Question Answering (ODQA) has been advancing rapidly in recent times, driven by significant developments in dense passage retrieval and pretrained language models. State-of-the-art models typically incorporate the FiD framework, which is composed by a neural retriever alongside an encoder-decoder neural reader. In the answer generation process, the retriever will retrieve numerous passages (around 100 for instance), each of which is then individually encoded by the encoder. Subsequently, the decoder makes predictions based on these encoded passages. Nevertheless, this framework can be relatively time-consuming, particularly due to the extensive length of the gathered passages. To address this, we introduce FastFiD in this paper, a novel approach that executes sentence selection on the encoded passages. This aids in retaining valuable sentences while reducing the context length required for generating answers. Experiments on three commonly used datasets (Natural Questions, TriviaQA and ASQA) demonstrate that our method can enhance the inference speed by 2.3X-5.7X, while simultaneously maintaining the model’s performance. Moreover, an in-depth analysis of the model’s attention reveals that the selected sentences indeed hold a substantial contribution towards the final answer. The codes are publicly available at https://github.com/thunlp/FastFiD.
Introduction
In Open Domain Question Answering (ODQA), systems typically follow a "Retriever-Reader" pipeline. The current gold standard for the "Reader" is the Fusion-in-Decoder (FiD) framework. FiD excels because it can process a large number of retrieved passages (often 100 or more) to find an answer. However, FiD has a major efficiency bottleneck: while it encodes passages independently, the decoder must perform cross-attention over the concatenated embeddings of all retrieved passages. If 100 passages are retrieved, the decoder processes a massive context of 10,000 to 40,000 tokens. Because the computational cost of attention grows with sequence length, this makes inference slow and expensive, especially as we move toward larger language models.
Methodology
FastFiD introduces a "Sentence Selection" step between the encoder and the decoder. Instead of handing the decoder the entire retrieved context, FastFiD identifies and keeps only the most relevant sentences.
Architecture and Training
The framework incorporates a Sentence Selection Head, a classification layer added to the encoder that predicts the start and end positions of valuable sentences across all retrieved passages. The selection mechanism uses "Global multi-passage-level" probability, ranking sentences against every other sentence in the entire retrieved set.
The model is trained in two distinct stages:
- Multi-task Training: The model is trained to simultaneously generate answers and identify sentences that contain the answer using a combination of language modeling loss and selection loss.
- Select Generation Training: This stage fine-tunes the model to generate answers using only the selected sentence embeddings. This bridges the gap between training on full context and inference on sparse, selected context, ensuring the decoder remains accurate with shorter, denser inputs.
Experimental Results
FastFiD was evaluated on three prominent ODQA benchmarks: Natural Questions (NQ), TriviaQA, and ASQA (a long-form QA benchmark).
Efficiency and Accuracy
The results demonstrate that FastFiD achieves massive speedups while maintaining or even slightly improving accuracy. On the NQ dataset with 100 retrieved passages, the context length was reduced from 38,400 tokens to just 1,008 tokens—a nearly 38x reduction. This translated to a 3.5x speedup on NQ and a staggering 5.7x speedup on the complex ASQA dataset.
Analysis
The authors found that sentence selection is superior to passage reranking because relevant passages often contain "filler" text. By extracting specific sentences, FastFiD increases the density of useful information. Analysis of cross-attention maps revealed that standard FiD decoders naturally ignore most of the provided context; FastFiD essentially "pre-computes" this filtering, saving significant computational resources.
Conclusion and Future Work
FastFiD provides a robust framework for accelerating ODQA without sacrificing performance. It is compatible with modern architectures like Llama2-7B, where it achieved a 1.9x speedup. Future work includes developing dynamic selection mechanisms to determine the optimal number of sentences per query and extending the approach to broader Retrieval-Augmented Generation (RAG) tasks.
Summary updated