Optimizing Retrieval-augmented Reader Models via Token Elimination

Moshe Berchansky, Peter Izsak, Avi Caciularu, Ido Dagan, Moshe Wasserblat

Abstract: Retrieval-augmented generation models like Fusion-in-Decoder (FiD) face significant latency bottlenecks due to the decoder's cross-attention over numerous retrieved passages. This paper introduces "Token Filtering," a dynamic pruning method that identifies and removes uninformative input tokens during generation. When combined with early-exit techniques, this approach achieves up to a 62% reduction in inference latency with minimal performance loss, setting a new state-of-the-art on the ELI5 KILT leaderboard.

The Decoder Bottleneck in Retrieval-Augmented Models

Open-Domain Question Answering (ODQA) systems typically follow a "retriever-reader" pipeline. The retriever pulls relevant passages from a knowledge base, and the reader (usually a generative model like T5) processes them to produce an answer.

The Fusion-in-Decoder (FiD) architecture is the gold standard for this, but it has a massive bottleneck: the decoder. Because FiD attends to all retrieved passages simultaneously, the cross-attention operation becomes incredibly heavy. For Long-Form Question Answering (LFQA), where answers can be hundreds of tokens long, the authors show that the decoder can account for over 90% of the total runtime. Most of the text being "read" by the decoder is actually irrelevant noise that doesn't contribute to the final answer.

Analysis of decoder runtime as a percentage of total inference time across varying answer lengths and passage counts.

Methods: Dynamic Token Filtering and CALM

The authors propose a two-pronged approach to efficiency that reduces both the "width" and "depth" of the computation:

  1. Token Filtering: The authors observed that the model's cross-attention scores in early decoder layers are highly predictive of which tokens are actually useful. They implement a "pruning" step: at a specific point during generation (e.g., after the 20th token), the model looks at the average cross-attention scores across all heads at a specific layer. It then keeps only the top $k\%$ of input tokens and discards the rest for all subsequent generation steps.
  2. CALM Integration: They combine Token Filtering with Confident Adaptive Language Modeling (CALM), an existing "early exit" strategy. CALM allows the model to skip higher decoder layers if it is already confident in its prediction for the current token.
Architecture of the Token Filtering mechanism integrated between decoder layers.

Key Results and Insights

The researchers evaluated their approach on three major benchmarks: ELI5, MS MARCO, and NaturalQuestions (NQ).

  • Massive Speedups: The combined approach reduced generation time by 62.2% on MS MARCO, 54.9% on NQ, and 40.9% on ELI5.
  • Minimal Accuracy Trade-off: These speedups were achieved with only a ~2% reduction in ROUGE-L scores. In some cases, removing the "noise" of irrelevant passages actually improved performance.
  • The "Gold" Signal: One of the most striking findings is that tokens from the "gold" (ground-truth) passage naturally rise to the top of the attention rankings very early in the generation process. This confirms that the model "knows" what's important long before it finishes the sentence.

Conclusion and Future Work

At the time of publication, this "Combined" model reached the #1 position on the ELI5 KILT leaderboard. While currently using static hyperparameters for filtering, the authors suggest that a more dynamic, per-query approach could be even more effective. This research proves that for long-form generation, optimizing the decoder's cross-attention is the primary path toward efficient inference in retrieval-augmented models.

Summary updated

Abstract

Fusion-in-Decoder (FiD) is an effective retrieval-augmented language model applied across a variety of open-domain tasks, such as question answering, fact checking, etc. In FiD, supporting passages are first retrieved and then processed using a generative model (Reader), which can cause a significant bottleneck in decoding time, particularly with long outputs. In this work, we analyze the contribution and necessity of all the retrieved passages to the performance of reader models, and propose eliminating some of the retrieved information, at the token level, that might not contribute essential information to the answer generation process. We demonstrate that our method can reduce run-time by up to 62.2%, with only a 2% reduction in performance, and in some cases, even improve the performance results.
BibTeX
@inproceedings{berchansky2023optimizing,
  author = {Moshe Berchansky and Peter Izsak and Avi Caciularu and Ido Dagan and Moshe Wasserblat},
  title = {{Optimizing Retrieval-augmented Reader Models via Token Elimination}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=HMI8__Ovae},
}
Figure