Optimizing Retrieval-augmented Reader Models via Token Elimination

Moshe Berchansky, Peter Izsak, Avi Caciularu, Ido Dagan, Moshe Wasserblat

TL;DR

This paper tackles the computational bottleneck in retrieval-augmented language models, specifically Fusion-in-Decoder (FiD), which struggles with long-form question answering due to extensive decoding time. They introduce "Token Filtering," a method that intelligently prunes less relevant tokens from retrieved passages during generation, significantly speeding up inference (up to 62.2%) with minimal performance loss (under 2%).

What Problem Does This Paper Address?

Retrieval-augmented language models, like Fusion-in-Decoder (FiD), are powerful tools for tasks requiring external knowledge, such as open-domain question answering (ODQA). FiD works by first retrieving relevant passages and then using a generative "reader" model to produce an answer. However, a major challenge arises with long-form answers, where the sheer volume of retrieved text and the iterative decoding process create a significant computational bottleneck, leading to slow inference times. This paper aims to optimize FiD by reducing this decoding latency without sacrificing answer quality.

Methods

The core innovation is Token Filtering, a method designed to reduce the computational load during the decoding phase of FiD. The authors analyze the cross-attention scores within the decoder, observing that certain input tokens consistently receive higher attention, indicating their importance for answer generation. Token Filtering leverages this by dynamically identifying and discarding tokens with low cross-attention scores at each decoding step. This effectively prunes the input sequence, reducing the amount of information the decoder needs to process.

To further enhance efficiency, the authors also incorporate CALM (Confidently Adaptive Language Modeling), a technique that allows the decoder to "exit early" from certain layers if it's confident enough in its prediction. The paper proposes a Combined approach that integrates both Token Filtering and CALM.

Datasets

The experiments were conducted on three commonly used datasets for Long-Form Question Answering (LFQA):

  • ELI5 (Explain Like I'm Five): A dataset derived from a Reddit forum.
  • MS MARCO: A collection of crowd-sourced responses to Bing queries, focusing on passage ranking.
  • NaturalQuestions (NQ): A dataset from Google designed for natural language understanding, pairing user queries with Wikipedia passages.

The authors used standard train/validation/test splits, with ELI5 also evaluated on the KILT benchmark.

Key Insights and Ideas

  • Decoder is the Bottleneck: For long-form question answering, the decoder module, not the encoder, is the primary source of latency and computational load. This is a crucial insight that guides the focus of optimization efforts.
  • Cross-Attention as a Salience Indicator: Cross-attention scores within the decoder effectively reveal which input tokens are most important for generating the answer. This provides a principled way to filter out less relevant information.
  • Early Stages Matter Most: The analysis shows that the decoder's initial layers are most indicative of which input tokens are relevant. This suggests that filtering early in the generation process is most beneficial.
  • Token Filtering Preserves Performance: Surprisingly, removing a significant portion of input tokens (up to 62.2% reduction in runtime) can be achieved with minimal impact on performance, and in some cases, even leads to slight improvements. This suggests that the model can often generate good answers even with less input information.

Key Results

The proposed Token Filtering method, especially when combined with CALM, demonstrates significant efficiency gains:

  • Runtime Reduction: The combined approach achieves substantial reductions in decoding time, with savings up to 62.2% on MS MARCO (Base model), 54.9% on NQ, and 40.9% on ELI5.
  • Performance Preservation: The performance drop is consistently kept below 2% across datasets and model sizes.
  • State-of-the-Art: The combined approach achieves state-of-the-art performance on the ELI5 KILT benchmark.
  • Qualitative Improvements: In some cases, the method even improved performance, likely by allowing the model to better focus on salient information.

Limitations and Future Work

  • Retriever Scope: The study primarily focused on optimizing the reader model and did not extensively experiment with a wide array of retrieval systems.
  • Hyperparameter Search Space: The hyperparameter search was conducted over a subspace of possible values due to computational constraints.
  • Future Directions: The authors suggest developing more dynamic methods for selecting relevant tokens, moving beyond predetermined hyperparameters. They also propose training the cross-attention patterns to better attend to salient information during generation.

Summary updated

Abstract

Fusion-in-Decoder (FiD) is an effective retrieval-augmented language model applied across a variety of open-domain tasks, such as question answering, fact checking, etc. In FiD, supporting passages are first retrieved and then processed using a generative model (Reader), which can cause a significant bottleneck in decoding time, particularly with long outputs. In this work, we analyze the contribution and necessity of all the retrieved passages to the performance of reader models, and propose eliminating some of the retrieved information, at the token level, that might not contribute essential information to the answer generation process. We demonstrate that our method can reduce run-time by up to 62.2%, with only a 2% reduction in performance, and in some cases, even improve the performance results.
BibTeX
@inproceedings{berchansky2023optimizing,
  author = {Moshe Berchansky and Peter Izsak and Avi Caciularu and Ido Dagan and Moshe Wasserblat},
  title = {{Optimizing Retrieval-augmented Reader Models via Token Elimination}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=HMI8__Ovae},
}
Figure