Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion

Shangyu Wu, Ying Xiong, Yufei Cui, Xue Liu, Buzhou Tang, Tei-Wei Kuo, Chun Jason XuearXiv 2024 · DOI · arXiv

Introduction: The "Concatenation Tax"

Retrieval-Augmented (RA) models have significantly advanced knowledge-intensive tasks like open-domain question answering. However, applying these techniques to non-knowledge-intensive (NKI) tasks—such as sentiment analysis or linguistic inference—introduces significant overhead. The standard approach, Retrieval Concatenation (RC), prepends retrieved documents to the input text. This creates a "concatenation tax": quadratic computational costs due to increased sequence length and the "truncation trap," where fixed model input limits force the cutting of vital context or original input.

Methodology: ReFusion

To address these bottlenecks, we propose ReFusion, a framework that shifts knowledge integration from the input level to the representation level. Instead of raw text concatenation, ReFusion retrieves and fuses vector representations directly into the model's hidden states.

Comparison between Retrieval Concatenation (RC) and Retrieval representation Fusion (RF)

The architecture comprises three main components:

  1. Retriever Module: Utilizes vector search (e.g., FAISS) to identify top-$k$ similar sentence representations.
  2. Retrieval Fusion Module: Refines these representations using either a learnable reranker-based weighting or an ordered-mask-based approach with differentiable "nested dropout."
  3. Adaptive Retrieval Integrator (ARI): Employs bi-level optimization to automatically determine which Transformer layers should integrate external knowledge and which fusion scheme to apply.

Experimental Evaluation

ReFusion was evaluated across 15 NKI tasks, primarily from the GLUE benchmark (including MNLI, SST-2, and CoLA), and the SQuAD dataset for knowledge-intensive verification.

Results and Key Insights

The experimental results highlight ReFusion's efficiency and robustness. Unlike RC, where FLOPs increase dramatically with more retrievals, ReFusion maintains a nearly flat computational profile because the input sequence length remains constant.

Impact of retrieval count on sequence length, accuracy, and computational cost (FLOPs)

Key findings include:

  • Breaking the Truncation Barrier: While RC performance often drops after 8 retrievals due to input truncation, ReFusion's accuracy continues to scale with $k$.
  • Computational Efficiency: At 16 retrievals, ReFusion operates at under 200G FLOPs—comparable to a base model—whereas RC exceeds 400G FLOPs.
  • Superior Performance: ReFusion achieved an average score of 74.3% across 15 tasks, outperforming the LM-BFF baseline (71.8%) and the CA-512 concatenation baseline (69.5%).
  • Layer-wise Specialization: ARI revealed that optimal fusion is non-uniform across layers, justifying the need for automated layer-specific integration.

Conclusion and Future Work

ReFusion effectively solves the "concatenation tax" by fusing knowledge in the representation space, offering a scalable and efficient alternative for RA models. While vector search adds some inference latency, the reduction in Transformer-level computation is substantial. Future work will explore more diverse ranking schemes and the application of ReFusion to larger-scale models.

Summary updated

Abstract

Retrieval-based augmentations (RA) incorporating knowledge from an external database into language models have greatly succeeded in various knowledge-intensive (KI) tasks. However, integrating retrievals in non-knowledge-intensive (NKI) tasks is still challenging. Existing works focus on concatenating retrievals with inputs to improve model performance. Unfortunately, the use of retrieval concatenation-based augmentations causes an increase in the input length, substantially raising the computational demands of attention mechanisms. This paper proposes a new paradigm of RA named \textbf{ReFusion}, a computation-efficient Retrieval representation Fusion with bi-level optimization. Unlike previous works, ReFusion directly fuses the retrieval representations into the hidden states of models. Specifically, ReFusion leverages an adaptive retrieval integrator to seek the optimal combination of the proposed ranking schemes across different model layers. Experimental results demonstrate that the proposed ReFusion can achieve superior and robust performance in various NKI tasks.
BibTeX
@misc{wu2024improving,
  author = {Shangyu Wu and Ying Xiong and Yufei Cui and Xue Liu and Buzhou Tang and Tei-Wei Kuo and Chun Jason Xue},
  title = {{Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion}},
  year = {2024},
  eprint = {2401.02993},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.02993},
}
Figure