Introduction: The "Concatenation Tax"
Retrieval-Augmented (RA) models have significantly advanced knowledge-intensive tasks like open-domain question answering. However, applying these techniques to non-knowledge-intensive (NKI) tasks—such as sentiment analysis or linguistic inference—introduces significant overhead. The standard approach, Retrieval Concatenation (RC), prepends retrieved documents to the input text. This creates a "concatenation tax": quadratic computational costs due to increased sequence length and the "truncation trap," where fixed model input limits force the cutting of vital context or original input.
Methodology: ReFusion
To address these bottlenecks, we propose ReFusion, a framework that shifts knowledge integration from the input level to the representation level. Instead of raw text concatenation, ReFusion retrieves and fuses vector representations directly into the model's hidden states.
The architecture comprises three main components:
- Retriever Module: Utilizes vector search (e.g., FAISS) to identify top-$k$ similar sentence representations.
- Retrieval Fusion Module: Refines these representations using either a learnable reranker-based weighting or an ordered-mask-based approach with differentiable "nested dropout."
- Adaptive Retrieval Integrator (ARI): Employs bi-level optimization to automatically determine which Transformer layers should integrate external knowledge and which fusion scheme to apply.
Experimental Evaluation
ReFusion was evaluated across 15 NKI tasks, primarily from the GLUE benchmark (including MNLI, SST-2, and CoLA), and the SQuAD dataset for knowledge-intensive verification.
Results and Key Insights
The experimental results highlight ReFusion's efficiency and robustness. Unlike RC, where FLOPs increase dramatically with more retrievals, ReFusion maintains a nearly flat computational profile because the input sequence length remains constant.
Key findings include:
- Breaking the Truncation Barrier: While RC performance often drops after 8 retrievals due to input truncation, ReFusion's accuracy continues to scale with $k$.
- Computational Efficiency: At 16 retrievals, ReFusion operates at under 200G FLOPs—comparable to a base model—whereas RC exceeds 400G FLOPs.
- Superior Performance: ReFusion achieved an average score of 74.3% across 15 tasks, outperforming the LM-BFF baseline (71.8%) and the CA-512 concatenation baseline (69.5%).
- Layer-wise Specialization: ARI revealed that optimal fusion is non-uniform across layers, justifying the need for automated layer-specific integration.
Conclusion and Future Work
ReFusion effectively solves the "concatenation tax" by fusing knowledge in the representation space, offering a scalable and efficient alternative for RA models. While vector search adds some inference latency, the reduction in Transformer-level computation is substantial. Future work will explore more diverse ranking schemes and the application of ReFusion to larger-scale models.
Summary updated