Fusion in Decoder for Large Scale Retrieval Augmentation

This direction is split into other directions: fusion-in-decoder. Its last version stays readable and citable here.

Version 1 of 1115 papers128 conceptsfrozen 2026-10-04

Large Language Models (LLMs) often struggle with "goldfish memory"—a limited context window that prevents them from considering vast amounts of external information. When we try to give a model 100 different documents to help it answer a question, we hit a mathematical wall: the quadratic cost of self-attention. Standard Transformers compare every token to every other token, meaning that doubling the input length quadruples the computational effort.

To bypass this, researchers have converged on a structural shift known as Fusion-in-Decoder (FiD). Instead of forcing the model to read everything at once, this architecture allows models to process dozens or even hundreds of documents independently before synthesizing them into a single coherent answer.

The Bottleneck of Joint Attention

In a traditional "early fusion" approach, you concatenate all retrieved documents into one long string and feed it into the encoder. If you have 100 documents of 100 tokens each, the encoder performs attention over 10,000 tokens. This is not only slow but often exceeds the hardware memory limits of even the most powerful GPUs.

As Yifan Gao and colleagues observed in their work on REFUEL, this "concatenation bottleneck" severely limits the amount of evidence a model can use to resolve ambiguous questions. If the model can only "see" five documents, it might miss the one interpretation that actually answers the user's query.

How Fusion in Decoder Works

The FiD architecture, utilized across diverse tasks from commonsense reasoning to code generation, breaks the problem into two distinct phases:

  1. Independent Encoding: Each retrieved document is paired with the user's query and processed by the encoder separately. Because the documents never "see" each other in this stage, the computational cost scales linearly ($O(N)$) with the number of documents, rather than quadratically ($O(N^2)$).
  2. Late Fusion in the Decoder: The decoder then performs cross-attention over the concatenated representations of all these documents. This is where the "fusion" happens. The decoder acts like a lead editor, scanning through the "notes" provided by the encoder for each document to pick out the relevant facts.

This modularity is what allows Wenhao Yu and the RACO team to scale to a 20-million-document library, or Alireza Salemi to inject massive amounts of user history for personalized outputs.

Scaling to Hundreds of Documents

The primary intuition behind FiD is that most documents in a retrieved set are noise. By encoding them independently, the model avoids the "distraction" of inter-document self-attention.

In the realm of long-term conversation, Arthur Szlam and Jason Weston demonstrated that this late-fusion approach allows chatbots to maintain multi-session personas by retrieving and processing many past memories without overwhelming the model's initial representation space. Similarly, Tiezheng Yu showed that for meeting summarization, this architecture enables the processing of significantly longer contexts than standard transformers by maintaining linear scaling.

Applications Across Domains

The versatility of the FiD mechanism has made it a "Swiss Army knife" for knowledge-intensive tasks:

  • Programming: Shuyan Zhou used FiD in DocCoder to let models "read" documentation for libraries they weren't trained on, processing multiple snippets to find the right function call.
  • Visual Reasoning: In Plug-and-Play VQA, researchers used FiD to aggregate dozens of generated image descriptions, allowing a smaller model to outperform giants like Flamingo by reasoning over text-based visual evidence.
  • Medical Research: For MS^2, the architecture was used to summarize hundreds of medical studies, where each study must be encoded independently to preserve its specific relationship to the research query.

By decoupling the reading of individual sources from the synthesis of the final answer, Fusion-in-Decoder provides a blueprint for AI that can reason over the "open world" of information rather than just the narrow window of its immediate memory.

Papers in this version

Showing 60 of 115 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fusion_in_decoder_for_large_scale_retrie_a4243ffc,
  title = {Fusion in Decoder for Large Scale Retrieval Augmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fusion-in-decoder-for-large-scale-retrieval-augmentation/rdr_a4243ffc08d046178f20917091f52fb8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_fusion_in_decoder_for_large_scale_retrie_a4243ffc_v1,
  title = {Fusion in Decoder for Large Scale Retrieval Augmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fusion-in-decoder-for-large-scale-retrieval-augmentation/rdr_a4243ffc08d046178f20917091f52fb8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}