Independent Encoding with Joint Decoding for Retrieval Augmentation

62 concepts56 papers1 proposal

When large language models need to answer complex questions, they rarely rely on a single source of information. Instead, modern retrieval-augmented generation (RAG) systems fetch dozens or even hundreds of documents to find relevant evidence. However, this introduces a severe computational bottleneck: how can a model process a massive pool of external texts without drowning in the computational cost of self-attention?

The traditional approach—concatenating all retrieved documents into a single long prompt—runs directly into the quadratic complexity wall of the Transformer encoder. If a model attempts to read 100 documents simultaneously, the cost of full self-attention scales quadratically with the total number of tokens, quickly exhausting hardware memory.

To bypass this bottleneck, researchers developed an architectural paradigm known as Fusion-in-Decoder (FiD). By decoupling how a model reads documents from how it synthesizes answers, this approach allows language models to scale linearly to massive evidence pools.

The Core Mechanism: Splitting the Workload

The core intuition behind independent encoding with joint decoding is simple: separate the reading phase from the synthesis phase. Instead of forcing every token in every document to attend to every token in every other document, the architecture splits the workload into two distinct steps.

Query + Doc 1  ──►  [ Encoder ]  ──┐
Query + Doc 2  ──►  [ Encoder ]  ──┼──►  Concatenate Representations  ──►  [ Decoder ]  ──► Final Answer
Query + Doc N  ──►  [ Encoder ]  ──┘

1. Independent Encoding

First, the system pairs the user's query with each retrieved document individually. If there are $N$ documents, the encoder runs $N$ separate times (or in parallel as a batch).

$$\text{Input}_i = [\text{Query}; \text{Document}_i]$$

Because the encoder only processes one document at a time, tokens in Document 1 never look at tokens in Document 2 during this stage. If each document has a length of $L$, the computational complexity for encoding a single document is $O(L^2)$. For $N$ documents, the total encoder complexity is:

$$\text{Total Encoder Cost} = N \times O(L^2)$$

This is the crucial breakthrough: the computational cost now scales linearly ($O(N)$) with the number of documents, rather than quadratically ($O(N^2)$). As noted by Marzieh S. Tahaei et al., this linear scaling allows even smaller language models to handle vast context pools efficiently. Shuyan Zhou et al. and Tianyuan Shi et al. similarly leverage this property to overcome context window limitations when querying large external documentation libraries or databases.

2. Joint Decoding

Once every document is encoded independently, their resulting vector representations are concatenated into a single, continuous sequence. This combined representation is then passed to the Transformer decoder.

During generation, the decoder uses cross-attention to look back at the encoded text. Crucially, cross-attention does not require the encoder tokens to attend to each other; it only requires the newly generated decoder tokens to attend to the encoder tokens. For each token the decoder generates, the cost of cross-attention is linear with respect to the total number of context tokens ($N \times L$).

By delaying the interaction between documents until the decoding stage, the model achieves the best of both worlds: efficient, isolated reading and global, centralized synthesis.

Why It Works: The Centralized Workspace of Cross-Attention

Why not go a step further and make the decoding process independent too? For instance, one could generate an answer for each document separately and then average the output probabilities. As Yizhe Zhang et al. observe, while independent generation passes can maintain a small context window, they sacrifice a vital capability: cross-document reasoning.

In complex reasoning tasks, the answer to a question is rarely contained within a single snippet of text. The evidence is often distributed across multiple sources. As Ji Wu et al. demonstrate, models must integrate information across distinct documents to solve case-based or multi-step queries.

By performing cross-attention over all document representations simultaneously, the decoder acts as a centralized workspace. When generating the next word, the decoder can weigh a fact from Document A against a premise from Document B. In their foundational work on Fusion-in-Decoder, Gautier Izacard et al. showed that this generative synthesis allows performance to scale monotonically as the model is given more context, effectively handling up to 100 retrieved passages without saturating.

This global workspace also plays a critical role in truthfulness and conflict resolution:

  • Reducing Hallucinations: Kurt Shuster et al. showed that independent encoding followed by joint decoding allows models to ground their responses firmly in the provided text, drastically reducing the tendency of language models to invent plausible-sounding falsehoods.
  • Resolving Evidence Conflicts: When external sources disagree, a model processing documents in isolation can become confused or default to its internal training biases. Hung-Ting Chen et al. found that exposing the decoder to all evidence simultaneously allows it to better navigate conflicting information and calibrate its confidence.
  • Correcting False Assumptions: In tasks where queries contain incorrect premises, Sewon Min demonstrates that joint decoding enables the model to synthesize a comprehensive counter-proof by pulling disparate facts from across the entire retrieved corpus.

Structural and Modality Flexibility

Because the encoder treats each source as an isolated input stream, the independent-encoding paradigm is highly modular. The architecture does not care if the inputs are uniform text snippets, structured data, or even different modalities.

Fusing Tables and Text

In open-domain question answering, knowledge is frequently split between unstructured text (like Wikipedia articles) and structured tables (like database records). Barlas Oguz et al. and Alexander Hanbo Li et al. exploit the modularity of FiD by linearizing text snippets and table rows into separate strings, encoding them independently, and letting the decoder fuse them. Wenting Zhao et al. utilize a similar strategy to generate free-form answers that remain faithful to hybrid data sources.

Multi-Image Processing

The same principle applies to computer vision. When answering questions about a multi-page document or a slide deck, processing multiple high-resolution images simultaneously is computationally prohibitive. Kyosuke Nishida et al. adapted the late-fusion architecture for visual QA by encoding individual slide images independently and using the decoder to perform cross-modal fusion across the entire deck.

Cross-Lingual Synthesis

Information-seeking queries often require pulling evidence from sources written in different languages. Nikita Sorokin et al. demonstrate that a joint decoder can accept independent encodings from multiple language sources simultaneously, allowing it to translate, cross-reference, and synthesize an answer back into the user's native language.

Alternative Paradigms: Fusing Earlier or Later

While late fusion in the decoder is the most common implementation, the trade-off between computational efficiency and deep inter-document interaction remains an active area of research.

To capture interactions between documents before the decoding stage without incurring full quadratic costs, some architectures introduce early-stage communication channels. For example, Akhil Kedia et al. proposed an alternative "Fusion-in-Encoder" approach that uses specialized global tokens to route information between independent passages during the encoding phase. Similarly, Liang Wen et al. developed a multi-stage network that aggregates evidence at both the encoding and decoding stages to capture fine-grained document interactions early in the pipeline.

Nevertheless, the clean mathematical separation of independent encoding remains a dominant strategy for end-to-end optimization. As Devendra Singh Sachan et al. demonstrate, treating independently encoded documents as latent variables makes it possible to propagate gradients directly from the decoder back to the retrieval system, allowing the reader and retriever to be trained jointly in a fully differentiable manner.

Key Foundational Work

Multi-Source Neural Machine Translation with Missing Data (2018) — contributed 1 concept elements to this direction.
Multi-Task Multi-Resolution Char-to-BPE Cross-Attention Decoder for End-to-End Speech Recognition (2019) — contributed 1 concept elements to this direction.
Medical Exam Question Answering with Large-scale Reading Comprehension (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fusion_in_decoder_597ed3b1,
  title = {fusion-in-decoder},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fusion-in-decoder/rdr_597ed3b191d247ec944ca3b2a6371e90},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_fusion_in_decoder_597ed3b1_v1,
  title = {fusion-in-decoder},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fusion-in-decoder/rdr_597ed3b191d247ec944ca3b2a6371e90/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake