Retrieval from Within: An Intrinsic Capability of Attention-Based Models

Elad Hoffer, Yochai Blau, Edan Kinderman, Ron Banner, Daniel Soudry, Boris GinsburgarXiv 2026 · arXiv

TL;DR

INTRA (INTrinsic Retrieval via Attention) unifies retrieval and generation within a single pretrained encoder-decoder model by repurposing its internal cross-attention mechanism to score and select evidence chunks. This approach eliminates the representation mismatch between separate retriever and generator models while allowing pre-encoded evidence states to be directly reused during generation. As a result, INTRA outperforms strong engineered RAG pipelines on multi-hop question answering while drastically reducing prefill latency.

What Problem Does This Paper Address?

Standard Retrieval-Augmented Generation (RAG) pipelines split the world into two separate systems: a retriever that finds relevant text chunks from a database, and a generator (LLM) that reads those chunks to answer a query.

This architectural separation introduces two major inefficiencies:

  1. Representation Mismatch: The retriever and the generator are trained separately and operate in different embedding spaces, meaning the retrieved documents might not perfectly align with what the generator actually needs to answer the question.
  2. Redundant Computation: At query time, the generator must re-encode the retrieved raw text chunks from scratch. For long contexts or sparse, distributed evidence, this prefilling step becomes computationally expensive and brittle.

The authors challenge this status quo by asking a fundamental question: Can a single pretrained model retrieve needed evidence directly from its own internal representation space?

Methods

The authors introduce INTRA (INTrinsic Retrieval via Attention), a framework that turns an encoder-decoder model (specifically T5Gemma2) into its own retriever.

Figure 1: Left: Standard RAG pipeline. An external retriever selects documents which are then re-encoded by the decoder to produce an answer. Retrieval and generation operate in separate representation spaces. Right: Our method, INTRA, uses a pretrained frozen encoder-decoder for both retrieval and generation. The decoder retrieves relevant chunks through its cross-attention queries, augmented with learnable retrieval tokens. The retriever and generator share a representation space, allowing pre-encoded evidence to be reused across queries. No external retriever is required. Numbers indicate the sequence of operations.

Key components of the method include:

  • Attention-Based Scoring: INTRA leverages the fact that cross-attention is already a query-conditioned matching mechanism. It augments the user's query with a small set of learnable retrieval tokens ($\rho$) to prompt the frozen decoder to generate retrieval queries.
  • Reverse Query-Key Projection (Reverse-QWK): In standard transformers, key projections are layer-specific, making it impossible to use a single index for corpus-wide retrieval. Reverse-QWK mathematically flips this by moving the layer-specific projections to the query side. This allows the model to maintain a single, head-agnostic normalized encoder pool ($\bar{k}$) that can be precomputed once and searched efficiently using Approximate Nearest Neighbor (ANN) methods.
  • Pooled Chunk Embeddings: To avoid computing token-to-token similarity (MaxSim) across massive document lengths, INTRA mean-pools the encoder states into short, fixed-length sequences (e.g., length 7), dramatically reducing retrieval scoring costs.

Datasets

The framework was evaluated on an open-domain retrieval setting using a shared candidate pool of 759K chunks (~100M tokens) derived from Wikipedia. Performance was tested across four prominent QA benchmarks:

  • HotPotQA & 2WikiMultihopQA: Bridge and comparison reasoning tasks.
  • MuSiQue: Harder, compositionally complex multi-hop questions.
  • Natural Questions (NQ): Standard single-hop open-domain QA.

Key Insights and Ideas

  • Attention is an Intrinsic Retriever: We do not necessarily need external vector databases or independent dual-encoders. A model’s internal cross-attention layers already possess the necessary interface to express what information the decoder requires.
  • The Power of Tight Coupling: When the retriever and generator share the exact same latent space, retrieval performance skyrockets on complex tasks. The decoder guides the retrieval process based on its exact generation needs.
  • Amortized Context Encoding: Instead of treating the knowledge base as static text, INTRA treats it as static encoder memory. By encoding the corpus once offline, the model completely bypasses the quadratic text-re-encoding cost during the generation prefill phase.

Key Results

1. Superior Multi-Hop Evidence Recall

INTRA shines brightest on multi-hop benchmarks that require assembling disparate pieces of evidence. On HotPotQA, INTRA achieved a complete-evidence recall@5 of 59.9%, significantly outperforming strong dedicated dense retrievers like BGE (54.8%) and Qwen-Emb-4B (44.6%).

Figure 1: Left: Standard RAG pipeline. An external retriever selects documents which are then re-encoded by the decoder to produce an answer. Retrieval and generation operate in separate representation spaces. Right: Our method, INTRA, uses a pretrained frozen encoder-decoder for both retrieval and generation. The decoder retrieves relevant chunks through its cross-attention queries, augmented with learnable retrieval tokens. The retriever and generator share a representation space, allowing pre-encoded evidence to be reused across queries. No external retriever is required. Numbers indicate the sequence of operations.

2. End-to-End QA Improvements

The high-quality retrieval translated directly to better answers. Using the exact same T5Gemma2 decoder for generation across all baselines, INTRA achieved the highest Exact Match (EM) scores on all multi-hop datasets (e.g., 46.4 EM on HotPotQA compared to BGE's 41.9 EM).

3. Massive Prefill Speedups

Because INTRA reuses cached encoder states rather than re-encoding text, its generator-side prefilling time remains nearly flat as more chunks ($k$) are retrieved. When retrieving 500 chunks, standard RAG requires 1.25 seconds to prefill the text, whereas INTRA takes just 65.7 milliseconds—a massive latency advantage for large-scale context injection.

Limitations and Future Work

  • Architecture Constraints: INTRA relies strictly on encoder-decoder cross-attention, meaning it cannot be applied out-of-the-box to popular decoder-only models (like standard Llama or Mistral models) without architectural modifications.
  • Storage Footprint: Storing raw, token-level encoder representations ($\bar{k}$) demands significantly more disk space than traditional compressed retrieval indices. Storing a 1-billion-token corpus quantized to 8-bit precision requires roughly 2.5 TB of storage.
  • Scale Limits: The experiments were bounded by a fixed 100M-token context pool; validating INTRA's capabilities on web-scale or dynamically changing corpora remains an open challenge.

Summary updated

Abstract

Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems. We ask whether an attention-based encoder-decoder can instead retrieve directly from its own internal representations. We introduce INTRA (INTrinsic Retrieval via Attention), a framework where decoder attention queries score pre-encoded evidence chunks that are then directly reused as context for generation. By construction, INTRA unifies retrieval and generation, eliminating the retriever-generator mismatch typical of RAG pipelines. This design also amortizes context encoding by reusing precomputed encoder states across queries. On question-answering benchmarks, INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality. Our results demonstrate that attention-based models already possess a retrieval mechanism that can be elicited, rather than added as an external module.
BibTeX
@misc{hoffer2026retrieval,
  author = {Elad Hoffer and Yochai Blau and Edan Kinderman and Ron Banner and Daniel Soudry and Boris Ginsburg},
  title = {{Retrieval from Within: An Intrinsic Capability of Attention-Based Models}},
  year = {2026},
  eprint = {2605.05806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.05806},
}
Figure