TL;DR
INTRA (INTrinsic Retrieval via Attention) unifies retrieval and generation within a single pretrained encoder-decoder model by repurposing its internal cross-attention mechanism to score and select evidence chunks. This approach eliminates the representation mismatch between separate retriever and generator models while allowing pre-encoded evidence states to be directly reused during generation. As a result, INTRA outperforms strong engineered RAG pipelines on multi-hop question answering while drastically reducing prefill latency.
What Problem Does This Paper Address?
Standard Retrieval-Augmented Generation (RAG) pipelines split the world into two separate systems: a retriever that finds relevant text chunks from a database, and a generator (LLM) that reads those chunks to answer a query.
This architectural separation introduces two major inefficiencies:
- Representation Mismatch: The retriever and the generator are trained separately and operate in different embedding spaces, meaning the retrieved documents might not perfectly align with what the generator actually needs to answer the question.
- Redundant Computation: At query time, the generator must re-encode the retrieved raw text chunks from scratch. For long contexts or sparse, distributed evidence, this prefilling step becomes computationally expensive and brittle.
The authors challenge this status quo by asking a fundamental question: Can a single pretrained model retrieve needed evidence directly from its own internal representation space?
Methods
The authors introduce INTRA (INTrinsic Retrieval via Attention), a framework that turns an encoder-decoder model (specifically T5Gemma2) into its own retriever.
Key components of the method include:
- Attention-Based Scoring: INTRA leverages the fact that cross-attention is already a query-conditioned matching mechanism. It augments the user's query with a small set of learnable retrieval tokens ($\rho$) to prompt the frozen decoder to generate retrieval queries.
- Reverse Query-Key Projection (Reverse-QWK): In standard transformers, key projections are layer-specific, making it impossible to use a single index for corpus-wide retrieval. Reverse-QWK mathematically flips this by moving the layer-specific projections to the query side. This allows the model to maintain a single, head-agnostic normalized encoder pool ($\bar{k}$) that can be precomputed once and searched efficiently using Approximate Nearest Neighbor (ANN) methods.
- Pooled Chunk Embeddings: To avoid computing token-to-token similarity (
MaxSim) across massive document lengths, INTRA mean-pools the encoder states into short, fixed-length sequences (e.g., length 7), dramatically reducing retrieval scoring costs.
Datasets
The framework was evaluated on an open-domain retrieval setting using a shared candidate pool of 759K chunks (~100M tokens) derived from Wikipedia. Performance was tested across four prominent QA benchmarks:
- HotPotQA & 2WikiMultihopQA: Bridge and comparison reasoning tasks.
- MuSiQue: Harder, compositionally complex multi-hop questions.
- Natural Questions (NQ): Standard single-hop open-domain QA.
Key Insights and Ideas
- Attention is an Intrinsic Retriever: We do not necessarily need external vector databases or independent dual-encoders. A model’s internal cross-attention layers already possess the necessary interface to express what information the decoder requires.
- The Power of Tight Coupling: When the retriever and generator share the exact same latent space, retrieval performance skyrockets on complex tasks. The decoder guides the retrieval process based on its exact generation needs.
- Amortized Context Encoding: Instead of treating the knowledge base as static text, INTRA treats it as static encoder memory. By encoding the corpus once offline, the model completely bypasses the quadratic text-re-encoding cost during the generation prefill phase.
Key Results
1. Superior Multi-Hop Evidence Recall
INTRA shines brightest on multi-hop benchmarks that require assembling disparate pieces of evidence. On HotPotQA, INTRA achieved a complete-evidence recall@5 of 59.9%, significantly outperforming strong dedicated dense retrievers like BGE (54.8%) and Qwen-Emb-4B (44.6%).
2. End-to-End QA Improvements
The high-quality retrieval translated directly to better answers. Using the exact same T5Gemma2 decoder for generation across all baselines, INTRA achieved the highest Exact Match (EM) scores on all multi-hop datasets (e.g., 46.4 EM on HotPotQA compared to BGE's 41.9 EM).
3. Massive Prefill Speedups
Because INTRA reuses cached encoder states rather than re-encoding text, its generator-side prefilling time remains nearly flat as more chunks ($k$) are retrieved. When retrieving 500 chunks, standard RAG requires 1.25 seconds to prefill the text, whereas INTRA takes just 65.7 milliseconds—a massive latency advantage for large-scale context injection.
Limitations and Future Work
- Architecture Constraints: INTRA relies strictly on encoder-decoder cross-attention, meaning it cannot be applied out-of-the-box to popular decoder-only models (like standard Llama or Mistral models) without architectural modifications.
- Storage Footprint: Storing raw, token-level encoder representations ($\bar{k}$) demands significantly more disk space than traditional compressed retrieval indices. Storing a 1-billion-token corpus quantized to 8-bit precision requires roughly 2.5 TB of storage.
- Scale Limits: The experiments were bounded by a fixed 100M-token context pool; validating INTRA's capabilities on web-scale or dynamically changing corpora remains an open challenge.
Summary updated