Traditional information retrieval (IR) relies on an explicit, multi-stage pipeline. First, an offline index is built—either an inverted index for sparse keyword matching or a dense vector database for semantic similarity. When a query arrives, a dual-encoder model maps the query into the same embedding space, performs an approximate nearest neighbor (ANN) search to pull candidates, and passes them to a heavy cross-encoder for reranking.
This architecture suffers from two fundamental limitations: the embedding bottleneck and index isolation. The embedding bottleneck occurs because an entire document must be compressed into a single, fixed-size vector, stripping away fine-grained contextual nuances. Index isolation means the retrieval model and the index structure are decoupled; the model cannot dynamically optimize how the index is structured or traversed.
To solve this, Bahri et al. and Qi Chen et al. proposed a radical alternative: the Differentiable Search Index (DSI), also known as Generative Retrieval. Instead of searching an external index, a single sequence-to-sequence Transformer internalizes the entire corpus within its parameters. The retrieval task is reframed as a direct mapping from an input query to a unique document identifier (DocID) via autoregressive text generation.
Why It Works: Parametric Compression and Full Cross-Attention
Why does shifting from vector matching to text generation improve retrieval? The answer lies in how information is stored and processed during search.
In traditional dense retrieval, storing high-dimensional embeddings for millions of documents requires massive external vector databases, creating a heavy memory footprint. Generative retrieval bypasses this entirely by using parametric corpus memorization. The mapping between a document's content and its identifier is stored directly within the weights of the language model.
As demonstrated by De Cao et al., Jiangui Chen et al., and Aly et al., this parameter-based indexing can achieve a 10x to 30x reduction in memory overhead compared to traditional dense indices. Instead of scaling linearly with the number of documents ($O(N)$ space), the memory footprint scales primarily with the model's vocabulary and parameter size.
Furthermore, generative retrieval eliminates the embedding bottleneck. When a dual encoder computes a similarity score, the query and document representations only interact via a simple dot product at the very end. In contrast, an autoregressive retriever uses full cross-attention (or causal self-attention) between the query tokens and the generated DocID tokens. As Lee et al. point out, this allows the model to leverage deep, token-level interactions across the entire context, which is particularly vital for complex, multi-hop reasoning tasks.
The Anatomy of a Document Identifier
If the model must generate a sequence of tokens representing a document, what should those tokens be? The choice of Document Identifier (DocID) is the core mechanism that makes generative retrieval work. Researchers have converged on three main strategies:
1. Semantic Hierarchical IDs
Documents are clustered hierarchically based on their semantic content, and each document is assigned a string of codes representing its path down the tree (e.g., 3-1-4-2). During inference, the model decodes this path token-by-token, effectively navigating a semantic taxonomy. For instance, Ji Won Kim et al. utilized a Residual Quantization Variational Auto-Encoder (RQ-VAE) to generate hierarchical codebook indices for item recommendation, transforming a massive database lookup into a structured sequence-to-sequence problem.
2. Substrings and N-grams
Instead of synthetic codes, a document can be identified by any unique substring or n-gram contained within its text. Bevilacqua et al. developed an autoregressive search engine (SEAL) that generates document substrings. To ensure the model only generates valid strings that actually exist in the corpus, the authors paired the language model with a compressed FM-Index to constrain the decoding trie.
3. Direct Content Generation
In some frameworks, the DocID is the document text itself. He introduced a "Self-Retrieval" architecture where the model internalizes the corpus by training on a sentence-to-passage generation task, allowing it to output relevant passages directly in response to a query without any intermediate indexing codes.
Overcoming the Scaling Wall
While generative retrieval excels on small-to-medium datasets, scaling the paradigm to millions of documents introduces significant challenges. Hui et al. conducted a large-scale stress test scaling up to 8.8 million passages, revealing that vanilla generative retrieval struggles to maintain accuracy against traditional dual encoders at massive scales due to the difficulty of memorizing millions of distinct identifier mappings.
To overcome this scaling wall, recent work focuses on optimizing the indexing and decoding objectives. Zeng et al. introduced RIPOR, a framework that optimizes generative models specifically for prefix ranking and relevance-based indexing. By aligning the training objective with the sequential nature of beam search, the authors achieved performance parity with traditional dense retrieval systems on large-scale benchmarks. Additionally, Chaeeun Kim et al. demonstrated that generative retrieval maintains a constant inference complexity relative to corpus size, providing a distinct scaling advantage over vector-based methods where search time scales with index size.
Generalizing Beyond Text
The elegance of framing retrieval as sequence generation is that it is fundamentally modality-agnostic. If an asset can be tokenized into discrete identifiers, it can be retrieved generatively.
- Computer Vision: Sun et al. developed IRGen, which reframes image retrieval by training a sequence-to-sequence Transformer to generate discrete identifiers of an image's nearest neighbors, bypassing distance-based vector matching.
- Audio Retrieval: Fang et al. introduced GTA, using a multi-scale audio tokenizer to convert sounds into discrete semantic IDs, allowing a language model to directly predict the correct audio for a text query.
- Structured Databases: Wang et al. (DBCopilot) and Yuxiang Guo et al. (Birdie) applied differentiable search indexing to navigate massive database schemas and discover tables by generating structured table identifiers directly from natural language queries.
- Biochemical Discovery: Sicherman et al. (BioCG) reformulated biochemical interaction prediction into a constrained sequence generation task, where the model outputs the unique discrete code of a target entity given a query entity.
Key Foundational Work
Natural Logic-guided Autoregressive Multi-hop Document Retrieval for Fact Verification (2022) — contributed 3 concept elements to this direction.
Generative Multi-hop Retrieval (2022) — contributed 3 concept elements to this direction.
Rethinking search: making domain experts out of dilettantes (2021) — contributed 3 concept elements to this direction.