Information retrieval has long been dominated by two paradigms: sparse retrieval (matching exact keywords via inverted indexes) and dense retrieval (comparing vector embeddings in a shared latent space). Recently, a radical alternative has emerged: Generative Retrieval (GR), also known as the Differentiable Search Index (DSI).
First proposed by Dara Bahri et al. in Transformer Memory as a Differentiable Search Index, Generative Retrieval collapses the traditional multi-stage retrieval pipeline into a single sequence-to-sequence neural network. Instead of scanning an external database to calculate similarity scores, the model internalizes the entire document corpus within its own parameters. Given a query, the model autoregressively generates a unique discrete token sequence—a document identifier (DocID)—corresponding directly to the relevant document.
As Liu Yang et al. note in Unifying Generative and Dense Retrieval for Sequential Recommendation, this paradigm offers a massive architectural advantage: while dense retrieval's computational cost scales linearly with the number of items in the catalog during vector search, generative retrieval's inference complexity remains relatively stable because its execution time depends primarily on the vocabulary size and the length of the generated identifier.
However, this elegant formulation introduces a severe counter-bottleneck: the parametric memory scaling limit.
The Memorization Capacity Bottleneck
The fundamental flaw of pure generative retrieval is that it forces a model with a fixed parameter count to memorize an arbitrarily expanding corpus. In dense retrieval, adding a billion new documents simply requires expanding an external vector database. In generative retrieval, adding new documents requires the model to learn new mappings within its fixed weights.
As a result, generative retrieval models hit a scaling wall when moving from small benchmarks to real-world, web-scale corpora. This bottleneck manifests in several distinct ways:
- Perplexity Divergence: As documented by Ruiyang Ren et al. in TOME: A Two-stage Approach for Model-based Retrieval, as the document corpus scales up, the model's training perplexity fails to converge. The network reaches a physical capacity limit where it can no longer compress and distinguish between unique identifiers.
- Diminishing Returns from Model Scaling: Intuitively, one might expect that increasing the parameter count of the underlying Transformer would solve the capacity issue. However, comprehensive stress tests by Kai Hui et al. in How Does Generative Retrieval Scale to Millions of Passages? revealed that simply expanding model capacity is insufficient. At extreme scales (millions of passages), larger sequence-to-sequence backbones exhibit severe performance degradation and fail to match the accuracy of traditional dual-encoder architectures.
- Linear Parameter Dependency: As argued by Penghao Lu et al. in DOGR: Leveraging Document-Oriented Contrastive Learning in Generative Retrieval, storing an entire collection within internal parameters creates an unsustainable, direct dependency between corpus size and model capacity, making web-scale deployment economically and computationally prohibitive.
To overcome this scaling wall, researchers have focused on two primary dimensions: optimizing the structure of the DocIDs themselves, and designing hybrid, coarse-to-fine architectures that offload the memory burden.
Structuring the Identifier Space
The earliest DSI models used arbitrary numeric strings or static semantic clustering to assign DocIDs. However, if a DocID carries no natural language meaning, the model must expend significant parameter capacity on brute-force memorization. Modern approaches attempt to make DocIDs more compressible and structured.
Traditional Arbitrary DocID: "Query" ──> [Transformer Parameters] ──> "DocID_84729" (Hard to memorize)
Structured Semantic DocID: "Query" ──> [Transformer Parameters] ──> "Cluster_3 -> Subcluster_1" (Easier to scale)
Weiwei Sun et al. in Learning to Tokenize for Generative Retrieval demonstrate that semantic tokenization—where identifiers reflect the underlying semantic hierarchy of the document—improves learning efficiency, though scaling this to billions of web pages remains an open challenge.
To make identifiers more natural for pre-trained language models, Ruiyang Ren et al. (TOME) proposed using tokenized URLs as identifiers, aligning the retrieval target with the string patterns the model already encountered during pre-training. Alternatively, Sunkyung Lee 0001 et al. developed GLEN: Generative Retrieval via Lexical Index Learning, which utilizes dynamic lexical identifiers (keywords that evolve during training) to map high-dimensional document spaces into discrete sequence-to-sequence tasks more fluidly.
Offloading the Memory Burden
Rather than forcing a single model to act as both the map and the compass, modern scalable frameworks decouple the retrieval process into a coarse-to-fine pipeline. The generative model is tasked only with narrowing down the search space to a specific region, while an external index handles the exact document pinpointing.
Generative Space Pruning
In Generative Dense Retrieval: Memory Can Be a Burden, Peiwen Yuan et al. introduce a hybrid framework (GDR) that uses a generative model to map queries to broad document clusters rather than individual documents. Once the cluster is generated, standard dense retrieval techniques isolate the specific document within that cluster. This approach drastically reduces the number of unique identifiers the generative backbone must memorize.
Similarly, Hailin Zhang et al. propose Model-enhanced Vector Index (MEVI), which leverages a sequence-to-sequence model to predict semantic cluster IDs. By using the generative model as a highly efficient space pruner, MEVI narrows down the candidate pool before executing a localized vector search, maintaining low latency and high recall even as the document catalog scales.
Nonparametric Decoding
Another strategy to bypass the fixed-parameter limit is to inject external knowledge directly into the decoding step. Hyunji Lee et al. developed Nonparametric Decoding for Generative Retrieval, a method that replaces static vocabulary embeddings in the model's generation layer with an external memory of contextualized embeddings. By allowing the model to reference an external non-parametric distribution during token generation, smaller models can achieve the accuracy of much larger architectures without requiring a proportional increase in parameter size.
The Challenge of Dynamic Corpora
A final, critical dimension of the scalability problem is the temporal aspect: the web is dynamic, but neural network parameters are static. In a production environment, new documents are added constantly.
Retraining a generative retrieval model from scratch every time a document is indexed is computationally impossible at scale. However, fine-tuning the model on new documents leads to catastrophic forgetting, where the model overwrites the parameter pathways used to retrieve older documents.
To address this, Jiangui Chen et al. introduced CLEVER in Continual Learning for Generative Retrieval over Dynamic Corpora, utilizing an Incremental Product Quantization method alongside a memory-augmented rehearsal mechanism to stabilize identifiers.
Concurrently, Sanket Vaibhav Mehta et al. designed DSI++ (DSI++: Updating Transformer Memory with New Documents). DSI++ combines Sharpness-Aware Minimization (SAM)—which forces the model weights into flatter minima to make them more resilient to modification—with a Generative Memory that simulates past queries during incremental updates. This dual approach allows the model to continuously index new documents while maintaining a stable internal representation of the historical corpus.
By shifting from brute-force parametric memorization to structured identifier spaces, hybrid cluster-based pruning, and continual learning frameworks, the field is steadily closing the scalability gap, moving generative retrieval from a compelling theoretical paradigm to a viable architecture for large-scale search.