Traditional information retrieval follows a "retrieve-and-rerank" pipeline. You turn documents into vectors, store them in an external database, and use a "dual encoder" to find the closest matches for a query.
Generative Information Retrieval (GIR) — often implemented as a Differentiable Search Index (DSI) — collapses this entire process into a single model. Instead of looking up a document in an external index, the model "memorizes" the corpus and learns to generate a unique Document Identifier (DocID) directly from a query.
The Shift from Matching to Generating
In a standard system, the model is a matcher. In a generative system, the model is a librarian who has memorized every book in the stacks. When you ask a question, the librarian doesn't go to the catalog; they simply tell you the call number of the book you need.
This approach, pioneered in the DSI framework, treats retrieval as a sequence-to-sequence task: Query -> DocID. As Qiaoyu Tang and Jiawei Chen demonstrate in Self-Retrieval: End-to-End Information Retrieval with One Large Language Model, this allows a single Large Language Model (LLM) to internalize the corpus, potentially eliminating the need for separate vector databases and rerankers.
The Language of Identifiers
The core challenge of GIR is the "DocID." If you give every document a random number, the model has to memorize millions of arbitrary mappings. To make retrieval scalable, researchers have focused on making DocIDs semantic or lexical.
- Semantic IDs: Bowen Jin and colleagues introduced LMINDEXER, which uses self-supervised learning to assign hierarchical, discrete IDs to documents. If two documents are about "Quantum Physics," their IDs will share a common prefix (e.g.,
102-45-1and102-45-9). This allows the model to navigate the document space like a tree, narrowing down the search with every token it generates. - Lexical IDs: Models like GLEN by Sunkyung Lee use keywords or "lexical identifiers" that evolve during training. Similarly, TOME, developed by Ruiyang Ren and Xin Zhao, uses tokenized URLs as identifiers, leveraging the natural language patterns the model already learned during pre-training.
The Scaling Wall
While GIR works beautifully on small datasets (like NQ320K), it often struggles when scaled to millions of documents. Kai Hui and Ronak Pradeep conducted a stress test in How Does Generative Retrieval Scale to Millions of Passages?, finding that as the corpus grows, the performance gap between generative models and traditional dual encoders widens.
The problem is "memorization capacity." A model has a finite number of parameters to store the "essence" of millions of documents. To combat this, Hansi Zeng and Chen Luo proposed RIPOR, which optimizes for document ID prefixes and uses relevance-based identifiers to help the model distinguish between similar documents at scale.
Solving the "False Pruning" Problem
Generative retrieval usually relies on constrained beam search. The model generates the first token of a DocID, then the second, and so on, restricted by a "trie" (a prefix tree) of all valid IDs.
However, if the model makes a mistake on the very first token, the correct document is "pruned" away forever. Peitian Zhang addressed this in TSGen by moving from rigid sequences to term sets. By allowing the model to generate document terms in any order (permutation-invariant decoding), the system becomes much more robust to small errors in the generation process.
Beyond Static Corpora
A major criticism of early DSI models was their "brittleness" to new data; adding a single document required retraining the whole model.
Recent work like MixLoRA-DSI by Thuy-Trang Vu attempts to solve this using a Mixture-of-Experts (MoE) approach. By dynamically expanding "LoRA experts" only when novel information is detected, the model can learn new documents continually without forgetting the old ones, making generative retrieval a viable candidate for the ever-changing web.