Differentiable Search Indices for Generative Retrieval

This direction is merged into another direction: generative retrieval scalability. Its last version stays readable and citable here.

Version 1 of 141 papers57 conceptsfrozen 2026-10-04

For decades, information retrieval (IR) has relied on a "matching" paradigm. To find a needle in a haystack, we first built a separate index—an inverted index for keywords or a vector database for embeddings—and then performed a lookup to find documents that matched a query.

A new research direction, often termed Generative Information Retrieval (Gen-IR) or Model-based IR, proposes a radical departure: what if the model is the index? Instead of searching an external database, a neural network learns to map a query directly to a document identifier (DocID) through sequence-to-sequence generation.

The Model as the Index

The foundational shift in this area was introduced by Dara Bahri, Yi Tay, and colleagues in their work on the Differentiable Search Index (DSI). In a DSI, a Transformer is trained to "memorize" a corpus by associating document content with unique DocIDs. At inference time, the model doesn't calculate similarity scores between vectors; it simply "speaks" the address of the relevant document.

As Xiaoxi Li and Zhicheng Dou argue in From Matching to Generation: A Survey on Generative Information Retrieval, this collapses the traditional multi-stage "index-retrieve-then-rank" pipeline into a single differentiable step. By treating retrieval as a generation task, the system can leverage the full reasoning power of Large Language Models (LLMs) to understand the semantic relationship between a user's intent and the stored knowledge.

The Memory Burden

If the model stores the index within its weights, a fundamental question arises: how many documents can a model actually remember? This is known as the memory burden.

In Generative Dense Retrieval: Memory Can Be a Burden, Peiwen Yuan and colleagues highlight that as a corpus scales to millions or billions of documents, the model parameters must store an increasingly dense mapping of identifiers. Unlike traditional indices that reside on disk or in RAM, a DSI's capacity is strictly limited by its parameter count.

Research by Kai Hui and Jimmy Lin in How Does Generative Retrieval Scale to Millions of Passages? suggests that simply increasing model size doesn't always solve the problem. At extreme scales, the model's ability to distinguish between millions of unique DocIDs can degrade, leading to "memory confusion" where the model fails to retrieve specific, fine-grained information.

Bridging the Gap with Hybrid Architectures

To overcome these scaling limits, researchers are exploring ways to offload the memory burden while retaining the benefits of generative retrieval.

One approach is to use semantic identifiers. Instead of arbitrary numbers, models like TOME, proposed by Ruiyang Ren, use tokenized URLs or natural language titles as DocIDs. This allows the model to use its pre-existing linguistic knowledge to "guess" the identifier rather than memorizing a random string. Yubao Tang furthered this in BootRet by allowing DocIDs to evolve dynamically during training, ensuring they remain semantically aligned with the model's internal representations.

Another direction is the hybrid index. The Model-enhanced Vector Index (MEVI) by Hailin Zhang uses a generative model to predict a "cluster ID" rather than a specific document. This narrows the search space significantly, after which a standard vector search identifies the exact document. This "coarse-to-fine" strategy combines the semantic precision of generation with the infinite scalability of traditional indexing.

Toward Unified Knowledge Models

The ultimate goal of this research is a unified model that doesn't just find documents, but reasons over them. CorpusLM, developed by Xiaoxi Li, demonstrates a framework where generative retrieval and answer generation happen in a single continuous decoding process.

By treating the entire corpus as a set of internal parameters, these models move us closer to the "domain expert" vision proposed by Donald Metzler in Rethinking search: making domain experts out of dilettantes. In this future, the search engine doesn't just point to a list of links; it internalizes the world's information to provide direct, cited, and authoritative answers.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_differentiable_search_indices_for_genera_128d5946,
  title = {Differentiable Search Indices for Generative Retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-search-indices-for-generative-retrieval/rdr_128d594654ef41beb50882652c551f67},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_differentiable_search_indices_for_genera_128d5946_v1,
  title = {Differentiable Search Indices for Generative Retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-search-indices-for-generative-retrieval/rdr_128d594654ef41beb50882652c551f67/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}