Generative Retrieval with Differentiable Search Indices

This direction is split into other directions: generative retrieval latency. Its last version stays readable and citable here.

Version 1 of 162 papers66 conceptsfrozen 2026-10-04

For decades, information retrieval has followed a "represent and match" pipeline. We encode documents into a vector space, store them in an external index, and at query time, perform a nearest-neighbor search. However, a new paradigm is emerging that collapses this entire process into a single neural network. In this framework, known as Generative Retrieval (GR) or the Differentiable Search Index (DSI), a model does not search for a document; it simply generates its name.

From Searching to Generating

In a traditional system, the model is a matchmaker. In a DSI system, the model is the index itself. As first proposed by Dara Bahri, Yi Tay, and colleagues in Transformer Memory as a Differentiable Search Index, the Transformer is trained to map document content directly to a unique document identifier (DocID).

When a user asks a question, the model uses autoregressive decoding—the same mechanism LLMs use to write stories—to output the DocID of the relevant passage. This approach, also explored by Qi Chen in the Neural Corpus Indexer, allows the model to learn the relationship between queries and documents in a fully end-to-end, differentiable way.

The Language of Identifiers

The success of generative retrieval hinges on how we "name" documents. If DocIDs are just random integers, the model struggles to learn semantic relationships. Researchers have developed several strategies to make these identifiers more meaningful:

  1. Semantic IDs: Models like TIGER and LMINDEXER by Bowen Jin use hierarchical clustering to create IDs. If two documents are about "Quantum Physics," their DocIDs might share the same prefix (e.g., 102.45.1 and 102.45.2), helping the model navigate the search space.
  2. N-gram Substrings: The SEAL framework, developed by Michele Bevilacqua and Patrick Lewis, trains the model to generate any unique substring found within the document. This allows the model to use the document's own language as its address.
  3. Attribute-Based Codes: In e-commerce, Ming Pang and the JD.com team introduced GRAM, which uses product attributes (like brand or category) to form DocIDs, bridging the gap between messy user queries and structured product data.

The Autoregressive Bottleneck

While generative retrieval is elegant, it faces a significant "latency penalty." Traditional vector search is incredibly fast because it relies on optimized dot-product operations. In contrast, GR requires generating tokens one by one. As noted by Rakesh Chada and Jiapeng Wang, this autoregressive decoding introduces computational overhead that can be prohibitive for real-time, high-throughput systems.

Furthermore, beam search—the standard way to find the best DocID—can be "short-sighted." If the first token of a relevant DocID has a low probability, the model might prune it early. To solve this, Hansi Zeng and Chen Luo developed PAG (Planning Ahead in Generative Retrieval), which uses a fast "lookahead" step to estimate total document relevance before committing to specific tokens, making the process both 22x faster and more accurate.

Scaling to Billions

The "memory burden" is the final frontier for generative retrieval. As a corpus grows to millions or billions of documents, the model's parameters must store an increasing amount of information.

Recent work has moved toward hybrid architectures to maintain scalability. MEVI, proposed by Hailin Zhang, uses a generative model to predict a "cluster ID" rather than a specific document, narrowing the search space before a traditional vector search takes over. Similarly, Hi-Gen by Jun Xiao demonstrates that for billion-scale e-commerce search, generative models work best as "seed generators" that trigger more efficient downstream retrieval processes.

By internalizing the index, generative retrieval moves us closer to a "Self-Retrieval" future, as envisioned by Qiaoyu Tang and Jiawei Chen, where a single Large Language Model handles indexing, retrieval, and reasoning in one unified, differentiable flow.

Papers in this version

Showing 60 of 62 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generative_retrieval_with_differentiable_09bd2898,
  title = {Generative Retrieval with Differentiable Search Indices},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-retrieval-with-differentiable-search-indices/rdr_09bd289859a24504ad79c37d4dc5cbbe},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_generative_retrieval_with_differentiable_09bd2898_v1,
  title = {Generative Retrieval with Differentiable Search Indices},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-retrieval-with-differentiable-search-indices/rdr_09bd289859a24504ad79c37d4dc5cbbe/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}