The Paradigm Shift: Indexing as Memorization
Traditional information retrieval relies on an explicit separation between the index and the retrieval model. Whether using an inverted index (like BM25) or a vector database (like dense retrievers), the index exists as an external data structure.
Recently, a new paradigm called Generative Retrieval (GR) or the Differentiable Search Index (DSI) collapsed this pipeline into a single neural network. As introduced by Dara Bahri et al. in Transformer Memory as a Differentiable Search Index, a DSI internalizes the entire corpus within its fixed parameters. Instead of matching query tokens to document vectors, the model acts as a sequence-to-sequence generator that maps a query directly to a unique document identifier (DocID). Similarly, Qi Chen et al. demonstrated with A Neural Corpus Indexer for Document Retrieval that unifying training and indexing into a single differentiable network can significantly outperform traditional multi-stage pipelines.
The Problem: The Static Parameter Trap
While parameterizing the index yields impressive retrieval accuracy, it introduces a fundamental flaw: the index is frozen inside the model's weights.
In the real world, document collections are highly dynamic—news breaks, articles are updated, and old content is deleted. In a traditional inverted index or vector database, adding a new document is trivial: you tokenize or embed the text and insert a new row into the database. However, as Kai Hui et al. point out in How Does Generative Retrieval Scale to Millions of Passages?, updating a generative retrieval system typically requires expensive full model retraining or fine-tuning.
If you try to fine-tune the model on only the new documents, it suffers from catastrophic forgetting—the neural network overwrites its previously memorized document mappings to accommodate the new data. As surveyed by Xiaoxi Li et al. in From Matching to Generation: A Survey on Generative Information Retrieval, overcoming this trade-off without full re-indexing is one of the most critical barriers to deploying generative retrieval in production environments.
Mechanisms for Dynamic Index Maintenance
To make neural indices practical, researchers have developed several distinct strategies to enable fast, incremental updates while preserving existing knowledge.
Constrained Optimization
One intuitive approach is to treat the addition of new documents as a localized surgical operation on the model's weights rather than a global retraining process. In IncDSI: Incrementally Updatable Document Retrieval, Varsha Kishore et al. formulated document ingestion as a constrained optimization problem. By defining a loss function that specifically binds new DocIDs to their text while minimizing changes to the parameters responsible for existing documents, they achieved millisecond-level updates with minimal degradation in overall retrieval performance.
Continual Learning and Memory Rehearsal
Another strategy borrows from the field of continual learning, using memory buffers to prevent catastrophic forgetting. Sanket Vaibhav Mehta et al. introduced DSI++: Updating Transformer Memory with New Documents, which combines Sharpness-Aware Minimization (SAM) to find flatter, more stable loss minima with a "Generative Memory" that simulates and replays past queries during incremental training.
Similarly, Jiangui Chen et al. proposed the CLEVER framework in Continual Learning for Generative Retrieval over Dynamic Corpora. CLEVER uses an incremental product quantization method alongside a memory-augmented rehearsal mechanism. This dual approach ensures that the model can generate stable identifiers for new documents while maintaining near-zero forgetting of the original corpus across multiple successive update cycles.
Coarse-to-Fine Hybrid Indexing
Rather than forcing the neural network to memorize exact, rigid DocIDs for every single document, some architectures offload fine-grained indexing to a hybrid structure. Hailin Zhang et al. designed the Model-enhanced Vector Index (MEVI) to map queries to semantic cluster IDs rather than individual documents. When a new document arrives, it is simply assigned to an existing cluster based on its embedding. Because the generative model only needs to predict the coarse cluster ID, the underlying neural network does not need to be retrained; the final document pinpointing is handled dynamically via standard vector search within that cluster.
Summary of the Intuition
Why do these methods work? They succeed because they break the rigid coupling between a document's content and its neural representation. Whether by constraining weight updates to a narrow subspace, replaying synthetic historical queries to anchor old memories, or shifting the model's target from individual documents to broader semantic clusters, these techniques transform the differentiable search index from a static snapshot into a living, adaptable data structure.
Key Foundational Work
String Transformation Learning (1997) — contributed 2 concept elements to this direction.
On Updating Problems in Latent Semantic Indexing (1999) — contributed 1 concept elements to this direction.