Semantic Identifiers for Generative Information Retrieval

This direction is split into other directions: semantic document identifiers. Its last version stays readable and citable here.

Version 1 of 136 papers53 conceptsfrozen 2026-10-04

Traditional information retrieval is a game of "search and rank." When you ask a question, a system looks through a massive index of vectors or keywords to find the most relevant document. However, a new paradigm called Generative Retrieval (GR) is flipping this process. Instead of searching for a document, the model simply "speaks" its name.

In this framework, a Large Language Model (LLM) is trained to map a user query directly to a unique Document Identifier (DocID). If the model generates the correct ID, the system retrieves that document. While the concept is elegant, its success hinges entirely on a single question: What should a document's name be?

The Failure of Arbitrary Names

Early attempts at generative retrieval used numeric IDs or random hashes to identify documents. As Zhen Zhang and Weiwei Sun explore in Replication and Exploration of Generative Retrieval over Dynamic Corpora, these arbitrary identifiers create a "memorization bottleneck."

To an LLM, the number 8472 has no inherent relationship to a document about quantum physics. The model must use its parameters to memorize a rote, meaningless mapping. This approach fails to generalize to new documents and struggles to scale, as the model eventually runs out of "memory" for these random associations.

Leveraging the Pre-training Signal

The breakthrough in this field is the shift toward Semantic Identifiers—using natural language strings that describe the document's content. Researchers like Yukun Huang and Bhuwan Dhingra argue in Cite Pretrain that using document titles as identifiers is far more effective.

The intuition is simple: an LLM has already spent billions of tokens learning the relationships between words. It already "knows" that a query about "monetary policy" is semantically close to the title "Federal Reserve Annual Report." By using the title as the DocID, we transform retrieval from a rote memorization task into a semantic mapping task—one that the model is already pre-trained to solve.

Strategies for Semantic Mapping

Different researchers have proposed various ways to "name" a document to maximize this semantic alignment:

  • Summaries and Pseudo-queries: In UniGen, Xiaoxi Li and Zhicheng Dou represent documents as concise semantic summaries. This allows the model to treat retrieval as a translation task: translating a user's messy query into a clean, descriptive summary of the target document.
  • Tokenized URLs: Xin Zhao demonstrates in TOME that URLs often contain hierarchical semantic clues (e.g., en.wikipedia.org/wiki/Machine_learning). Because LLMs see millions of URLs during pre-training, these strings serve as effective, structured anchors for retrieval.
  • N-gram Identifiers: Shanbao Qiao and Seung-Hoon Na take this further in DiffusionRet by using unique n-grams (short sequences of words) extracted directly from the document. This ensures the model stays entirely within the domain of natural language, avoiding "hallucinated" IDs that don't exist in the corpus.

Why Semantic Identifiers Work

The power of this approach lies in Zero-Shot Generalization. In ZeroGR, Weiwei Sun shows that when identifiers are semantically rich, a model can retrieve documents it has never seen during fine-tuning. If a new document is added with the title "The Biology of Coral Reefs," the model can infer that a query about "underwater ecosystems" should point to that title based solely on its pre-existing linguistic knowledge.

Furthermore, Hongru Cai and Yongqi Li have found that semantic identifiers follow more predictable scaling laws. As the model size increases, its ability to map queries to semantic titles improves much faster than its ability to memorize numeric IDs.

This shift is enabling new types of "semantic" interfaces. Zeru Shi and Wenyue Hua apply this to file systems in LSFS, allowing users to find files by describing their meaning rather than remembering their exact path. Similarly, in formal mathematics, Junqi Liu and Zihao Zhou use semantic search to help agents locate relevant theorems within massive libraries like Lean, bridging the gap between high-level mathematical intent and rigid code syntax.

By treating the identity of a document as a piece of language rather than a piece of data, researchers are turning retrieval into a core competency of the language model itself, rather than an external tool it has to manage.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_identifiers_for_generative_info_1d4ca4c2,
  title = {Semantic Identifiers for Generative Information Retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-identifiers-for-generative-information-retrieval/rdr_1d4ca4c288164a6795fd70a5df6f87eb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_identifiers_for_generative_info_1d4ca4c2_v1,
  title = {Semantic Identifiers for Generative Information Retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-identifiers-for-generative-information-retrieval/rdr_1d4ca4c288164a6795fd70a5df6f87eb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}