Can Generative Retrievers Learn Semantic IDs Without Forgetting How to Speak?

Junchen Fu, Kleomenis Katevas, Vandana Rajan, Sofía Celi, Hamed HaddadiarXiv 2026 · arXiv

This summary explores the paper "Can Generative Retrievers Learn Semantic IDs Without Forgetting How to Speak?", which tackles a critical hurdle in building unified AI agents: the tendency of models to lose their conversational abilities when trained to act as search engines.

TL;DR

Generative retrieval (GR) turns LLMs into powerful search engines by teaching them to generate document "Semantic IDs," but this specialized training causes the models to rapidly "forget" how to speak natural language. The authors introduce SpeakGR, a framework that uses on-policy distillation to preserve a model's original language distribution while it learns to retrieve, enabling a single model to both find documents and chat about them effectively.

What Problem Does This Paper Address?

Generative Retrieval is a paradigm shift where a neural model generates a document's identifier (SID) directly from a query, effectively internalizing the search index. While powerful, standard fine-tuning for this task is "narrow." The model learns to map queries to a tiny vocabulary of SIDs, often at the expense of its massive pre-trained knowledge of human language.

The authors demonstrate that this "catastrophic forgetting" is not just a minor side effect—it's a total collapse. As shown in Figure 1, within just 25 training updates, a model's perplexity on natural language can skyrocket from ~32 to nearly 7,000.

Figure 1: Language forgetting during SID indexing. Qwen3-0.6B+MS MARCO over 2,000 updates. (A) Recall@10; (B) WikiText- 2 PPL (log scale). PPL rises from 31.69 to 6,956 within 25 updates. Stage 1 consists of document-to-SID indexing.

This makes the model useless for interactive systems that need to ask clarifying questions, summarize retrieved results, or engage in follow-up dialogue.

Methods

The authors propose SpeakGR, a dual-objective training framework. It balances two goals:

  1. Retrieval Specialization ($\mathcal{L}_{ret}$): Standard supervised learning to predict the correct SID for a given query.
  2. Speak-Preserving Regularization ($\mathcal{L}_{speak}$): This is the "secret sauce." Inspired by on-policy distillation, the student model generates its own text continuations from a pool of prompts. A frozen "teacher" (the original, un-specialized LLM) then provides the "correct" next-token distribution for those same prefixes. The student is penalized (via forward KL divergence) if its distribution drifts too far from the teacher's.

They further introduce Adaptive SpeakGR, which uses a control loop to dynamically adjust the weight of the preservation loss ($\lambda$). If the model starts drifting too much from its original language behavior, the system automatically cranks up the regularization pressure.

Figure 2: Overview of SpeakGR and its adaptive variant. A shared student learns SID retrieval while preserving language behavior through forward KL from a frozen teacher at student-sampled text prefixes. SpeakGR fixes the preservation weight at λt = 1; Adaptive SpeakGR uses the dashed controller to update λt+1 from observed preservation KL for the next step.

Datasets

  • Retrieval: The models were trained and evaluated on subsets of MS MARCO (319,872 documents) and Natural Questions (NQ) (109,739 documents).
  • Language Preservation: A pool of 4,096 prompts was used, mixing WikiText-2 continuations with instruction-style prompts (summarization, translation, logic, etc.) to ensure the model maintains diverse "speaking" skills.
  • Evaluation: Language drift was measured primarily using WikiText-2 test sets.

Key Insights and Ideas

  • The "Cliff" of Forgetting: Catastrophic forgetting in GR isn't a slow erosion; it's an immediate crash. The model's internal representations for language are highly sensitive to the shift toward SID generation.
  • On-Policy is Essential: By regularizing the model on states it actually visits during generation, SpeakGR keeps the model grounded in its own evolving distribution rather than just replaying static historical data.
  • Retrieval is not a Zero-Sum Game: Interestingly, keeping the "language brain" healthy doesn't necessarily hurt search. In several settings, Adaptive SpeakGR actually achieved better retrieval performance (Recall@10) than the standard, retrieval-only fine-tuning.
  • Internal Representation Drift: Analysis showed that without regularization, the model's internal "residual stream" (the data flowing through its layers) drifts significantly in the middle and later layers. SpeakGR successfully stabilizes these representations.

Key Results

The results, summarized in Table 1, show a massive improvement in language retention:

  • Language Drift: SpeakGR reduced language drift (measured by forward KL) by 81.3–93.8% on MS MARCO and 81.2–85.2% on NQ compared to standard fine-tuning.
  • Perplexity (PPL): While standard fine-tuning caused PPL to explode into the thousands or millions, SpeakGR kept it in a usable range (e.g., ~66-88 for Qwen models).
  • Retrieval Quality: The models maintained effective retrieval. For instance, on MS MARCO with Qwen3-1.7B, Adaptive SpeakGR reached an R@10 of 0.6745, outperforming the SFT-only baseline (0.6696) while being vastly better at "speaking."
Table 1: Retrieval and language-preservation results across three backbones on MS MARCO and NQ. The best and second-best results among adapted models within each dataset–backbone setting are highlighted in green and light green , respectively. AdaSpeakGR denotes Adaptive SpeakGR.

Limitations and Future Work

  • Capability Scope: The study focused on "distributional preservation" (how well the model mimics its original output probabilities). It did not exhaustively test if specific complex capabilities, like advanced mathematical reasoning or niche coding skills, were perfectly preserved.
  • Evaluation Benchmarks: The primary language evaluation relied on WikiText-2. Future work could involve broader, more diverse benchmarks for conversational AI.
  • End-to-End Testing: While the paper proves a model can do both, it doesn't demonstrate a full, multi-turn "retrieve-resolve-generate" pipeline in a single live interaction, which remains the ultimate goal for this line of research.

Summary updated

Abstract

Generative retrieval (GR) enables end-to-end retrieval by generating document semantic identifiers (SIDs). However, retrieval-only fine-tuning can over-specialize pretrained language models to SID prediction, substantially distorting their natural-language distribution and limiting their suitability for interactive systems that must both retrieve documents and generate natural-language responses. We introduce SpeakGR, a dual-objective framework that learns SIDs while preserving language generation. It combines supervised SID learning with speak-preserving regularization: an on-policy distillation objective that aligns the current model with a frozen copy of the original model on student-generated prefixes using forward KL over the original text vocabulary. We further propose Adaptive SpeakGR, which dynamically adjusts the preservation strength based on observed language drift. Compared with SFT-only, SpeakGR reduces WikiText-2 forward KL by 81.3-93.8% on MS MARCO and 81.2-85.2% on Natural Questions (NQ) while retaining effective retrieval across three different LLMs. Adaptive SpeakGR further improves retrieval over SpeakGR in most settings while maintaining substantially lower language drift than SFT-only.
BibTeX
@misc{fu2026can,
  author = {Junchen Fu and Kleomenis Katevas and Vandana Rajan and Sofía Celi and Hamed Haddadi},
  title = {{Can Generative Retrievers Learn Semantic IDs Without Forgetting How to Speak?}},
  year = {2026},
  eprint = {2609.35430},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.35430},
}
Figure