This summary explores the paper "Can Generative Retrievers Learn Semantic IDs Without Forgetting How to Speak?", which tackles a critical hurdle in building unified AI agents: the tendency of models to lose their conversational abilities when trained to act as search engines.
TL;DR
Generative retrieval (GR) turns LLMs into powerful search engines by teaching them to generate document "Semantic IDs," but this specialized training causes the models to rapidly "forget" how to speak natural language. The authors introduce SpeakGR, a framework that uses on-policy distillation to preserve a model's original language distribution while it learns to retrieve, enabling a single model to both find documents and chat about them effectively.
What Problem Does This Paper Address?
Generative Retrieval is a paradigm shift where a neural model generates a document's identifier (SID) directly from a query, effectively internalizing the search index. While powerful, standard fine-tuning for this task is "narrow." The model learns to map queries to a tiny vocabulary of SIDs, often at the expense of its massive pre-trained knowledge of human language.
The authors demonstrate that this "catastrophic forgetting" is not just a minor side effect—it's a total collapse. As shown in Figure 1, within just 25 training updates, a model's perplexity on natural language can skyrocket from ~32 to nearly 7,000.
This makes the model useless for interactive systems that need to ask clarifying questions, summarize retrieved results, or engage in follow-up dialogue.
Methods
The authors propose SpeakGR, a dual-objective training framework. It balances two goals:
- Retrieval Specialization ($\mathcal{L}_{ret}$): Standard supervised learning to predict the correct SID for a given query.
- Speak-Preserving Regularization ($\mathcal{L}_{speak}$): This is the "secret sauce." Inspired by on-policy distillation, the student model generates its own text continuations from a pool of prompts. A frozen "teacher" (the original, un-specialized LLM) then provides the "correct" next-token distribution for those same prefixes. The student is penalized (via forward KL divergence) if its distribution drifts too far from the teacher's.
They further introduce Adaptive SpeakGR, which uses a control loop to dynamically adjust the weight of the preservation loss ($\lambda$). If the model starts drifting too much from its original language behavior, the system automatically cranks up the regularization pressure.
Datasets
- Retrieval: The models were trained and evaluated on subsets of MS MARCO (319,872 documents) and Natural Questions (NQ) (109,739 documents).
- Language Preservation: A pool of 4,096 prompts was used, mixing WikiText-2 continuations with instruction-style prompts (summarization, translation, logic, etc.) to ensure the model maintains diverse "speaking" skills.
- Evaluation: Language drift was measured primarily using WikiText-2 test sets.
Key Insights and Ideas
- The "Cliff" of Forgetting: Catastrophic forgetting in GR isn't a slow erosion; it's an immediate crash. The model's internal representations for language are highly sensitive to the shift toward SID generation.
- On-Policy is Essential: By regularizing the model on states it actually visits during generation, SpeakGR keeps the model grounded in its own evolving distribution rather than just replaying static historical data.
- Retrieval is not a Zero-Sum Game: Interestingly, keeping the "language brain" healthy doesn't necessarily hurt search. In several settings, Adaptive SpeakGR actually achieved better retrieval performance (Recall@10) than the standard, retrieval-only fine-tuning.
- Internal Representation Drift: Analysis showed that without regularization, the model's internal "residual stream" (the data flowing through its layers) drifts significantly in the middle and later layers. SpeakGR successfully stabilizes these representations.
Key Results
The results, summarized in Table 1, show a massive improvement in language retention:
- Language Drift: SpeakGR reduced language drift (measured by forward KL) by 81.3–93.8% on MS MARCO and 81.2–85.2% on NQ compared to standard fine-tuning.
- Perplexity (PPL): While standard fine-tuning caused PPL to explode into the thousands or millions, SpeakGR kept it in a usable range (e.g., ~66-88 for Qwen models).
- Retrieval Quality: The models maintained effective retrieval. For instance, on MS MARCO with Qwen3-1.7B, Adaptive SpeakGR reached an R@10 of 0.6745, outperforming the SFT-only baseline (0.6696) while being vastly better at "speaking."
Limitations and Future Work
- Capability Scope: The study focused on "distributional preservation" (how well the model mimics its original output probabilities). It did not exhaustively test if specific complex capabilities, like advanced mathematical reasoning or niche coding skills, were perfectly preserved.
- Evaluation Benchmarks: The primary language evaluation relied on WikiText-2. Future work could involve broader, more diverse benchmarks for conversational AI.
- End-to-End Testing: While the paper proves a model can do both, it doesn't demonstrate a full, multi-turn "retrieve-resolve-generate" pipeline in a single live interaction, which remains the ultimate goal for this line of research.
Summary updated