When we ask a Large Language Model (LLM) to "forget" a piece of sensitive information—such as a copyrighted book or a private address—we usually do so in English. However, modern LLMs are inherently multilingual, mapping diverse languages into a shared semantic space. A growing body of research reveals a critical "polyglot persistence" problem: a model that has successfully unlearned a fact in English often retains that exact same knowledge when prompted in French, Chinese, or Arabic.
Achieving true unlearning requires moving beyond surface-level token suppression toward the erasure of language-agnostic conceptual representations.
The Lexical Suppression Trap
The primary reason unlearning fails across languages is that many current methods do not actually delete knowledge; they merely "obfuscate" it. As Guangzhi Sun and colleagues argue in Unlearning vs. Obfuscation: Are We Truly Removing Knowledge?, many techniques simply teach the model to refuse specific queries or flatten the predictive distribution for certain English tokens.
Nianwen Si et al. note in Knowledge Unlearning for LLMs: Tasks, Methods, and Challenges that this creates a "cross-lingual leakage" effect. Because the model's internal weights still contain the underlying concept, a prompt in a different language bypasses the English-specific "refusal filter." Jai Doshi demonstrates this vulnerability in Does Unlearning Truly Unlearn?, showing that translating a forbidden question into another language can effectively "jailbreak" the model and recover supposedly deleted information.
Shared Semantics and Language-Specific Neurons
Why does this knowledge persist? The answer lies in how LLMs store information. Research by Minseok Choi and Kyunghyun Min in Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models suggests a dual-storage mechanism:
- Language-Agnostic Representations: Core conceptual facts are often stored in a shared embedding space.
- Language-Specific Neurons: Specific associations are tied to the unique syntax and vocabulary of individual languages.
If an unlearning algorithm only updates the language-specific neurons for English, the language-agnostic core remains untouched. Chaoyi Xiang et al. further clarify this in Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility, finding that unlearning is often localized in later decoding layers. This superficiality allows the model to "remember" the fact in its early layers and only suppress it at the very end of the generation process—a suppression that fails to generalize when the output tokens belong to a different language.
Evaluating the Multilingual Security Gap
To address these failures, researchers are developing more rigorous benchmarks. Saemi Moon and the team behind the Holistic Unlearning Benchmark (HUB) emphasize "multilingual robustness" as a core metric for success. They argue that an unlearning intervention is only successful if it provides "pinpoint-ness"—erasing the concept without damaging general intelligence—across the model's entire linguistic repertoire.
Similarly, Aengus Lynch and Stephen Casper propose Eight Methods to Evaluate Robust Unlearning in LLMs, which include cross-lingual probing to see if "forgotten" knowledge (like the world of Harry Potter) can still be extracted through non-English prompts.
Strategies for Global Forgetting
How can we ensure a model forgets a concept globally? Several promising directions have emerged:
- Stochastic Language Sampling: Minseok Choi et al. demonstrate that unlearning signals can propagate across the multilingual space more effectively if the model is trained on a diverse sample of languages during the unlearning process, rather than just one.
- Targeted Weight Reversal: Instead of fine-tuning on "refusal" data, Harry J. Davies and Danilo Mandic propose Targeted Angular Reversal of Weights (TARS) in Targeted Angular Reversal of Weights (TARS) for Knowledge Removal in Large Language Models. By identifying and reversing the specific weight vectors in feed-forward layers that align with a concept, they aim for permanent, multilingual removal.
- Causal Intervention: Yang Zhang and Shiyu Chang in Revisiting Who's Harry Potter use causal lenses to ensure that unlearning an entity (like a person) transfers across languages, indicating that the intervention has reached the conceptual level rather than just the lexical one.
The persistence of knowledge across languages serves as a "litmus test" for unlearning. As researchers like Rohit Gandikota and David Bau continue to refine methods like Erasure of Language Memory (ELM), the goal remains clear: to ensure that when a model is told to forget, it forgets the idea, not just the words.