Abstract
This paper investigates the propagation of harmful information in multilingual large language models (LLMs) and evaluates the efficacy of various unlearning methods. We demonstrate that fake information, regardless of the language it is in, once introduced into these models through training data, can spread across different languages, compromising the integrity and reliability of the generated content. Our findings reveal that standard unlearning techniques, which typically focus on English data, are insufficient in mitigating the spread of harmful content in multilingual contexts and could inadvertently reinforce harmful content across languages. We show that only by addressing harmful responses in both English and the original language of the harmful data can we effectively eliminate generations for all languages. This underscores the critical need for comprehensive unlearning strategies that consider the multilingual nature of modern LLMs to enhance their safety and reliability across diverse linguistic landscapes.
Introduction
As Large Language Models (LLMs) scale, they are increasingly trained on massive, often uncurated multilingual datasets. While English data is heavily filtered, misinformation in other languages frequently slips through. This creates a "multilingual-to-multilingual" safety risk where misinformation learned in one language leaks into others, regardless of linguistic similarity. This study investigates how "fake facts" propagate across languages and evaluates whether standard English-centric safety "unlearning" is sufficient for multilingual models.
Propagation of Misinformation
The researchers found that misinformation is largely language-agnostic. If a model learns a "fake fact" in a low-resource language like Hausa, it can still be triggered when queried in English or Urdu. This suggests that LLMs map information to a shared semantic space, making linguistic barriers porous for harmful content.
As shown in the heatmaps above, while the highest occurrence of fake information ($O_{\mathcal{F}}$) typically happens when the query language matches the source language of the fake data (the diagonal), the spread is significant across all language pairs. Even for low-resource languages, the model demonstrates a concerning ability to transfer misinformation to high-resource languages like English.
Unlearning Strategies and Mitigation
To address this, the authors employed a Retain-Forget unlearning objective, using gradient descent to preserve accurate information and gradient ascent to erase misinformation. They compared four configurations:
- English-only: Erasing fake info using only English translations.
- Same-language: Erasing info in the original source language.
- Multilingual: Erasing info across 20 different languages.
- Combined (Proposed): A 50/50 mix of English and the original source language.
Key Results
The study revealed several critical insights:
- English-Centric Failure: English-only unlearning is effective (94% reduction) only if the source of misinformation was English. For non-English sources, it only reduced fake generations by about 55%, leaving a significant residue of misinformation.
- The Multilingual Paradox: Attempting to unlearn across many random languages (Multilingual Unlearning) can backfire, sometimes reinforcing the fake content. This suggests the model may find "shortcuts" to hide information rather than deleting it.
- The Combined Solution: The proposed "Combined" approach (English + Original Language) was the only method that effectively eliminated nearly all fake responses across all languages without degrading the model's general performance.
Conclusion
The findings underscore that safety in multilingual LLMs cannot be achieved through English-centric methods alone. Effective unlearning requires a strategy that targets both the global bridge language (English) and the specific language of contamination. Future work should explore these dynamics in web-scale datasets and apply these findings to other safety domains like bias and toxicity.
Summary updated