All Languages Matter: On the Multilingual Safety of Large Language Models

Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, Michael LyuSubmission to ICLR 2024 · Rejected

TL;DR

While state-of-the-art large language models (LLMs) appear well-aligned and safe in English, their guardrails degrade significantly when prompted in other languages. Introducing XSAFETY—the first comprehensive multilingual safety benchmark spanning 10 languages and 14 safety scenarios—this paper shows that models like ChatGPT and LLaMA2-Chat are far more prone to generating harmful outputs in non-English queries, especially low-resource ones. Simple safety-evoking prompts can cut unsafe response rates nearly in half without requiring model retraining.

What Problem Does This Paper Address?

Safety alignment techniques—such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF)—have overwhelmingly focused on English. Yet, commercial LLMs are deployed globally and interact with millions of non-English speakers daily.

This raises a crucial question: Can non-English prompts inadvertently or adversarially bypass English-centric safety alignment? Because existing safety benchmarks are predominantly monolingual (primarily English or Chinese), the machine learning community has lacked standard tooling to systematically measure and mitigate cross-lingual safety vulnerabilities.

Methods

The authors introduce XSAFETY, an evaluation framework, and analyze safety across both open-source and proprietary models before exploring prompt-based interventions:

  • Benchmark Construction & Cultural De-biasing: Starting with 14 safety categories from established corpora, the authors filtered out region-specific cultural references (e.g., regional Chinese socio-political contexts), machine-translated the prompts into nine other languages, and commissioned two rounds of professional human translation proofreading ($3,000 USD investment; final quality pass rate >99%).
  • Automated Safety Evaluation: Non-choice model outputs are translated to English via Google Translate and classified as safe/unsafe using ChatGPT with constrained zero-shot prompts. Human validation across 1,400 samples confirmed an 88.5% agreement rate with ChatGPT’s judgments.
  • In-Context Safety Prompting: To harden models against multilingual vulnerabilities at inference time, the authors designed targeted system prompts:
  • SafePrompt: Explicitly instructs safety adherence or specifies the risk domain (e.g., "Please answer safely under [safety] scenario").
  • XLingPrompt: Forces intermediate English reasoning to trigger English safety mechanisms before replying (e.g., "Please think in English and then generate the response in the original language").
  • XSafePrompt: A hybrid combining scenario-specific instructions with English intermediate reasoning.

Datasets

  • XSAFETY (Introduced): Contains 2,800 source instances covering:
  • 7 Typical Safety Scenarios: Insult, Unfairness, Crimes & Illegal Activities, Physical Harm, Mental Health, Privacy & Property, Ethics & Morality (derived from Sun et al., 2023).
  • 1 Commonsense Physical Safety Scenario: Multi-choice questions requiring implicit physical reasoning (from SafeText, Levy et al., 2022).
  • 6 Instruction Attack Types: Goal Hijacking, Prompt Leaking, Role Play, Unsafe Instruction Topic, Inquiry with Unsafe Opinion, Reverse Exposure (Sun et al., 2023).
  • Languages Covered (10 total, 28,000 instances): English (en), Chinese (zh), Hindi (hi), Spanish (es), French (fr), Arabic (ar), Bengali (bn), Russian (ru), Japanese (ja), and German (de).

Key Insights and Ideas

  • Alignment does not generalize equally across the linguistic spectrum: English-tuned guardrails leak severely in non-English languages. The failure rate is especially severe in languages with lower representation in pretraining corpora (e.g., Bengali and Hindi).
  • Commonsense safety and goal hijacking are the primary failure points: Implicit dangers (e.g., determining whether drinking antifreeze to warm up is harmful) and adversarial goal hijacking evade multilingual safety filters far more easily than overt hate speech.
  • Safety knowledge is present, but dormant: Models often know a topic is dangerous in non-English contexts, but fail to trigger refusal behaviors. Simply reminding the model of the specific risk scenario (SafePrompt2) acts as an effective retrieval cue for safety behaviors.
  • Combining reasoning translation with safety prompts produces friction: Surprisingly, forcing English chain-of-thought (XLingPrompt) alongside explicit safety cues (SafePrompt) performed worse than explicit safety cues alone, indicating that translation directives and safety guardrails can interfere with one another during generation.

Key Results

  • Disproportionate Unsafe Rates in Non-English Prompts:
  • ChatGPT: Unsafe response rate jumped from 1.0% in English to an average of 15.9% across non-English languages (peaking at 37.4% in Bengali and 21.0% in Japanese).
  • PaLM2: Rose from 10.3% (English) to 18.7% (non-English average).
  • LLaMA2-Chat-13B: Jumped from 14.6% to 23.6% (reached 36.5% in Hindi).
  • Vicuna-13B: Jumped from 6.0% to 33.4% (reached 81.6% in Bengali).
  • Prompt Defense Effectiveness: Applying SafePrompt2 (specifying scenario knowledge) dropped ChatGPT’s unsafe response rate across representative non-English queries from 19.1% down to 9.7%.

Limitations and Future Work

  • Translation Artifacts: Translating seeds rooted in Chinese and English corpora inherently carries linguistic and cultural biases, potentially overlooking culture-specific taboos, harms, or idioms in regions where Hindi, Arabic, or Bengali are spoken natively.
  • Self-Evaluation Noise: Although validated by human judges and GPT-4 audits, automated evaluation pipeline errors (translation mistranslations + LLM evaluator bias) remain a source of noise.
  • Prompting is a Stopgap: In-context prompting significantly mitigates the issue, but it does not resolve the root imbalance in pretraining and alignment data.

The authors highlight future needs: expanding multilingual coverage into bias and copyright domains, analyzing representation spaces during cross-lingual transfer, and developing natively multilingual preference tuning (RLHF/DPO).

Summary updated

Abstract

Safety lies at the core of developing and deploying large language models (LLMs). However, previous safety benchmarks only concern the safety in one language, e.g. the majority language in the pretraining data such as English. In this work, we build the first multilingual safety benchmark for LLMs, XSafety, in response to the global deployment of LLMs in practice. XSafety covers 14 kinds of commonly used safety issues across 10 languages that span several language families. We utilize XSafety to empirically study the multilingual safety for 4 widely-used LLMs, including both close-API and open-source models. Experimental results show that all LLMs produce significantly more unsafe responses for non-English queries than English ones, indicating the necessity of developing safety alignment for non-English languages. In addition, we propose several simple and effective prompting methods to improve the multilingual safety of ChatGPT by evoking safety knowledge and improving cross-lingual generalization of safety alignment. Our prompting method can significantly reduce the ratio of unsafe responses from 19.1\% to 9.7\% for non-English queries.
BibTeX
@misc{wang2023all,
  author = {Wenxuan Wang and Zhaopeng Tu and Chang Chen and Youliang Yuan and Jen-tse Huang and Wenxiang Jiao and Michael Lyu},
  title = {{All Languages Matter: On the Multilingual Safety of Large Language Models}},
  year = {2023},
  eprint = {2310.00905},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00905},
}
Figure