Tongue-Tied: Breaking LLMs Safety Through New Language Learning

Bibek Upadhayay, Vahid BehzadanNAACL 2025

TL;DR

This paper reveals a critical vulnerability in Large Language Models (LLMs): their safety mechanisms, primarily trained on English, can be easily bypassed by fine-tuning them with new, low-resource languages, even with harmless data. This "language-dependent safety" means LLMs can be jailbroken by simply teaching them a new linguistic system, highlighting a significant gap in current LLM safety training.

What Problem Does This Paper Address?

The rapid deployment of LLMs has brought to light significant safety concerns, particularly their susceptibility to adversarial attacks that exploit their fine-tuning processes. While many attacks focus on malicious prompts or datasets, this paper investigates a more subtle yet potent threat: the impact of introducing new languages on LLM safety. Existing research suggests that multilingual LLMs can exhibit cross-lingual safety failures, but this work specifically probes whether fine-tuning with novel, low-resource languages, even without explicitly harmful content, can systematically compromise an LLM's safety alignment. The core problem is understanding if LLM safety is truly universal or if it's heavily anchored to the languages it was primarily trained on, and if so, how this can be exploited.

Methods

The researchers employed a novel approach by creating synthetic low-resource languages and using them for LoRA (Low-Rank Adaptation) fine-tuning. They introduced Newari, a low-resource language from Nepal, and two modified versions: "Reverse Newari" and a "Devanagari Symbol" mapping. These synthetic languages were used to fine-tune Llama-3-7B-Instruct models. To understand the underlying mechanisms, they utilized a "Logit Lens" approach to analyze internal token probabilities across model layers, specifically examining how English and the new language tokens compete and influence safety decisions. They also explored transferability by testing attacks on black-box models like GPT-4o and Claude-3.5-Sonnet, using both fine-tuning and in-context learning (ICL) techniques.

Datasets

The primary dataset used for fine-tuning was the 52K-Alpaca Dataset, which was translated into Newari. To investigate the impact of translation, they created additional datasets with English-to-Newari (en↔nl) translation pairs. For evaluating jailbreak success, they used a translated version of the JailbreakBench dataset and the modified Forbidden Question dataset.

Key Insights and Ideas

  • Language-Dependent Safety: LLM safety is not a universal property but is heavily influenced by the languages it has been trained on. Introducing new languages can create a "language-dependent safety" vulnerability.
  • Low-Resource Languages as Attack Vectors: Even harmless data in a low-resource language can be sufficient to jailbreak an LLM, demonstrating that safety training is particularly ineffective in these linguistic contexts.
  • The "Pivoting" Mechanism: In later layers of the LLM, a shift from English tokens to new language tokens correlates with unsafe generations, suggesting a "pivot" where English acts as a safety anchor and the new language facilitates the jailbreak.
  • Translation Pairs Degrade Safety: Surprisingly, fine-tuning with English-to-new language translation pairs, while potentially improving multilingual capabilities, can paradoxically degrade safety alignment, especially when attacked in English.
  • Transferability to Black-Box Models: The attack is not limited to open-source models; it can be transferred to proprietary black-box LLMs like GPT-4o and Claude-3.5-Sonnet, even through few-shot learning or in-context learning without full fine-tuning.

Key Results

  • Attack Success Rate (ASR): Fine-tuning with Newari led to ASRs up to 67% when queried in the new language, compared to ≤ 44% in English.
  • Safety Degradation: The number of unsafe responses significantly increased (p<0.05) when models were attacked in the new language compared to English.
  • GPT-4o Jailbreak: GPT-4o was successfully jailbroken through fine-tuning with just 4,000 data points in a new language.
  • Claude-3.5-Sonnet Jailbreak: Claude-3.5-Sonnet could be compelled to generate unsafe responses in a new language via in-context learning, achieving a 57% ASR.
  • Translation Impact: Models trained with en↔nl translation pairs showed higher unsafe response counts and quality when attacked in English, indicating a trade-off between multilingual capability and safety.

Limitations and Future Work

The authors acknowledge that their analysis of layer-wise token probabilities was based on the first 100 generated tokens, which might not fully capture all model behaviors. They also note that while translation pairs improved English preferences, they degraded safety, suggesting a need for new fine-tuning strategies.

Future work should focus on developing robust multilingual safety training approaches that generalize across languages, including synthetic ones. Investigating how safety information is encoded in model parameters across languages, moving beyond English-centric safety, is crucial. Further research is needed to understand the precise relationship between translation training and safety degradation, aiming to maintain safety alignment while expanding language capabilities.

Summary updated

BibTeX
@inproceedings{upadhayay2025tongue,
  author = {Bibek Upadhayay and Vahid Behzadan},
  title = {{Tongue-Tied: Breaking LLMs Safety Through New Language Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure