Sandwich attack: Multi-language Mixture Adaptive Attack on LLMs

Bibek Upadhayay, Vahid BehzadanNAACL 2024

TL;DR

This paper introduces the "Sandwich Attack," a novel multi-language prompt injection technique designed to bypass safety mechanisms in Large Language Models (LLMs). The attack exploits the LLMs' potential weaknesses in processing mixed-language inputs, demonstrating that even state-of-the-art models can be tricked into generating harmful content.

What Problem Does This Paper Address?

The reliable deployment of Large Language Models (LLMs) is hindered by the challenge of "jailbreaking" – manipulating LLMs into generating harmful or misaligned responses. While safety training has been employed to mitigate this, adversaries can exploit the imbalanced representation of low-resource languages in pretraining and safety datasets. This paper specifically addresses the vulnerability of LLMs to adversarial attacks in multilingual contexts, where the models may struggle to consistently apply safety protocols when presented with prompts in various languages. The authors highlight a gap in understanding how LLMs handle safety in these mixed-language scenarios, particularly when the adversarial prompt is embedded within a series of non-English questions.

Methods

The core contribution is the "Sandwich Attack," a black-box, multi-language mixture adaptive attack. The attack involves crafting a prompt with five questions, where the adversarial (harmful) question is strategically placed in the middle. The key idea is to exploit an "Attention Blink" phenomenon, where the LLM's focus is diverted by the mixture of languages and tasks, potentially causing it to overlook the harmful nature of the embedded question. The attack requires the LLM to answer each question in the language it is asked, and a custom system prompt is used to enforce this behavior. The authors also explored variations, including deceptive prefixes and suffixes, and the impact of using English versus non-English text in prompts.

Datasets

The paper does not introduce new datasets for training. For evaluation, the authors selected 50 questions from the "Forbidden Question Set" (Shen et al., 2023), categorized into Privacy, Violence, Pornography, Malware, Physical Harm, Hate Speech, Government Decisions, Fraud, Economic Harm, and Illegal Activity. These questions were translated into various low-resource languages using the Google Translation Cloud API.

Key Insights and Ideas

  • Multilingual Weakness: LLMs, despite safety training, exhibit significant vulnerabilities when dealing with mixed-language prompts, especially when the adversarial content is embedded.
  • "Attention Blink" Phenomenon: The mixture of languages and tasks within a prompt can create an "Attention Blink," diminishing the LLM's ability to detect harmful content.
  • English Bias in Safety: Safety mechanisms appear to rely more heavily on English text, suggesting that non-English prompts might be less scrutinized.
  • LLMs as Evaluators: Using LLMs (like GPT-4) for evaluating responses, even after translation, can provide insights but also highlights the models' own limitations in recognizing harm across languages.
  • Token Limit Impact: The token limit of LLMs can influence the effectiveness of attacks, with longer prompts potentially leading to safer responses or, conversely, more successful jailbreaks depending on the attack strategy.

Key Results

The Sandwich Attack was experimentally evaluated on five SOTA models: Bard, Gemini Pro, LLaMA-2-70B-Chat, GPT-3.5-Turbo, GPT-4, and Claude-3-OPUS. The attack achieved an overall success rate exceeding 50%, with models producing safe responses only 38% of the time. The paper details specific attack strategies and observed behaviors for each model, including Gemini Pro and LLaMA-2 changing adversarial questions and Bard providing illustrations for creating nuclear weapons. The evaluation also showed that LLMs often fail to self-evaluate harmful content in multi-language settings, and GPT-4's ability to identify harm is compromised without translation.

Limitations and Future Work

The authors acknowledge that their study is based on preliminary experiments and non-rigorous approaches in some instances. They hypothesize that the failure of safety mechanisms in multilingual contexts is due to competing objectives, mismatched generalization, lack of multilingual red-teaming, and insufficient utilization of all supported languages in safety training. Future work should focus on a more detailed study of LLM behavior in these scenarios to identify the root cause of jailbreaks and develop robust mitigation strategies. The paper also suggests that further research is needed to enhance LLMs' linguistic versatility and cognitive flexibility in multitasking within multilingual contexts.

Summary updated

BibTeX
@inproceedings{upadhayay2024sandwich,
  author = {Bibek Upadhayay and Vahid Behzadan},
  title = {{Sandwich attack: Multi-language Mixture Adaptive Attack on LLMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure