Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Models

Zhenhua Wang, Wei Xie, Kai Chen, Baosheng Wang, Zhiwen Gui, Enze Wang

TL;DR

This paper introduces "self-deception," a novel attack method that bypasses LLM semantic firewalls by making the LLM "forget" its safety rules through a series of carefully crafted prompts. The authors demonstrate this attack's effectiveness on GPT-3.5-Turbo and GPT-4, achieving high jailbreak success rates and highlighting the need for more robust defenses against such sophisticated prompt manipulation.

What Problem Does This Paper Address?

Large Language Models (LLMs) like ChatGPT have become incredibly powerful, but this power also makes them susceptible to generating harmful content. To combat this, developers have implemented "semantic firewalls" – defenses that analyze prompts for malicious intent based on meaning, not just keywords. However, these defenses aren't perfect. Attackers have found ways to "jailbreak" LLMs, tricking them into ignoring their safety guidelines. The core problem this paper addresses is the lack of a clear understanding of how these semantic-level attacks and defenses work, and the need for a systematic approach to both attacking and defending against them. The authors aim to shed light on the principles behind these interactions and propose a new, automated method for jailbreaking LLMs.

Methods

The paper introduces a novel attack called "self-deception." This method is inspired by traditional network security techniques like reverse tunnels, where an attacker establishes an outbound connection to bypass inbound security. In the LLM context, the "self-deception" attack works by guiding the LLM to generate prompts that facilitate the jailbreak itself.

Here's how it generally works:

  1. Initial Prompt: The attacker provides a prompt with minimal malicious semantics, ensuring it passes the semantic firewall. This prompt is designed to subtly steer the LLM towards generating content that sets up a more problematic scenario.
  2. LLM Generates "Settings": The LLM, following the initial prompt, generates descriptive text (e.g., describing a violent world or a villain's history). This generated text contains the "malicious settings" that the LLM is supposed to avoid.
  3. Iterative Escalation: The attacker then uses the LLM's own output from the previous step to construct a new prompt. This new prompt is still designed to be just below the semantic firewall's detection threshold, but it leverages the previously generated "malicious settings" to encourage the LLM to produce even more violative content.
  4. "Forgetting" Defense: By repeatedly guiding the LLM to generate content that implicitly leads to more problematic outputs, the attacker effectively "hypnotizes" the LLM into forgetting its content defense rules, allowing it to generate increasingly inappropriate responses.

The authors also propose three conceptual approaches for implementing a semantic firewall:

  • Hidden Prompt Prefix: Pre-setting rules at the beginning of the LLM's context.
  • Independent Review Model: Using a separate classifier to check prompts before they reach the LLM.
  • Fine-Tuning the LLM: Training the LLM itself to have inherent defense capabilities.

Datasets

The paper does not explicitly mention pre-existing datasets for training. Instead, it focuses on generating its own experimental data. The authors created 2,520 attack payloads by combining:

  • Six languages: English, Russian, French, Spanish, Chinese, and Arabic.
  • Seven virtual scenarios: Game, teleplay, opera, script, novel, comic, and movie.
  • Three common violation types: Violence, hate, and pornography.

This generated dataset was used to evaluate the proposed "self-deception" attack against GPT-3.5-Turbo and GPT-4.

Key Insights and Ideas

  • "Self-Deception" as a Novel Attack Vector: The core innovation is the idea of making the LLM complicit in its own jailbreak by having it generate the "malicious settings" that enable the attack. This is a significant departure from previous methods that relied solely on carefully crafted external prompts.
  • Semantic Firewall Vulnerability: The research demonstrates that current semantic firewalls, which are primarily input-focused, can be bypassed by exploiting the LLM's generative capabilities. The attack effectively turns the LLM's own output against its safety mechanisms.
  • Iterative Prompting is Key: The "self-deception" attack relies on a multi-step conversational approach, where each step builds upon the LLM's previous output to gradually escalate the violation level. This highlights the dynamic nature of LLM interactions.
  • Cross-Lingual and Cross-Scenario Effectiveness: The attack proved effective across multiple languages and diverse virtual scenarios, suggesting a generalizable vulnerability in LLM defenses.
  • GPT-4's Nuance: While GPT-4 showed a lower rejection rate than GPT-3.5-Turbo, it also exhibited a higher rate of "uncertain" cases, suggesting a more complex internal reasoning or a different approach to handling potentially harmful content.

Key Results

  • High Success Rates: The "self-deception" attack achieved a 86.2% success rate on GPT-3.5-Turbo and a 67% success rate on GPT-4.
  • Low Failure Rates: Correspondingly, the failure rates were very low: 4.7% for GPT-3.5-Turbo and 2.2% for GPT-4.
  • Cross-Lingual Efficacy: The attack was effective across all six tested languages, with English showing the least uncertainty and highest success.
  • Scenario Agility: The attack demonstrated effectiveness across all seven virtual scenarios, though the "game" scenario showed a slightly lower success rate.
  • Comparison to Existing Methods: The proposed "self-deception" method significantly outperformed existing jailbreak prompts, achieving an average success rate of 47% (GPT-3.5-Turbo) and 38.4% (GPT-4) compared to the paper's method's 86.2% and 67% respectively.

Limitations and Future Work

  • Limited Model Scope: The experiments were primarily conducted on OpenAI's GPT models (GPT-3.5-Turbo and GPT-4). The authors acknowledge that this might not fully represent the behavior of LLMs from other developers.
  • Language Bias: While six languages were tested, the authors note that limitations in time and cost prevented testing a wider range of major world languages. Potential semantic biases introduced during translation are also a concern.
  • Empirical Speculation: Some interpretations of the results are based on empirical speculation, and other objective factors might contribute to the findings.
  • Future Work: The authors emphasize that prompts are the primary interface and attack surface for LLMs. Future research should focus on:
  • Exploring other threats to prompts beyond injection attacks.
  • Designing and implementing better semantic firewalls that balance security and functionality.
  • Understanding the mechanism behind how subsequent prompts can override or fail to override previous ones.
  • Developing more robust defenses against "self-deception" and similar attacks.

Summary updated

Abstract

Large language models (LLMs), such as ChatGPT, have emerged with astonishing capabilities approaching artificial general intelligence. While providing convenience for various societal needs, LLMs have also lowered the cost of generating harmful content. Consequently, LLM developers have deployed semantic-level defenses to recognize and reject prompts that may lead to inappropriate content. Unfortunately, these defenses are not foolproof, and some attackers have crafted "jailbreak" prompts that temporarily hypnotize the LLM into forgetting content defense rules and answering any improper questions. To date, there is no clear explanation of the principles behind these semantic-level attacks and defenses in both industry and academia. This paper investigates the LLM jailbreak problem and proposes an automatic jailbreak method for the first time. We propose the concept of a semantic firewall and provide three technical implementation approaches. Inspired by the attack that penetrates traditional firewalls through reverse tunnels, we introduce a "self-deception" attack that can bypass the semantic firewall by inducing LLM to generate prompts that facilitate jailbreak. We generated a total of 2,520 attack payloads in six languages (English, Russian, French, Spanish, Chinese, and Arabic) across seven virtual scenarios, targeting the three most common types of violations: violence, hate, and pornography. The experiment was conducted on two models, namely the GPT-3.5-Turbo and GPT-4. The success rates on the two models were 86.2% and 67%, while the failure rates were 4.7% and 2.2%, respectively. This highlighted the effectiveness of the proposed attack method. All experimental code and raw data will be released as open-source to inspire future research. We believe that manipulating AI behavior through carefully crafted prompts will become an important research direction in the future.
BibTeX
@misc{wang2023self,
  author = {Zhenhua Wang and Wei Xie and Kai Chen and Baosheng Wang and Zhiwen Gui and Enze Wang},
  title = {{Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Models}},
  year = {2023},
  eprint = {2308.11521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.11521},
}
Figure