Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks

Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, Monojit ChoudhuryCOLING 2024

TL;DR

Large Language Models (LLMs) are susceptible to "jailbreaks" – malicious prompts designed to make them disobey their intended instructions and generate harmful or unintended outputs. This paper formalizes these attacks, proposes a taxonomy of jailbreak techniques and intents, and empirically analyzes their effectiveness across various LLMs, revealing significant vulnerabilities and challenges in detection.

What Problem Does This Paper Address?

LLMs have demonstrated remarkable capabilities in understanding and executing natural language instructions. However, this flexibility also opens them up to "prompt injection attacks," commonly known as "jailbreaks." These attacks allow malicious users to bypass safety guardrails and content policies, leading to the generation of offensive content, privacy breaches, or security vulnerabilities. While the phenomenon is widely observed, there's a lack of formal understanding, a comprehensive taxonomy, and systematic analysis of these attacks and their detection. This paper aims to bridge this gap by providing a structured framework for understanding, analyzing, and detecting LLM jailbreaks.

Methods

The authors propose a formal definition of a "jailbreak" as a deliberate misalignment of an LLM's output with the developer's intent. They introduce a taxonomy that categorizes jailbreaks based on two orthogonal dimensions:

  • Jailbreak Technique: How the malicious input is constructed, categorized into:
  • Orthographic: Manipulating script, encoding (e.g., LeetSpeak, Base64).
  • Lexical: Using specific words or phrases.
  • Morpho-Syntactic: Exploiting word sequence structure (e.g., Text Completion as Instruction - TCINS).
  • Semantic: Exploiting meaning, including direct instructions (INSTR) and few-shot examples (FSH).
  • Pragmatic: Exploiting contextual meaning beyond literal semantics, including instruction repetition (IR), indirect task deflection (ITD), and cognitive hacking (COG).
  • Jailbreak Intent: The intended harm, categorized into:
  • Information Leakage: Revealing sensitive or proprietary information.
  • Misaligned Content Generation: Producing toxic, hateful, or harmful content.
  • Performance Degradation: Reducing the model's accuracy or availability.

They also define "property tests" (automated checks on output alignment) and "intent tests" (evaluating if the attacker's intent was satisfied). For evaluation, they use both programmatic property tests and GPT-4 as a "property-tester," alongside manual human evaluations.

Datasets

The paper utilizes several datasets:

  • Prompts: Sourced from OpenAI, Promptsource, and various academic publications.
  • Base-Inputs: 100 base-inputs sampled per task from existing datasets. For code generation, they used meta-prompting GPT-3.5 to generate queries.
  • Jailbreaks: A curated set of 55 jailbreaks across four tasks (Translation, Text Classification, Summarization, Code Generation), derived from online sources and academic work.
  • User Inputs: 100 user inputs sampled per task for 37 of the jailbreaks, resulting in approximately 3700 potential jailbreaks fed to each model.
  • YouTube Comments: Scraped comments related to "Attack Prompts," "Jailbreak gpt," "dan gpt," and "code injection in gpt" to analyze the prevalence of jailbreak-related terms.
  • Reddit Posts: Scraped posts from r/ChatGPT and r/OpenAI to analyze the frequency of terms like "DAN," "JIM," "Jailbreak," "Prompt Injection," and "Prompt Leakage."

No new benchmarks were explicitly introduced, but the paper's curated dataset of jailbreaks and prompts serves as a valuable resource for future research.

Key Insights and Ideas

  • Jailbreaks are a fundamental vulnerability: The paper demonstrates that LLMs, despite their advanced capabilities, are systematically vulnerable to prompt-based attacks that can override their intended behavior.
  • Taxonomy provides structure: The proposed taxonomy of techniques and intents offers a much-needed structured way to understand and discuss the diverse landscape of jailbreaks, moving beyond ad-hoc categorizations.
  • Cognitive Hacking is highly effective: The "Cognitive Hacking" (COG) technique, which involves creating a fictional scenario or persona for the LLM, emerged as the most successful jailbreak type across many models. This highlights the LLM's susceptibility to role-playing and contextual manipulation.
  • Instruction tuning can increase vulnerability: Models that are heavily instruction-tuned (like GPT-3.5 Turbo and code-davinci-002) appear to be more susceptible to certain jailbreaks, particularly those involving misaligned content generation.
  • Detection is challenging and brittle: The study reveals a poor agreement between automated property tests and even GPT-4-based evaluations, indicating that current detection methods are not robust. The "jailbreak paradox" is highlighted, where detection methods themselves can be vulnerable to jailbreaks.
  • Model size and architecture matter, but not always predictably: While larger models sometimes show more robustness, instruction tuning and specific architectures can introduce new vulnerabilities.

Key Results

  • The paper presents confusion matrices and success rates for various LLMs (OPT, BLOOM, FLAN-T5-XXL, GPT-3.5 Turbo, code-davinci-002, etc.) across different jailbreak types and intents.
  • Performance Degradation was the most common intent targeted, likely due to its relative ease of execution.
  • Cognitive Hacking (COG) was found to be the most successful jailbreak technique.
  • GPT-4 was occasionally jailbroken itself during the evaluation process, underscoring the difficulty of robust detection.
  • Manual evaluations showed a significant disagreement with automated metrics, suggesting the limitations of current programmatic tests.
  • Analysis of Reddit and YouTube data indicated that "Jailbreak" is a more popular term than "Prompt Injection" among users.

Limitations and Future Work

  • Limited scope of property tests: The authors acknowledge that their programmatic property tests are rudimentary and can be bypassed by sophisticated attacks.
  • Jailbreak paradox: The difficulty in creating robust detection methods that are themselves resistant to jailbreaks is a significant challenge.
  • Human evaluation subjectivity: Disagreements among human annotators highlight the subjective nature of evaluating jailbreak success and intent satisfaction.
  • Language bias: The current study focuses on English jailbreaks, while real-world attacks may not be language-restricted.
  • Future work: The authors suggest further research into more sophisticated intent tests, exploring prompt-level mitigation strategies, and developing a deeper understanding of how jailbreaks affect LLM embeddings. They also emphasize the need for user-level understanding of misalignment and explainability for end-users.

Summary updated

BibTeX
@inproceedings{rao2024tricking,
  author = {Abhinav Rao and Sachin Vashistha and Atharva Naik and Somak Aditya and Monojit Choudhury},
  title = {{Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure