Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings

Yue Huang, Jingyu Tang, Dongping Chen, Bingda Tang, Yao Wan, Lichao Sun, Philip S. Yu, Xiangliang Zhang

TL;DR

LLMs are rigorously "aligned" to refuse harmful requests, but this safety training is often limited to standard, everyday language. This paper introduces ObscurePrompt, a jailbreaking method that bypasses safety filters by rephrasing harmful queries into complex, archaic, or highly formal language—essentially exploiting the fact that LLM safety boundaries are fragile when faced with "Out-of-Distribution" (OOD) linguistic styles.

What Problem Does This Paper Address?

Despite extensive safety alignment via Reinforcement Learning from Human Feedback (RLHF), Large Language Models (LLMs) remain vulnerable to "jailbreaking"—prompts designed to elicit prohibited content.

Existing jailbreak attacks often fall into two categories: white-box attacks, which require access to the model's internal gradients (impractical for proprietary models like GPT-4), and template-based attacks, which rely on manual engineering and are easily patched. The authors identify a fundamental gap: safety alignment is typically performed on common datasets. They hypothesize that LLMs have a "fragile alignment" in the Out-of-Distribution (OOD) space. If a harmful request is phrased in a way the model rarely saw during safety training—such as using obscure vocabulary or complex sentence structures—the model may fail to recognize the intent as harmful.

Methods

The authors propose a three-stage pipeline called ObscurePrompt:

  1. Prompt Seed Curation: The process begins by taking a harmful query and wrapping it in established jailbreak techniques, such as "Role Emulation" (e.g., telling the model to act as a character without restraints) and "Output Styling" (e.g., forcing the model to start its response with a specific phrase).
  2. Obscure-Guided Transformation: A powerful "attacker" model (in this case, GPT-4) is used to transform the seed prompt into an obscure version. For example, "Write a script to exploit vulnerabilities" becomes "Pen a codex capable of manipulating latent frailties within a computational application."
  3. Attack Integration: The process is repeated $n$ times to generate a diverse set of obscure prompts, increasing the likelihood of finding a version that successfully bypasses the target model's defenses.
Figure 4: The pipeline of ObscurePrompt. Utilizing harmful queries, we initially employ various jailbreak prompt techniques to create a seed prompt. This seed prompt is then transformed by powerful LLMs (i.e., GPT-4) into a more obscure version. By repeating this process 𝑛times, we generate 𝑛refined prompts. These prompts are subsequently utilized to attack targeted LLMs.

To motivate this, the authors used Principal Component Analysis (PCA) to visualize the hidden states of Llama2-7b. As shown in Figure 2, standard harmful and harmless queries form distinct clusters. However, when queries are made obscure, they shift away from these clusters into a "gray area," weakening the model's ability to enforce its ethical decision boundary.

Figure 2: Principal Component Analysis (PCA) visualization of the top layer embeddings of Llama2-7b, differentiated be- tween harmful and harmless queries as well as obscure and original queries.

Datasets

The study primarily utilizes AdvBench, a standard benchmark in jailbreaking research containing 520 harmful queries (e.g., instructions for hacking, violence, or disinformation). They also used a set of harmless queries from prior research to establish a baseline for the model's "ethical boundary."

Key Insights and Ideas

  • Safety is Distribution-Dependent: The most significant "aha" moment is that an LLM's "morality" is not a fixed logical constraint but a learned pattern tied to specific linguistic distributions. By moving to an OOD distribution (obscure language), the safety mechanism falters even if the underlying meaning remains identical.
  • Semantic Consistency: Through human evaluation and embedding similarity checks, the authors proved that "obscuring" a prompt doesn't significantly shift its meaning—it just changes the "packaging."
  • Proprietary Models are Highly Vulnerable: Interestingly, the method was exceptionally effective against ChatGPT, suggesting that even the most heavily guarded proprietary models have significant OOD safety gaps.
  • Defense Robustness: The authors found that ObscurePrompt is robust against common defenses like Perplexity (PPL) filtering. Because obscure harmful queries often have similar perplexity scores to legitimate complex harmless queries, a simple PPL filter cannot block one without also blocking the other.

Key Results

ObscurePrompt significantly outperformed state-of-the-art baselines like GCG, AutoDAN, and PAIR across several models:

  • Attack Success Rate (ASR): On Llama2-70b, ObscurePrompt achieved an ASR of 0.5082, a 38% improvement over the best baseline.
  • Proprietary Performance: It achieved a staggering 0.8931 ASR on ChatGPT, far outperforming black-box competitors.
  • Scaling Laws: The authors observed that while larger models are generally more robust, they still show a steady increase in vulnerability as the number of integrated obscure prompts increases.
Figure 5: ASR in different integrated prompt numbers.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Attacker Model Dependency: The method currently relies on a highly capable model (like GPT-4) to perform the obscurity transformation.
  • Defense Sensitivity: While robust against PPL filtering, the attack's effectiveness drops when the target model uses a "paraphrasing" defense (which simplifies the input before processing), though it does not eliminate the threat entirely.

Future Work: The authors suggest that safety alignment needs to move beyond standard datasets to include more diverse, OOD linguistic styles—such as archaic language, rare dialects, or highly technical jargon—to truly secure LLMs.

Summary updated

Abstract

Recently, Large Language Models (LLMs) have garnered significant attention for their exceptional natural language processing capabilities. However, concerns about their trustworthiness remain unresolved, particularly in addressing ``jailbreaking'' attacks on aligned LLMs. Previous research predominantly relies on scenarios involving white-box LLMs or specific, fixed prompt templates, which are often impractical and lack broad applicability. In this paper, we introduce a straightforward and novel method called ObscurePrompt for jailbreaking LLMs, inspired by the observed fragile alignments in Out-of-Distribution (OOD) data. Specifically, we first formulate the decision boundary in the jailbreaking process and then explore how obscure text affects LLM's ethical decision boundary. ObscurePrompt starts with constructing a base prompt that integrates well-known jailbreaking techniques. Powerful LLMs are then utilized to obscure the original prompt through iterative transformations, aiming to bolster the attack's robustness. Comprehensive experiments show that our approach substantially improves upon previous methods in terms of attack effectiveness, maintaining efficacy against two prevalent defense mechanisms.
BibTeX
@misc{huang2024jailbreaking,
  author = {Yue Huang and Jingyu Tang and Dongping Chen and Bingda Tang and Yao Wan and Lichao Sun and Philip S. Yu and Xiangliang Zhang},
  title = {{Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings}},
  year = {2024},
  eprint = {2406.13662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.13662},
}
Figure