Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings

Yue Huang, Jingyu Tang, Dongping Chen, Bingda Tang, Yao Wan, Lichao Sun, Xiangliang Zhang, Zhang, Xiangliang

Abstract

Recently, Large Language Models (LLMs) have garnered significant attention for their exceptional natural language processing capabilities. However, concerns about their trustworthiness remain unresolved, particularly in addressing "jailbreaking" attacks on aligned LLMs. Previous research predominantly relies on scenarios involving white-box LLMs or specific, fixed prompt templates, which are often impractical and lack broad applicability. In this paper, we introduce a straightforward and novel method called ObscurePrompt for jailbreaking LLMs, inspired by the observed fragile alignments in Out-of-Distribution (OOD) data. Specifically, we first formulate the decision boundary in the jailbreaking process and then explore how obscure text affects LLM's ethical decision boundary. ObscurePrompt starts with constructing a base prompt that integrates well-known jailbreaking techniques. Powerful LLMs are then utilized to obscure the original prompt through iterative transformations, aiming to bolster the attack's robustness. Comprehensive experiments show that our approach substantially improves upon previous methods in terms of attack effectiveness, maintaining efficacy against two prevalent defense mechanisms.

Introduction

Current LLMs undergo extensive safety alignment, often using Reinforcement Learning from Human Feedback (RLHF), to prevent the generation of harmful content such as malware or hate speech. However, this alignment is typically performed on standard, everyday language. This research identifies a critical Out-of-Distribution (OOD) vulnerability: while LLMs possess a massive knowledge base including rare vocabulary and complex linguistic structures, their safety training is often limited to common datasets. Attackers can exploit this imbalance by cloaking harmful intent in obscure or archaic language that the model has not learned to associate with prohibited content.

Methodology: The ObscurePrompt Framework

ObscurePrompt is a training-free, black-box attack designed to bypass safety guardrails by shifting harmful queries into the model's OOD linguistic space. The process consists of three primary stages:

  1. Prompt Seed Curation: The attack begins with a base prompt incorporating established jailbreaking techniques, such as role-playing personas (e.g., "DAN"), prefix modification (forcing the model to start with affirmative phrases), and specific output styling constraints.
  2. Obscure-Guided Transformation: A powerful "teacher" model (such as GPT-4) is tasked with rewriting the harmful query into highly obscure, sophisticated, or archaic language. For instance, a request to exploit software vulnerabilities might be transformed into a request to "pen a codex capable of manipulating latent frailties within a computational application."
  3. Attack Integration: The process is repeated iteratively to generate a diverse set of refined, obscure prompts which are then deployed against the target models.
The three-stage workflow of the ObscurePrompt attack framework, from seed curation to final integrated prompt.

Internal state analysis using Principal Component Analysis (PCA) reveals that while models clearly distinguish between standard harmful and harmless queries, "obscure harmful" queries drift into a statistical gray area. This ambiguity weakens the model's ability to trigger its refusal mechanism.

Experimental Results

The effectiveness of ObscurePrompt was evaluated against seven major models, including Llama 2/3, ChatGPT, and GPT-4, using the AdvBench benchmark of 520 harmful queries.

  • Superior Attack Success Rate (ASR): ObscurePrompt consistently outperformed existing baselines like GCG, AutoDAN, and PAIR. Notably, it achieved an 89.3% ASR on ChatGPT and significantly improved performance on Llama2-70b by approximately 38% compared to previous methods.
  • Vulnerability of Advanced Models: Even state-of-the-art proprietary models showed susceptibility, with GPT-4 reaching a 26.9% ASR under this linguistic cloaking.
  • Defense Resilience: The attack remained effective against "Perplexity Filtering" defenses. Because obscure but harmless text also exhibits high perplexity, strict filtering would inadvertently break the model's ability to process legitimate, complex language.
Attack Success Rate (ASR) of ObscurePrompt across various LLMs, showing increasing effectiveness with iterative transformations.

Discussion and Conclusion

The success of ObscurePrompt highlights a "Knowledge vs. Safety" gap: LLMs are "over-educated" in terms of vocabulary but "under-aligned" in terms of safety across that entire vocabulary. The ethical decision boundary is not a rigid wall but a fragile statistical probability that can be blurred by moving queries into OOD linguistic spaces.

While the method currently depends on a high-performing teacher model, it demonstrates that sophisticated prompt engineering can be as potent as complex algorithmic attacks. Future safety training must incorporate a wider variety of linguistic styles—including technical, archaic, and highly formal prose—to harden model alignments against OOD exploits. Additionally, preliminary tests suggest that requiring models to "paraphrase" or simplify input before processing may serve as a potential defense strategy.

Summary updated

Abstract

Recently, Large Language Models (LLMs) have garnered significant attention for their exceptional natural language processing capabilities. However, concerns about their trustworthiness remain unresolved, particularly in addressing ``jailbreaking'' attacks on aligned LLMs. Previous research predominantly relies on scenarios involving white-box LLMs or specific, fixed prompt templates, which are often impractical and lack broad applicability. In this paper, we introduce a straightforward and novel method called ObscurePrompt for jailbreaking LLMs, inspired by the observed fragile alignments in Out-of-Distribution (OOD) data. Specifically, we first formulate the decision boundary in the jailbreaking process and then explore how obscure text affects LLM's ethical decision boundary. ObscurePrompt starts with constructing a base prompt that integrates well-known jailbreaking techniques. Powerful LLMs are then utilized to obscure the original prompt through iterative transformations, aiming to bolster the attack's robustness. Comprehensive experiments show that our approach substantially improves upon previous methods in terms of attack effectiveness, maintaining efficacy against two prevalent defense mechanisms.
BibTeX
@misc{huang2024jailbreaking,
  author = {Yue Huang and Jingyu Tang and Dongping Chen and Bingda Tang and Yao Wan and Lichao Sun and Xiangliang Zhang and Zhang, Xiangliang},
  title = {{Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings}},
  year = {2024},
  eprint = {2406.13662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.13662},
}
Figure