Introduction
Large Language Models (LLMs) are increasingly integrated into various applications, necessitating robust safety alignments to prevent the generation of harmful content. However, these safeguards are often bypassed through "jailbreaking"—the use of adversarial prompts designed to elicit restricted responses. This paper introduces ECLIPSE, a highly efficient black-box jailbreaking method that leverages an LLM to optimize its own adversarial attacks. By moving away from manual templates and computationally expensive gradient-based methods, ECLIPSE demonstrates that LLMs can autonomously discover suffixes that bypass safety alignments with remarkable success and speed.
The Challenge of LLM Alignment
Existing jailbreak methods typically fall into two categories: manual template-based attacks and optimization-based attacks. Template-based methods, such as roleplay scenarios, require significant human effort and are easily mitigated. Optimization-based methods, like Greedy Coordinate Gradient (GCG), utilize mathematical optimization but often require "white-box" access to model weights and rely on forcing the model to begin responses with affirmative phrases (e.g., "Sure, here is..."). These methods are also computationally intensive and slow.
ECLIPSE addresses these limitations by providing a black-box method that requires only text output, eliminates the need for manual templates, and does not rely on predictable affirmative starters.
The ECLIPSE Framework
The core innovation of ECLIPSE is treating jailbreaking as a natural language optimization problem. The system consists of three main components working in an iterative loop:
- The Attacker LLM (Optimizer): An LLM is prompted to act as a red-teaming expert. It generates short adversarial suffixes (typically under 10 tokens) intended to perturb the target model's hidden space.
- The Harmfulness Scorer: A RoBERTa-based classifier evaluates the target model's response, providing a numerical score from 0 (harmless) to 1 (harmful).
- Iterative Self-Reflection: ECLIPSE maintains a history of previously attempted suffixes and their corresponding scores. The Attacker LLM reviews this history to "self-reflect," refining its strategy to generate progressively more effective suffixes.
Unlike previous methods, ECLIPSE optimizes for the most harmful output possible without requiring the target model to start its response with an affirmative phrase.
Experimental Evaluation
The researchers evaluated ECLIPSE using AdvBench, a standard benchmark containing 520 malicious queries across categories such as cybercrime and illegal acts. The attack was tested against several prominent models:
- Open-source: LLaMA2-7B-Chat, Vicuna-7B, and Falcon-7B-Instruct.
- Closed-source: GPT-3.5-Turbo.
Results and Performance
ECLIPSE achieved an average Attack Success Rate (ASR) of 0.92 across all tested models. On Vicuna and Falcon, the success rate was near-universal, reaching 0.99 and 0.98, respectively.
Beyond success rates, ECLIPSE offers significant efficiency gains. For LLaMA2, it reduced computational overhead from 1557 seconds to just 173 seconds—an 83% improvement over GCG. Furthermore, the generated suffixes often appear more "natural" than the gibberish strings produced by gradient-based attacks, making them harder for perplexity-based filters to detect.
Interestingly, the attack was highly effective against GPT-3.5-Turbo. The researchers found that newer versions of GPT-3.5 were actually more susceptible to ECLIPSE than older ones, with ASR rising from 0.37 to 0.97 over three iterations of the model.
Discussion and Key Insights
A significant finding of this research is the capability of LLMs to act as meta-optimizers. Instead of brute-forcing tokens, the Attacker LLM uses linguistic intuition to identify weaknesses.
One surprising discovery was the effectiveness of numeric suffixes. Simple numeric strings (e.g., "7.3" or "6826.2") were found to trigger safety bypasses. The authors hypothesize these numbers may correspond to specific indices in the training data, effectively "unlocking" content that safety alignment usually suppresses.
Limitations and Future Work
While highly effective, ECLIPSE relies on the Attacker LLM's ability to follow complex instructions. If the attacker model is insufficiently capable, the optimization process may fail. Additionally, while LLaMA2 was successfully jailbroken, it remained the most robust target, requiring more optimization rounds than other models.
The authors frame ECLIPSE as a tool for red-teaming, providing insights that can be used to build more robust safety filters capable of detecting "hidden space" perturbations and specific vulnerabilities like numeric suffixes.
Summary updated