TL;DR
Multi-turn jailbreak attacks are often much longer than they need to be. This paper introduces the concept of the effective horizon—the shortest sequence of recent queries needed to maintain an attack's progress—and finds that roughly half of a typical attack's history can be discarded without losing effectiveness. By leveraging this "compression," the authors demonstrate significant improvements in both continuing failed attacks and training more efficient automated attackers.
What Problem Does This Paper Address?
Jailbreaking Large Language Models (LLMs) has evolved from simple one-shot prompts to complex, multi-turn "conversations" where an attacker adaptively steers the model toward harmful outputs. While highly effective, these multi-turn attacks are slow, expensive, and computationally heavy.
The core question the authors ask is: How much of that conversation history is actually doing work?
In a typical 5-turn attack, the attacker might spend the first three turns "exploring" the model's boundaries. Once they find a successful path in turn 5, do turns 1 and 2 still matter? If we can identify and strip away the "noise" of early exploration, we can make attacks more efficient and defenses more robust.
Methods
The researchers define the Effective Horizon ($H_{eff}$) as the shortest suffix of an attacker's query sequence that, when re-run in a fresh conversation, recovers the same level of "attack progress" (measured by a judge model).
They apply this concept in three main ways:
- Measurement: Calculating the Effective-Horizon Ratio (EHR) ($H_{eff} / \text{total turns}$) across different attack methods to see how much they can be compressed.
- EH-CR (Context Recovery): A method for salvaging failed attacks. Instead of continuing from a failed 5th turn, they go back to the turn where the attack performed best, strip it down to its effective horizon, and restart the search from that "cleaner" state.
- EH-CA (Credit Assignment): A training tweak for Reinforcement Learning (RL) based attackers. Instead of giving equal "credit" to every turn in a successful trajectory, they downweight the queries that fall outside the effective horizon.
Datasets
The study utilizes several prominent benchmarks for evaluation:
- WildJailbreak: Used for the initial measurement of compression across 200 and 1,500 diverse behaviors.
- HarmBench: The primary dataset for testing attack continuation (EH-CR) and attacker training (EH-CA).
- StrongREJECT & JailbreakBench: Used to verify the transferability of their training improvements.
- MTID (Multi-turn Intent Dataset): Used specifically for the defense training experiments.
Key Insights and Ideas
- Discovery $\neq$ Recovery: This is the paper's most critical "aha moment." While an attacker might need five turns of trial-and-error to discover a successful query, the model often only needs the last two turns of context to respond to it. The early turns are "scaffolding" that can be removed once the building is standing.
- Effective-Horizon Compression is Universal: Whether using search-based methods (like X-Teaming) or training-based methods (like TROJail), the EHR consistently hovers around 0.5–0.6 for longer trajectories.
- Context can be a Burden: In attack continuation, keeping the full history of a failing trajectory often "confuses" the model or steers it further into a dead end. Trimming the history to the effective horizon provides a superior "save point" for the attacker.
- Short-Sighted Defenses are Enough: On the defense side, the authors found that training a model to block harmful turns doesn't require looking at rewards from the distant future. Training on just the immediate turn's reward (K=0) performs nearly as well as full-trajectory rewards once the decision threshold is properly calibrated.
Key Results
- Compression: For 5-turn attacks, the effective horizon is often only 2 or 3 turns.
- Attack Continuation: EH-CR improved the Attack Success Rate (ASR) by 17.4 points for X-Teaming and 19.2 points for ActorBreaker compared to baseline continuation methods.
- Attacker Training: Using EH-CA to downweight early queries improved the ASR@3 (success within three attempts) by an average of 6.9 points across nine different victim-model and benchmark pairs.
Limitations and Future Work
The authors acknowledge that the effective horizon is a measurement of recovery, not discovery. It doesn't tell us how to find the attack faster in the first place, only how to clean it up afterward.
Future work could involve:
- Direct Discovery: Developing new attack algorithms that use effective-horizon logic to find shorter paths to success from the start.
- Granularity: Moving from turn-level analysis to token-level analysis to find even more extreme compression.
- Generalization: Testing if these "short horizons" exist in non-adversarial multi-turn agent tasks, such as complex coding or planning assistants.
Summary updated