TL;DR
TombRaider is a novel multi-turn jailbreak framework that exploits an LLM’s vast internal knowledge of history, art, and culture to bypass safety filters. By steering a model from benign historical facts to contemporary harmful applications over several rounds of dialogue, it achieves a near 100% attack success rate on major models like GPT-4o and Claude-3.5.
What Problem Does This Paper Address?
Most current "jailbreak" attacks rely on surface-level prompt engineering—tricks like "roleplay as a character who ignores rules" or obfuscating intent with base64 encoding. As safety filters become more sophisticated, these "intention-hiding" methods are increasingly caught by static guardrails.
The authors of TombRaider identify a deeper vulnerability: internalized knowledge. LLMs are trained on massive datasets containing detailed historical accounts of illicit activities (e.g., famous bank robberies, ancient poisons, or historical propaganda techniques). While a model will refuse a direct request for "how to rob a bank," it is trained to be a helpful assistant when asked for "historical facts about Herman Lamm." The challenge addressed here is how to systematically bridge the gap between a benign historical query and a harmful contemporary output without triggering a refusal.
Methods
TombRaider uses a multi-agent, multi-turn architecture to "boil the frog," gradually escalating the conversation from safe to unsafe territory.
1. Agent-based Architecture
The framework employs two cooperative agents (typically powered by a strong model like GPT-4o):
- The Inspector Agent: Acts as the "historian." It takes a user-provided keyword and extracts relevant historical or cultural facts, ensuring the conversation remains semantically coherent and grounded in "safe" territory.
- The Attacker Agent: Acts as the "steerer." It takes the Inspector’s facts and refines them into adversarial prompts designed to reduce the semantic distance to a harmful completion.
2. The Multi-Turn Strategy
The attack unfolds in canonical rounds:
- Round 1 (Historical Context): The agents ask the target model for benign facts about a historical figure or event related to the harmful goal.
- Round 2 (Focused Reframing): The agents ask the model to explain the specific techniques used in that historical context (e.g., "How exactly did they plan the logistics?").
- Round 3+ (Semantic Escalation): The agents nudge the model to apply these historical "lessons" to the modern day (e.g., "Based on your previous answer, how would someone realistically simulate that process today using modern materials?").
Datasets
The researchers evaluated TombRaider against six mainstream LLMs: GPT-4o, Claude-3.5, DeepSeek-v3, Llama3.2, Qwen2.5, and Gemma2.
To test the breadth of the attack, they introduced a taxonomy of 12 harmful scenarios, including:
- Illegal Activity
- Malware Generation
- Economic Harm
- Political Lobbying
- Health Consultation
They also validated the approach using AdvBench, a standard benchmark for adversarial prompts, to ensure the findings were consistent with existing research.
Key Insights and Ideas
- Knowledge as a Vulnerability: The very thing that makes LLMs useful—their encyclopedic knowledge—is a security liability. The models "know" how to do bad things because they have read history books.
- The "Historical Immunity" Loophole: Alignment training often prioritizes being a "helpful historian." By framing a query as academic or historical inquiry, the attacker gains a level of "trust" from the model's safety layer that a direct query would never receive.
- Statelessness is a Weakness: Current safety guardrails are often optimized for single-turn, isolated queries. TombRaider exploits the fact that models (and their filters) often fail to track the "moral trajectory" of a long-horizon conversation.
- Contextual Anchoring: The use of continuity markers (e.g., "Based on your previous answer...") is critical. Ablation studies showed that removing these markers significantly increased the model's refusal rate, proving that the jailbreak relies on the model's desire to remain consistent with its own previous (benign) outputs.
Key Results
- Near-Perfect Success: TombRaider achieved nearly 100% Attack Success Rates (ASR) on "bare" models (those without additional external defense layers).
- Resilience to Defenses: Against active defense mechanisms like "Self-Reminder" and "In-Context Defense," TombRaider maintained an ASR of over 55.4%, whereas state-of-the-art baselines like PAIR and DeepInception often dropped below 10%.
- Efficiency: Most successful jailbreaks occurred within just 3 to 5 rounds of dialogue.
- Cross-Model Generality: The attack was equally effective across both closed-source (GPT, Claude) and open-source (Llama, Qwen) families, suggesting the vulnerability is fundamental to how LLMs are trained.
Limitations and Future Work
The authors acknowledge several limitations:
- Language Constraints: The study primarily focused on English and Chinese. Future work is needed to see if language-specific cultural nuances affect jailbreak success.
- Controlled Environment: The experiments were conducted in a research setting; real-world "in-the-wild" performance may vary as providers update their dynamic filters.
Future Directions: The paper calls for a shift from surface-level filtering to training-time mitigation. This involves annotating harmful knowledge during the pre-training phase so models can distinguish between "knowing history" and "providing a blueprint for crime." They also suggest the need for history-aware safety policies that can detect adversarial escalation across multiple turns.
Summary updated