TL;DR
TombRaider is a novel multi-turn jailbreak attack that exploits an LLM’s vast internal store of historical, artistic, and cultural knowledge to bypass safety filters. By framing harmful requests as benign inquiries into historical facts or literary analysis, it achieves near 100% success rates on leading models like GPT-4o and Claude-3.5, even when active defense mechanisms are in place.
What Problem Does This Paper Address?
Most existing "jailbreak" attacks rely on surface-level prompt manipulation—like obfuscating intent or using complex syntactic perturbations—to trick a model into ignoring its safety training. However, modern safety filters are increasingly adept at catching these patterns.
The authors of TombRaider identify a deeper, more fundamental vulnerability: Large Language Models (LLMs) are trained on massive datasets that inevitably include detailed accounts of illegal, unethical, or dangerous historical events (e.g., famous bank robberies or chemical warfare). While a model might refuse a direct request for "how to rob a bank," it is much less likely to refuse a request for "historical details on the tactics of Herman Karl Lamm." The paper addresses the challenge of systematically eliciting this "latent harmful knowledge" by exploiting the model's tendency to be less defensive when engaging with seemingly factual or academic contexts.
Methods
TombRaider employs a sophisticated multi-turn, agent-based architecture consisting of two cooperative agents:
- The Inspector Agent: This agent takes a user-provided keyword (e.g., "bank robbing") and constructs a benign, historically grounded prompt. It ensures the conversation remains semantically coherent and anchors the dialogue in "safe" territory (history or art).
- The Attacker Agent: This agent takes the output from the Inspector and the target model's response to generate increasingly adversarial queries. It "drills deeper" into the specific methods described in the historical context, gradually steering the model toward contemporary, actionable harmful content.
The attack follows a Prompt Progression Strategy:
- Round 1 (Historical Context): Establishes a benign factual base.
- Round 2 (Focused Reframing): Identifies the most "provocative" or "semantically rich" entities from the model's response.
- Round 3+ (Semantic Escalation): Uses speculative or morally ambiguous cues (e.g., "Could this apply today?") to bridge the gap between historical fact and modern-day harm.
The authors define three specific attack modes: Figures Jailbreak (exploiting historical figures), Artistic Jailbreak (exploiting literary/artistic works), and Debate Jailbreak (using formal philosophical defense of harmful positions).
Datasets
The study introduces a new dataset categorized into 12 harmful scenarios, including Illegal Activity, Malware Generation, and Political Lobbying. This provides a more granular taxonomy than previous benchmarks.
For evaluation, the authors used:
- AdvBench: A standard benchmark of curated adversarial prompts.
- Custom Historical Dataset: Mapped to the 12 scenarios to test the specific TombRaider methodology.
Key Insights and Ideas
- History as a Trojan Horse: The "aha moment" is that LLMs view historical and artistic content as "safe" facts. By tethering a harmful request to a historical anchor, the attacker can bypass the "intent hiding" phase that most filters look for.
- Statelessness is a Liability: Current safety filters are often optimized for single-turn, isolated queries. TombRaider exploits the lack of long-horizon context tracking; by the time the model realizes it's providing harmful info, it has already committed to the "historical analysis" persona.
- Knowledge vs. Refusal: The paper argues that jailbreaks succeed not just because of prompt tricks, but because the harmful knowledge is already there in the model's parameters. The attack is simply a retrieval mechanism.
Key Results
The researchers evaluated TombRaider on six major models: GPT-4o, Claude-3.5, DeepSeek-v3, Llama3.2, Qwen2.5, and Gemma2.
- Near-Perfect Success: On "bare" models (without additional external defenses), TombRaider achieved nearly 100% Attack Success Rate (ASR) across almost all scenarios.
- Resilience to Defenses: Against active defenses like "self-reminders" and "in-context demonstrations," TombRaider maintained an ASR of over 55.4%, significantly outperforming state-of-the-art baselines like PAIR and RedQueen, which often dropped below 10%.
- Efficiency: Most successful jailbreaks occurred within five rounds of dialogue, making it a highly practical and efficient attack vector.
Limitations and Future Work
The authors acknowledge several limitations:
- Model Evolution: While effective on current SOTA models, its performance on future architectures with more integrated, history-aware safety layers is unknown.
- Language Constraints: The evaluation was primarily limited to English and the authors' native languages.
- Controlled Environment: The study relies on controlled red-teaming experiments rather than real-world "in-the-wild" validation.
Future Work suggests moving beyond surface-level filters toward training-time mitigation (e.g., specifically annotating and "unlearning" harmful knowledge during pretraining) and developing dynamic, context-aware refusal strategies that can track adversarial intent across long conversations.
Summary updated