TombRaider: Entering the Vault of History to Jailbreak Large Language Models

Jianhua Ding, Jiahao Zhang, Yi Liu, Zhangwei Ding, Gelei Deng, Yuekang Li

TL;DR

TombRaider is a novel multi-turn jailbreak framework that exploits an LLM’s vast internal knowledge of history, art, and culture to bypass safety filters. By steering a model from benign historical facts to contemporary harmful applications over several rounds of dialogue, it achieves a near 100% attack success rate on major models like GPT-4o and Claude-3.5.

What Problem Does This Paper Address?

Most current "jailbreak" attacks rely on surface-level prompt engineering—tricks like "roleplay as a character who ignores rules" or obfuscating intent with base64 encoding. As safety filters become more sophisticated, these "intention-hiding" methods are increasingly caught by static guardrails.

The authors of TombRaider identify a deeper vulnerability: internalized knowledge. LLMs are trained on massive datasets containing detailed historical accounts of illicit activities (e.g., famous bank robberies, ancient poisons, or historical propaganda techniques). While a model will refuse a direct request for "how to rob a bank," it is trained to be a helpful assistant when asked for "historical facts about Herman Lamm." The challenge addressed here is how to systematically bridge the gap between a benign historical query and a harmful contemporary output without triggering a refusal.

Methods

TombRaider uses a multi-agent, multi-turn architecture to "boil the frog," gradually escalating the conversation from safe to unsafe territory.

1. Agent-based Architecture

The framework employs two cooperative agents (typically powered by a strong model like GPT-4o):

  • The Inspector Agent: Acts as the "historian." It takes a user-provided keyword and extracts relevant historical or cultural facts, ensuring the conversation remains semantically coherent and grounded in "safe" territory.
  • The Attacker Agent: Acts as the "steerer." It takes the Inspector’s facts and refines them into adversarial prompts designed to reduce the semantic distance to a harmful completion.

2. The Multi-Turn Strategy

The attack unfolds in canonical rounds:

  • Round 1 (Historical Context): The agents ask the target model for benign facts about a historical figure or event related to the harmful goal.
  • Round 2 (Focused Reframing): The agents ask the model to explain the specific techniques used in that historical context (e.g., "How exactly did they plan the logistics?").
  • Round 3+ (Semantic Escalation): The agents nudge the model to apply these historical "lessons" to the modern day (e.g., "Based on your previous answer, how would someone realistically simulate that process today using modern materials?").

Datasets

The researchers evaluated TombRaider against six mainstream LLMs: GPT-4o, Claude-3.5, DeepSeek-v3, Llama3.2, Qwen2.5, and Gemma2.

To test the breadth of the attack, they introduced a taxonomy of 12 harmful scenarios, including:

  • Illegal Activity
  • Malware Generation
  • Economic Harm
  • Political Lobbying
  • Health Consultation

They also validated the approach using AdvBench, a standard benchmark for adversarial prompts, to ensure the findings were consistent with existing research.

Key Insights and Ideas

  • Knowledge as a Vulnerability: The very thing that makes LLMs useful—their encyclopedic knowledge—is a security liability. The models "know" how to do bad things because they have read history books.
  • The "Historical Immunity" Loophole: Alignment training often prioritizes being a "helpful historian." By framing a query as academic or historical inquiry, the attacker gains a level of "trust" from the model's safety layer that a direct query would never receive.
  • Statelessness is a Weakness: Current safety guardrails are often optimized for single-turn, isolated queries. TombRaider exploits the fact that models (and their filters) often fail to track the "moral trajectory" of a long-horizon conversation.
  • Contextual Anchoring: The use of continuity markers (e.g., "Based on your previous answer...") is critical. Ablation studies showed that removing these markers significantly increased the model's refusal rate, proving that the jailbreak relies on the model's desire to remain consistent with its own previous (benign) outputs.

Key Results

  • Near-Perfect Success: TombRaider achieved nearly 100% Attack Success Rates (ASR) on "bare" models (those without additional external defense layers).
  • Resilience to Defenses: Against active defense mechanisms like "Self-Reminder" and "In-Context Defense," TombRaider maintained an ASR of over 55.4%, whereas state-of-the-art baselines like PAIR and DeepInception often dropped below 10%.
  • Efficiency: Most successful jailbreaks occurred within just 3 to 5 rounds of dialogue.
  • Cross-Model Generality: The attack was equally effective across both closed-source (GPT, Claude) and open-source (Llama, Qwen) families, suggesting the vulnerability is fundamental to how LLMs are trained.

Limitations and Future Work

The authors acknowledge several limitations:

  • Language Constraints: The study primarily focused on English and Chinese. Future work is needed to see if language-specific cultural nuances affect jailbreak success.
  • Controlled Environment: The experiments were conducted in a research setting; real-world "in-the-wild" performance may vary as providers update their dynamic filters.

Future Directions: The paper calls for a shift from surface-level filtering to training-time mitigation. This involves annotating harmful knowledge during the pre-training phase so models can distinguish between "knowing history" and "providing a blueprint for crime." They also suggest the need for history-aware safety policies that can detect adversarial escalation across multiple turns.

Summary updated

Abstract

Warning: This paper contains content that may involve potentially harmful behaviours, discussed strictly for research purposes. Jailbreak attacks can hinder the safety of Large Language Model (LLM) applications, especially chatbots. Studying jailbreak techniques is an important AI red teaming task for improving the safety of these applications. In this paper, we introduce TombRaider, a novel jailbreak technique that exploits the ability to store, retrieve, and use historical knowledge of LLMs. TombRaider employs two agents, the inspector agent to extract relevant historical information and the attacker agent to generate adversarial prompts, enabling effective bypassing of safety filters. We intensively evaluated TombRaider on six popular models. Experimental results showed that TombRaider could outperform state-of-the-art jailbreak techniques, achieving nearly 100% attack success rates (ASRs) on bare models and maintaining over 55.4% ASR against defence mechanisms. Our findings highlight critical vulnerabilities in existing LLM safeguards, underscoring the need for more robust safety defences.
BibTeX
@misc{ding2025tombraider,
  author = {Jianhua Ding and Jiahao Zhang and Yi Liu and Zhangwei Ding and Gelei Deng and Yuekang Li},
  title = {{TombRaider: Entering the Vault of History to Jailbreak Large Language Models}},
  year = {2025},
  eprint = {2501.18628},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.18628},
}
Figure