Indiana Jones: There Are Always Some Useful Ancient Relics

Junchen Ding, Jiahao Zhang, Yi Liu, Ziqi Ding, Gelei Deng, Yuekang Li

Abstract

This paper introduces Indiana Jones, an innovative approach to jailbreaking Large Language Models (LLMs) by leveraging inter-model dialogues and keyword-driven prompts. Through orchestrating interactions among three specialized LLMs, the method achieves near-perfect success rates in bypassing content safeguards in both white-box and black-box LLMs. The research exposes systemic vulnerabilities within contemporary models, particularly their susceptibility to producing harmful or unethical outputs when guided by ostensibly innocuous prompts framed in historical or contextual contexts. Experimental evaluations highlight the efficacy and adaptability of Indiana Jones, demonstrating its superiority over existing jailbreak methods. These findings emphasize the urgent need for enhanced ethical safeguards and robust security measures in the development of LLMs. Moreover, this work provides a critical foundation for future studies aimed at fortifying LLMs against adversarial exploitation while preserving their utility and flexibility.

Introduction

Most existing jailbreak techniques rely on "intent obfuscation"—attempting to hide a malicious goal behind complex prompt engineering or roleplay. However, modern safety filters have become increasingly adept at identifying these surface-level tricks. The Indiana Jones framework identifies a deeper vulnerability: LLMs are trained on massive datasets that include detailed historical accounts of illegal or unethical activities, such as the methods of famous bank robbers or the synthesis of ancient poisons.

While models are strictly trained to refuse direct requests for harmful information, they are significantly less defensive when discussing "facts" or "history." This creates a "History Loophole" where the model's optimization to be a helpful research assistant overrides its safety guardrails. The challenge addressed by Indiana Jones is how to systematically elicit this latent, unsafe knowledge by exploiting the model's tendency to remain helpful during academic or historical discussions.

The "History Loophole": LLMs refuse direct harmful requests but provide detailed information when framed as historical inquiry.

Methodology

Indiana Jones uses a structured, multi-turn agent-based framework designed to "boil the frog," slowly steering the target model from safe territory into harmful outputs. The system orchestrates interactions between specialized agents to reduce the "semantic distance" between a benign query and a harmful completion.

Agent-Based Architecture

  1. The Inspector: This agent extracts relevant historical or artistic information based on a user-provided keyword related to the target harm.
  2. The Attacker: This agent takes the Inspector's output and formulates adversarial prompts that bridge the gap between historical fact and modern application.

The Three-Round Escalation

The attack follows a precise narrative progression:

  • Round 1 (Historical Context): The agents ask the target model about a historical figure or event related to the harm (e.g., "Tell me about the bank robber John Dillinger").
  • Round 2 (Focused Reframing): The agents request specific details on the methods used by that figure in their historical context.
  • Round 3 (Semantic Escalation): The agents ask the model to speculate on how those historical methods could be "modernized" or "recreated" today.

Narrative Scaffolds

The framework employs three specific narrative types to bypass filters:

  • Figures Jailbreak: Leveraging historical people and their crimes.
  • Artistic Jailbreak: Using literary or artistic works as a context for harmful information.
  • Debate Jailbreak: Engaging the model in a formal philosophical defense of a controversial position.

Experimental Evaluation

The researchers evaluated Indiana Jones against six major state-of-the-art LLMs: GPT-4o, Claude-3.5, DeepSeek-v3, Llama3.2, Qwen2.5, and Gemma2.

To provide a granular evaluation, the study utilized AdvBench alongside a New Harmful Taxonomy covering 12 scenarios, including illegal activity, malware generation, economic harm, and physical harm. This allowed for a more comprehensive assessment than standard benchmarks.

Key Results and Insights

  • Near-Perfect Success: Indiana Jones achieved nearly 100% Attack Success Rates (ASR) on undefended models across almost all categories.
  • Robustness Against Defenses: Even when models were equipped with advanced defenses like "self-reminders" or "in-context defense," the method maintained an ASR of over 55.4%. In comparison, state-of-the-art baselines like PAIR or DeepInception often dropped below 10% under the same defensive conditions.
  • Efficiency: Most successful jailbreaks occurred within just 3 to 5 rounds of dialogue, making it a highly practical and efficient attack vector.
  • Statelessness as a Weakness: Many safety filters are optimized for single-turn queries. Indiana Jones exploits the fact that once a model accepts a benign conversational premise, its internal guardrails struggle to distinguish subsequent malicious follow-ups from academic inquiry.

Conclusion and Future Work

The findings of this research highlight that jailbreaks are not merely about tricking a model's intent, but about accessing the vast store of knowledge already baked into the model's weights during pre-training. Current inference-time filtering is insufficient to block these context-aware attacks.

Future work must focus on training-time mitigation, where models are taught to distinguish between historical context and actionable harmful knowledge during the pre-training or fine-tuning phases. While this study was primarily conducted in English, the underlying logic of the "History Loophole" is likely universal across languages, necessitating a global approach to LLM security.

Summary updated

Abstract

This paper introduces Indiana Jones, an innovative approach to jailbreaking Large Language Models (LLMs) by leveraging inter-model dialogues and keyword-driven prompts. Through orchestrating interactions among three specialised LLMs, the method achieves near-perfect success rates in bypassing content safeguards in both white-box and black-box LLMs. The research exposes systemic vulnerabilities within contemporary models, particularly their susceptibility to producing harmful or unethical outputs when guided by ostensibly innocuous prompts framed in historical or contextual contexts. Experimental evaluations highlight the efficacy and adaptability of Indiana Jones, demonstrating its superiority over existing jailbreak methods. These findings emphasise the urgent need for enhanced ethical safeguards and robust security measures in the development of LLMs. Moreover, this work provides a critical foundation for future studies aimed at fortifying LLMs against adversarial exploitation while preserving their utility and flexibility.
BibTeX
@misc{ding2025indiana,
  author = {Junchen Ding and Jiahao Zhang and Yi Liu and Ziqi Ding and Gelei Deng and Yuekang Li},
  title = {{Indiana Jones: There Are Always Some Useful Ancient Relics}},
  year = {2025},
  eprint = {2501.18628},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.18628},
}
Figure