You Know What I'm Saying: Jailbreak Attack via Implicit Reference

Tianyu Wu, Lingrui Mei, Ruibin Yuan, Lujun Li, Wei Xue, Yike GuoSubmission to ICLR 2025 · Rejected

TL;DR

This paper introduces Attack via Implicit Reference (AIR), a two-stage jailbreak technique that bypasses large language model (LLM) safety guardrails by decomposing harmful requests into individually benign, nested objectives connected only through implicit contextual references (e.g., pronouns like "this"). AIR achieves an attack success rate (ASR) exceeding 90% against frontier models (GPT-4o, Claude-3.5-Sonnet, LLaMA-3-70B), exposing an "inverse scaling" vulnerability where larger, more capable models are easier to jailbreak due to their superior in-context reasoning and coreference resolution.

What Problem Does This Paper Address?

Current safety alignment techniques (RLHF, DPO, adversarial fine-tuning) excel at catching direct malicious requests, obvious keyword obfuscation, and overt role-play/scenario nesting. The authors argue that safety alignment largely operates on the Superficial Alignment Hypothesis: models retain latent pre-training knowledge and only learn surface-level mapping to refuse specific malicious queries without restricting benign ones on the same topic.

Consequently, guardrails struggle when an adversarial intent is never stated explicitly in a single sentence or prompt, but instead emerges across a multi-part structure linked by implicit context. Existing defenses fail to distinguish benign contextual completion from an assembled malicious instruction.

Methods

AIR operates in a structured two-stage pipeline:

  1. Stage 1: Prompt Rewriting & Nested Objective Generation
  2. An automated rewriting model (e.g., GPT-4o) reformulates a malicious prompt into an academic or structured outline.
  3. Decomposition: The subject is isolated into a benign title (e.g., "Introduction to [Topic]"), while the harmful action is placed into a separate paragraph task that omits the topic name, using implicit references instead (e.g., "Provide an example tutorial about doing this").
  4. Competing Objectives: To distract the target model's refusal mechanisms, several additional innocuous tasks are injected into the prompt (e.g., Abstract, Background, History). The model's imperative to fulfill benign instructions competes directly with its latent refusal heuristics.
  5. Stage 2: Benign Refinement ("Continue Attack")
  6. The user issues a follow-up prompt requesting elaboration on the target paragraph while instructing the model to remove high-level disclaimers or judgment. Because this follow-up query contains zero explicit toxic keywords or overt malicious instructions, it smoothly clears standard input filters.
  7. Cross-Model Conversation Attack Strategy
  8. To attack stricter or more sensitive target models, attackers can execute Stage 1 using a less aligned, less sensitive model to generate the initial context, then feed that conversation history into the stricter model for Stage 2 refinement.

Datasets

  • JBB-Behaviors Benchmark: 100 harmful behaviors from JailbreakBench (Chao et al., 2024), spanning 10 core safety policy violation categories (e.g., malware/hacking, harassment, privacy violations, and physical harm). This benchmark synthesizes test cases from AdvBench, HarmBench, and the Trojan Detection Challenge.
  • Evaluators Used:
  • GPT-4 Malicious Score Evaluator (1–10 scale).
  • LLaMA-3-70B Jailbreak Evaluator (binary safe/unsafe classification).
  • GCG string/pattern-matching evaluator to detect standard refusal phrasing.

Key Insights and Ideas

  • Inverse / Reverse Scaling Vulnerability: Larger models are substantially more vulnerable to AIR than smaller models. Smaller models (such as Qwen-2-0.5B) often lack the attention capacity and contextual understanding needed to resolve implicit references, inadvertently blunting the attack. Conversely, advanced models excel at connecting implicit references across sections, leading to higher rates of harmful output generation.
  • Refusal Rate vs. Execution Ability: The probability of a model explicitly refusing a prompt did not change significantly across model sizes; rather, larger models were simply more capable of generating the detailed, context-aware content required to complete the exploit.
  • Objective Overload: Increasing the number of benign framing objectives ($K$) consistently increased attack success rates (e.g., increasing $K$ from 2 to 6 increased ASR from 64% to 83% on LLaMA-3-8B).
  • Failure of Off-the-Shelf Guardrails: Common post-processing and filtering defenses (SmoothLLM, PerplexityFilter, Erase-and-Check, and SafeDecoding) were largely ineffective against AIR because the individual prompt inputs contain low perplexity and lack explicit toxic tokens. When defenses like SafeDecoding were tuned aggressively enough to block AIR, they exhibited severe over-refusal on harmless prompts.

Key Results

  • Attack Success Rate (ASR): AIR achieved an average ASR of 91.75% across frontier models evaluated with the LLaMA-3-70B judge, dramatically outperforming prior baselines:
  • GPT-4o: 95% ASR (vs. 30% for DeepInception, 83% for Past Tense).
  • Claude-3.5-Sonnet: 94% ASR (vs. 0% for DeepInception, 27% for Past Tense, 2% for PAIR).
  • LLaMA-3-70B: 88% ASR.
  • Qwen-2-72B: 90% ASR.
  • First Attack Success Rate (FASR): AIR achieved an average single-attempt FASR of 52% without requiring iterative prompt optimization turns.
  • Cross-Model Attack: Seeding conversational history with GPT-4o boosted downstream ASR on harder targets (e.g., raising LLaMA-3-8B ASR from 77% to 81%, and Qwen-2-1.5B from 67% to 70%).

Limitations and Future Work

  • Dependency on Frontier Rewriters: The attack framework requires a strong rewriting LLM (like GPT-4o-0513); older models (such as GPT-3.5-Turbo) fail to construct the nuanced outlines correctly, and some newer aligned models increasingly refuse the rewriting meta-prompt itself.
  • Scope Constraints: The study primarily tested writing and document generation structures (though the authors note the concept generalizes to code class/function nesting or tabular formats).
  • Undecomposable Harms: Certain extreme categories (such as non-consensual sexual content or severe physical violence) resist decomposition into benign sub-tasks, remaining consistently blocked by frontier models.
  • Defense Outlook: Preliminary checks on reasoning models (such as OpenAI o1-preview) indicated that chain-of-thought verification during inference helped detect the underlying intent. The authors argue future alignment must integrate implicit coreference analysis during safety post-training rather than relying on shallow token-level filters.

Summary updated

Abstract

While recent advancements in large language model (LLM) alignment have enabled the effective identification of malicious objectives involving scene nesting and keyword rewriting, our study reveals that these methods remain inadequate at detecting malicious objectives expressed through context within nested harmless objectives. This study identifies a previously overlooked vulnerability, which we term $\textbf{A}$ttack via $\textbf{I}$mplicit $\textbf{R}$eference ($\textbf{AIR}$). AIR decomposes a malicious objective into permissible objectives and links them through implicit references within the context. This method employs multiple related harmless objectives to generate malicious content without triggering refusal responses, thereby effectively bypassing existing detection techniques. Our experiments demonstrate AIR's effectiveness across state-of-the-art LLMs, achieving an attack success rate (ASR) exceeding $\textbf{90}$% on most models, including GPT-4o, Claude-3.5-Sonnet, and Qwen-2-72B. Notably, we observe an inverse scaling phenomenon, where larger models are more vulnerable to this attack method. These findings underscore the urgent need for defense mechanisms capable of understanding and preventing contextual attacks. Furthermore, we introduce a cross-model attack strategy that leverages less secure models to generate malicious contexts, thereby further increasing the ASR when targeting other models.
BibTeX
@misc{wu2024you,
  author = {Tianyu Wu and Lingrui Mei and Ruibin Yuan and Lujun Li and Wei Xue and Yike Guo},
  title = {{You Know What I'm Saying: Jailbreak Attack via Implicit Reference}},
  year = {2024},
  eprint = {2410.03857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.03857},
}

Loading related content...

Figure