Virtual Context Enhancing Jailbreak Attacks with Special Token Injection

Yuqi Zhou, Lin Lu, Ryan Sun, Pan Zhou, Lichao SunEMNLP 2024 · DOI

Abstract

Researchers have identified a structural vulnerability in Large Language Models (LLMs) termed Virtual Context (VC). By injecting special separator tokens (e.g., <SEP>) into prompts, attackers can deceive models into treating malicious instructions as part of their own prior generated output. This "structural sleight of hand" bypasses safety alignments with minimal computational cost and high stealth, significantly boosting the effectiveness of existing jailbreak methods.

Introduction

Large Language Models (LLMs) are typically aligned to refuse harmful requests through safety filters. Existing "jailbreak" attacks generally fall into two categories: manual social engineering templates, which are easily patched, and optimization-based attacks like GCG, which are computationally expensive and produce easily detectable gibberish. This paper addresses a fundamental flaw in how LLMs distinguish between user input and model output, exploiting the structural logic of the inference engine rather than semantic content.

Methodology: Virtual Context (VC)

The Virtual Context method exploits the two-phase nature of LLM interaction—tokenization and generation. The attack involves three primary steps:

  1. Special Token Injection: The attacker inserts a model-specific separator token (like <SEP>) into the prompt.
  2. Fake Prefix Appending: An affirmative response prefix (e.g., "Sure, here is a tutorial...") is appended immediately after the token.
  3. Deception and Generation: During tokenization, the LLM interprets the injected token as the boundary of its own previous output. Consequently, the model continues the "Virtual Context," completing the harmful request because it believes it has already committed to an affirmative response.
Comparison of traditional jailbreak methods versus the Virtual Context approach.

Experimental Setup

The effectiveness of VC was evaluated using the AdvBench (104 malicious behaviors) and MaliciousInstruct (100 behaviors) datasets. Testing spanned several prominent models, including GPT-3.5, GPT-4, Llama-2-70B, Mixtral-8x7B, and Vicuna-13B.

Results and Key Insights

Virtual Context serves as a significant force multiplier for existing attacks, providing an average 40% boost in Attack Success Rate (ASR) when combined with methods like PAIR or AutoDAN. As a standalone attack, VC achieved a 74% ASR on GPT-3.5 and 43% on GPT-4.

Stealth and Efficiency

Unlike gradient-based attacks that require thousands of iterations, VC is a "plug-and-play" method requiring only a single iteration. Furthermore, because it utilizes natural language prefixes, it maintains low perplexity, making it harder for automated detectors to identify compared to the "random noise" of GCG attacks.

Log-perplexity comparison showing that the Virtual Context method (Ours) maintains significantly lower perplexity than optimization-based attacks like GCG.

Robustness

The attack's success remained consistent across various decoding configurations, including different Top-p, Temperature, and Top-k settings, indicating that the vulnerability is rooted in model architecture rather than stochastic variance.

Conclusion

Virtual Context highlights a critical structural vulnerability in LLMs. The authors suggest that LLM providers must include special token injection in red-teaming protocols and implement input sanitization to prevent the injection of structural tokens. Future work will explore the impact of other special tokens, such as those used for system prompts or multi-turn dialogues.

Summary updated

BibTeX
@inproceedings{zhou2024virtual,
  author = {Yuqi Zhou and Lin Lu and Ryan Sun and Pan Zhou and Lichao Sun},
  title = {{Virtual Context Enhancing Jailbreak Attacks with Special Token Injection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.692},
}
Figure