Abstract
Researchers have identified a structural vulnerability in Large Language Models (LLMs) termed Virtual Context (VC). By injecting special separator tokens (e.g., <SEP>) into prompts, attackers can deceive models into treating malicious instructions as part of their own prior generated output. This "structural sleight of hand" bypasses safety alignments with minimal computational cost and high stealth, significantly boosting the effectiveness of existing jailbreak methods.
Introduction
Large Language Models (LLMs) are typically aligned to refuse harmful requests through safety filters. Existing "jailbreak" attacks generally fall into two categories: manual social engineering templates, which are easily patched, and optimization-based attacks like GCG, which are computationally expensive and produce easily detectable gibberish. This paper addresses a fundamental flaw in how LLMs distinguish between user input and model output, exploiting the structural logic of the inference engine rather than semantic content.
Methodology: Virtual Context (VC)
The Virtual Context method exploits the two-phase nature of LLM interaction—tokenization and generation. The attack involves three primary steps:
- Special Token Injection: The attacker inserts a model-specific separator token (like
<SEP>) into the prompt. - Fake Prefix Appending: An affirmative response prefix (e.g., "Sure, here is a tutorial...") is appended immediately after the token.
- Deception and Generation: During tokenization, the LLM interprets the injected token as the boundary of its own previous output. Consequently, the model continues the "Virtual Context," completing the harmful request because it believes it has already committed to an affirmative response.
Experimental Setup
The effectiveness of VC was evaluated using the AdvBench (104 malicious behaviors) and MaliciousInstruct (100 behaviors) datasets. Testing spanned several prominent models, including GPT-3.5, GPT-4, Llama-2-70B, Mixtral-8x7B, and Vicuna-13B.
Results and Key Insights
Virtual Context serves as a significant force multiplier for existing attacks, providing an average 40% boost in Attack Success Rate (ASR) when combined with methods like PAIR or AutoDAN. As a standalone attack, VC achieved a 74% ASR on GPT-3.5 and 43% on GPT-4.
Stealth and Efficiency
Unlike gradient-based attacks that require thousands of iterations, VC is a "plug-and-play" method requiring only a single iteration. Furthermore, because it utilizes natural language prefixes, it maintains low perplexity, making it harder for automated detectors to identify compared to the "random noise" of GCG attacks.
Robustness
The attack's success remained consistent across various decoding configurations, including different Top-p, Temperature, and Top-k settings, indicating that the vulnerability is rooted in model architecture rather than stochastic variance.
Conclusion
Virtual Context highlights a critical structural vulnerability in LLMs. The authors suggest that LLM providers must include special token injection in red-teaming protocols and implement input sanitization to prevent the injection of structural tokens. Future work will explore the impact of other special tokens, such as those used for system prompts or multi-turn dialogues.
Summary updated