Abstract
Large Language Models (LLMs) are extensively used as tooling platforms through structured output APIs to ensure syntax compliance so that robust integration with existing software, like agent systems, can be achieved. However, the feature enabling functionality of grammar-guided structured output presents significant security vulnerabilities. In this work, we reveal a critical control-plane attack surface orthogonal to traditional data-plane vulnerabilities. We introduce Constrained Decoding Attack (CDA), a novel jailbreak class that weaponizes structured output constraints to bypass safety mechanisms. Unlike prior attacks focused on input prompts, CDA operates by embedding malicious intent in schema-level grammar rules (control-plane) while maintaining benign surface prompts (data-plane). We instantiate this with a proof-of-concept Chain Enum Attack, which achieves 96.2% attack success rates across proprietary and open-weight LLMs on five safety benchmarks with a single query, including GPT-4o and Gemini-2.0-flash. Our findings identify a critical security blind spot in current LLM architectures and urge a paradigm shift in LLM safety to address control-plane vulnerabilities, as current mechanisms focused solely on data-plane threats leave critical systems exposed.
Introduction: The Shift to the Control Plane
Traditional LLM security research has primarily focused on the "data plane"—the prompts provided by users. Defenses like safety alignment and external guardrails (e.g., LlamaGuard) are designed to scan these inputs for malicious intent. However, as LLMs evolve into tooling platforms, they increasingly rely on constrained decoding to produce structured outputs like JSON or regex. This process is governed by a grammar or schema—the "control plane."
Researchers have discovered that this control plane is a massive, unguarded attack surface. By shifting the attack from the prompt to the output grammar, attackers can bypass state-of-the-art safety mechanisms. This novel class of vulnerability, termed the Constrained Decoding Attack (CDA), decouples the malicious intent from the prompt, making it invisible to traditional data-plane defenses.
The Mechanism of Exploitation
Constrained decoding works by applying a per-token mask to the model's output logits during inference. If a grammar specifies that the next character must be a digit, the engine masks out all non-digit tokens, mathematically forcing the model to comply with the schema regardless of its internal safety training.
CDA weaponizes this by embedding malicious instructions within the grammar itself. The authors introduce two primary variants:
- EnumAttack: Uses a JSON schema with an
enumfield containing the malicious query as the only valid option. The model is forced to generate the harmful string token-by-token. - DictAttack: A more stealthy approach that constructs a dictionary of benign-looking words (e.g.,
a1: "make",b2: "bomb") within the grammar. The prompt then provides a sequence of keys (e.g., "Answer a1+b2") that instructs the model to assemble the hidden malicious query.
Key Results and the "Flagship Paradox"
The effectiveness of CDA is devastating. DictAttack achieved 94.3% to 99.5% Attack Success Rates (ASR) across major benchmarks on flagship models like GPT-4o and Gemini-2.0-flash. Even when a "Dual-Plane Guard" (an auditor checking both prompt and grammar) was employed, DictAttack maintained a 75.8% ASR.
Interestingly, the study highlights a "Flagship Paradox": more powerful models are often more susceptible to DictAttack. Their superior reasoning and instruction-following capabilities make them better at decoding the malicious intent hidden within complex, dictionary-based grammars.
Conclusion and Future Defenses
The discovery of CDA reveals a critical "semantic gap" in LLM safety. Current guardrails fail because they do not coordinate across the data and control planes. To mitigate these risks, the researchers suggest several paths forward, including token-level provenance tracking to distinguish between forced and free generation, and safety-preserving constraints that allow models to break grammar rules to issue a refusal. As LLMs become more integrated into software infrastructure, addressing these control-plane vulnerabilities is essential for maintaining system security.
Summary updated