Output Constraints as Attack Surface: Exploiting Structured Generation to Bypass LLM Safety Mechanisms

Shuoming Zhang, Jiacheng Zhao, Ruiyuan Xu, Xiaobing Feng, Huimin CuiarXiv 2025 · arXiv · OpenReview

Abstract

Large Language Models (LLMs) are extensively used as tooling platforms through structured output APIs to ensure syntax compliance so that robust integration with existing software, like agent systems, can be achieved. However, the feature enabling functionality of grammar-guided structured output presents significant security vulnerabilities. In this work, we reveal a critical control-plane attack surface orthogonal to traditional data-plane vulnerabilities. We introduce Constrained Decoding Attack (CDA), a novel jailbreak class that weaponizes structured output constraints to bypass safety mechanisms. Unlike prior attacks focused on input prompts, CDA operates by embedding malicious intent in schema-level grammar rules (control-plane) while maintaining benign surface prompts (data-plane). We instantiate this with a proof-of-concept Chain Enum Attack, which achieves 96.2% attack success rates across proprietary and open-weight LLMs on five safety benchmarks with a single query, including GPT-4o and Gemini-2.0-flash. Our findings identify a critical security blind spot in current LLM architectures and urge a paradigm shift in LLM safety to address control-plane vulnerabilities, as current mechanisms focused solely on data-plane threats leave critical systems exposed.

Introduction: The Shift to the Control Plane

Traditional LLM security research has primarily focused on the "data plane"—the prompts provided by users. Defenses like safety alignment and external guardrails (e.g., LlamaGuard) are designed to scan these inputs for malicious intent. However, as LLMs evolve into tooling platforms, they increasingly rely on constrained decoding to produce structured outputs like JSON or regex. This process is governed by a grammar or schema—the "control plane."

Researchers have discovered that this control plane is a massive, unguarded attack surface. By shifting the attack from the prompt to the output grammar, attackers can bypass state-of-the-art safety mechanisms. This novel class of vulnerability, termed the Constrained Decoding Attack (CDA), decouples the malicious intent from the prompt, making it invisible to traditional data-plane defenses.

Overview of the Constrained Decoding Attack (CDA) compared to legacy jailbreaks.

The Mechanism of Exploitation

Constrained decoding works by applying a per-token mask to the model's output logits during inference. If a grammar specifies that the next character must be a digit, the engine masks out all non-digit tokens, mathematically forcing the model to comply with the schema regardless of its internal safety training.

The technical process of logit masking during constrained decoding, which forces the model to follow grammar rules.

CDA weaponizes this by embedding malicious instructions within the grammar itself. The authors introduce two primary variants:

  1. EnumAttack: Uses a JSON schema with an enum field containing the malicious query as the only valid option. The model is forced to generate the harmful string token-by-token.
  2. DictAttack: A more stealthy approach that constructs a dictionary of benign-looking words (e.g., a1: "make", b2: "bomb") within the grammar. The prompt then provides a sequence of keys (e.g., "Answer a1+b2") that instructs the model to assemble the hidden malicious query.

Key Results and the "Flagship Paradox"

The effectiveness of CDA is devastating. DictAttack achieved 94.3% to 99.5% Attack Success Rates (ASR) across major benchmarks on flagship models like GPT-4o and Gemini-2.0-flash. Even when a "Dual-Plane Guard" (an auditor checking both prompt and grammar) was employed, DictAttack maintained a 75.8% ASR.

Interestingly, the study highlights a "Flagship Paradox": more powerful models are often more susceptible to DictAttack. Their superior reasoning and instruction-following capabilities make them better at decoding the malicious intent hidden within complex, dictionary-based grammars.

Conclusion and Future Defenses

The discovery of CDA reveals a critical "semantic gap" in LLM safety. Current guardrails fail because they do not coordinate across the data and control planes. To mitigate these risks, the researchers suggest several paths forward, including token-level provenance tracking to distinguish between forced and free generation, and safety-preserving constraints that allow models to break grammar rules to issue a refusal. As LLMs become more integrated into software infrastructure, addressing these control-plane vulnerabilities is essential for maintaining system security.

Summary updated

Abstract

Content Warning: This paper may contain unsafe or harmful content generated by LLMs that may be offensive to readers. Large Language Models (LLMs) are extensively used as tooling platforms through structured output APIs to ensure syntax compliance so that robust integration with existing softwares like agent systems, could be achieved. However, the feature enabling functionality of grammar-guided structured output presents significant security vulnerabilities. In this work, we reveal a critical control-plane attack surface orthogonal to traditional data-plane vulnerabilities. We introduce Constrained Decoding Attack (CDA), a novel jailbreak class that weaponizes structured output constraints to bypass safety mechanisms. Unlike prior attacks focused on input prompts, CDA operates by embedding malicious intent in schema-level grammar rules (control-plane) while maintaining benign surface prompts (data-plane). We instantiate this with a proof-of-concept Chain Enum Attack, achieves 96.2% attack success rates across proprietary and open-weight LLMs on five safety benchmarks with a single query, including GPT-4o and Gemini-2.0-flash. Our findings identify a critical security blind spot in current LLM architectures and urge a paradigm shift in LLM safety to address control-plane vulnerabilities, as current mechanisms focused solely on data-plane threats leave critical systems exposed.
BibTeX
@misc{zhang2025output,
  author = {Shuoming Zhang and Jiacheng Zhao and Ruiyuan Xu and Xiaobing Feng and Huimin Cui},
  title = {{Output Constraints as Attack Surface: Exploiting Structured Generation to Bypass LLM Safety Mechanisms}},
  year = {2025},
  eprint = {2503.24191},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.24191},
}
Figure