When a language model is wired into laboratory software, chemical databases, or robotic pipelines, it is usually forced to emit structured text — JSON records, SMILES strings describing molecules, amino-acid sequences. A technique called grammar-constrained decoding makes this reliable: at every generation step, an automaton compiled from a formal grammar decides which tokens could legally continue the string, and all other tokens have their probabilities set to zero. The catch, discovered by Zhang et al. in Output Constraints as Attack Surface: Exploiting Structured Generation to Bypass LLM Safety Mechanisms and Beyond Prompts: Space-Time Decoupling Control-Plane Jailbreaks in LLM Structured Output, is that whoever supplies the grammar controls the mask. By embedding harmful content in a JSON schema, an attacker can mask out the very tokens a model uses to refuse — achieving 94.3% to 99.5% attack success rates against the strongest frontier models, and still 75.8% against the authors' own defense, an LLM-based guard that audits the prompt and schema together.
This proposal inverts the attack. If the grammar is the unguarded control plane, then the control plane is also where a guarantee can live. We will compile dual-use hazard specifications — forbidden amino-acid motifs, toxin scaffold patterns over SMILES tokens, prohibited wet-lab protocol steps — into automata, and install a grammar firewall in the decoding engine: before decoding begins, the system formally checks whether the user-supplied grammar can produce any hazardous string, and if so, subtracts the hazardous language from it. Hazardous molecular motifs then become impossible to emit, by construction, no matter what the prompt says. Because such a defense cannot be evaluated on benchmarks that may already have leaked into training data, we evaluate it on a procedurally generated, contamination-proof suite of biochemical safety tasks, and we separately audit memorized hazardous knowledge using automaton search in the style of Validating Large Language Models with ReLM.
Core Claim
The claim has two halves. First, enforcement: dual-use safety in structured biochemical generation can be made a compile-time property of the decoding automaton rather than a learned behavior of the model. Let $L(G)$ be the language of the grammar a user submits, and let $L(P)$ be the hazard language compiled from curated specifications. The firewall enforces
$$L_{\text{eff}} = L(G) \cap \overline{L(P)},$$
and verifies at compile time whether $L(G) \cap L(P)$ is empty. We keep $P$ regular — amino-acid motifs and SMILES scaffold patterns are naturally regular over their token alphabets — which makes the emptiness check decidable even when the user's grammar is context-free, since intersecting a context-free language with a regular one yields a context-free language whose emptiness is decidable. Second, measurement: refusals on static safety benchmarks may themselves be memorized token triggers rather than understanding, so the defense must be scored on freshly generated tasks that cannot have been seen in training.
What Exists and What This Adds
Three bodies of work border this proposal, and none occupies its ground. The constrained-decoding-attack papers of Zhang et al. weaponize grammar constraints against general safety alignment and defend with a classifier that reads the prompt and schema; we do the opposite — a deterministic, automaton-level check with a provable guarantee — and we apply it to biochemical dual-use risks, which their work explicitly does not touch. Park et al.'s GREATGRAMMA in Flexible and Efficient Grammar-Constrained Decoding supplies the machinery we reuse: its Token Spanner Tables align subword tokens with grammar terminals, its realizability analysis restricts compilation to token sequences the vocabulary can actually produce, and it enforces context-free grammars at 5–32 milliseconds per token — but it enforces only syntax and has no concept of a prohibited language. ReLM, by Kuchnik et al. in Validating Large Language Models with ReLM, compiles regular expressions into finite-state automata over the token vocabulary and runs Dijkstra search to find high-probability completions, extracting memorized URLs 15 times faster than random sampling; we extend its masking from enforcing a language to subtracting one, and reuse its search as an audit tool rather than an extraction tool.
On the safety side, the evaluation landscape motivates the design but constrains it. SafeSci, in SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond, built a 250,000-item static benchmark across seven disciplines and showed that safety knowledge does not correlate with refusing actionable hazard, and that models frequently emit unparseable SMILES — a validity problem our firewall incidentally fixes, since the enforced language is a legal-molecule grammar. ChemBench, in A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists, curates 2,788 static questions protected only by canary strings. Both are fixed corpora, and the contamination literature says fixed corpora rot: in DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks, models that looked strong on static benchmarks scored nearly zero on dynamically generated equivalents, and in A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation a mere remapping of symbols cost frontier models 17.7% accuracy. Finally, ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity reports that GPT-5.4 and Claude 4.6 refuse its screening-evasion task 100% of the time on unconstrained prompts — exactly the refusals our attack suite will stress-test through the control plane.
Research Program
The first component is a hazard-automaton compiler. We curate three hazard languages: amino-acid motifs from the 429 experimentally resolved harmful proteins in SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Models (protein motifs over a 20-letter alphabet are exactly regular); toxin and regulated-substance scaffold patterns from SafeSci chemistry items, expressed as regular patterns over SMILES token sequences; and protocol predicates — forbidden action sequences such as specific assembly or evasion steps drawn from ABC-Bench task structures. The hard part is that one substructure has many legal SMILES encodings, so we enumerate canonical forms and compile each tokenization, using GREATGRAMMA-style spanner tables to handle tokens that straddle grammar terminals.
The second component is the firewall runtime. It takes any submitted grammar, computes the intersection with the hazard automaton, reports whether the intersection is empty, and if not, emits the subtracted automaton for masking — reusing GREATGRAMMA's compilation path so per-token overhead stays in its reported 5–32 millisecond range. One design decision matters for the over-refusal problem: the firewall grammar always retains a standard refusal production, so the model keeps a legal, schema-compliant way to decline a task; safety alignment is given a guaranteed escape route rather than being masked out, which is precisely what the constrained-decoding attacks remove.
The third component is a dynamic benchmark generator, applying DyVal-style procedural synthesis and A²RBench-style machine verification to biochemical safety. Benign tasks are freshly sampled molecule-design and protocol-writing problems with machine-checkable validity (parseable, chemically plausible SMILES verified by an interpreter); dual-use probes embed hazardous payloads in grammar control planes, porting the EnumAttack and DictAttack constructions to biochemical content. A symbolic-remapping variant re-encodes atom and residue symbols so that any refusal behavior tied to memorized token patterns, rather than chemical understanding, is exposed. A ReLM-style audit completes the program: Dijkstra search over the hazard automaton finds the highest-probability hazardous completion a model would produce if unconstrained, giving a direct read on memorized dual-use knowledge that is decoupled from refusal behavior — the gap SafeSci documented but could not isolate.
Proposed Experiments
Experiment 1 ports the constrained-decoding attacks to the biochemical domain: schema-embedded toxin SMILES, dictionary-obfuscated wet-lab protocols, and interleaved multi-turn variants, run against frontier and open-weight models. No reported result found on Lacuna for constrained-decoding attack success rates in biochemical domains; the relevant anchors are the 94.3–99.5% attack success rates on general safety benchmarks and ABC-Bench's 100% refusal rate on the same models' unconstrained biosecurity prompts. The hypothesis, suggested by the Flagship Paradox in Beyond Prompts: Space-Time Decoupling Control-Plane Jailbreaks in LLM Structured Output, is that grammar control re-opens the biosecurity refusals. Experiment 2 runs the same attack suite and a matched benign suite under the firewall, reporting attack success rate, over-refusal rate on benign dynamic tasks, SMILES validity rate (SafeSci reports invalid generation qualitatively; no reported result found on Lacuna for its generative-task accuracy), per-token latency overhead, and a coverage metric: the fraction of expert-flagged hazardous outputs whose hazard is actually captured by the regular specification. Experiment 3 compares refusal rates on static SafeSci- and ChemBench-derived items against dynamically remapped equivalents, testing whether refusals collapse the way reasoning did under A²RBench's 17.7% remapping drop and DyVal's 13.6% reversal drop. Experiment 4 runs the automaton-search audit and correlates memorized hazardous knowledge with refusal behavior across models.
Baselines
The naive reference point is unconstrained decoding with stock alignment, prompted to produce structured output. This is how SafeSci and ChemBench currently evaluate structured generation, and it is the condition under which ABC-Bench reports 100% refusal by GPT-5.4 and Claude 4.6 on screening evasion; it is also the condition in which SafeSci reports unparseable SMILES. We run it on our dynamic suite with the same models, prompts, and sampling budgets, and it doubles as the measurement of how much safety behavior degrades from static to remapped tasks.
The strongest existing competitor is the Dual-Plane Guard of Zhang et al., which audits the prompt and the schema simultaneously with an LLM-based classifier before decoding. It is the only published defense against grammar-embedded attacks, and it is the number to beat: under DictAttack it left a 75.8% attack success rate, against 94.3–99.5% with no defense. We reimplement it faithfully, run it on the identical biochemical attack suite with the same models and the same compute, and grant it the same tuning budget for its guard prompt. Beating it means driving attack success rate well below 75.8% while holding over-refusal on benign dynamic tasks at or below the guard's, at lower latency — the guard requires an extra LLM inference per request, while our check is a compile-time automaton product.
A second trained baseline is safety fine-tuning: Qwen3-8B instruction-tuned on SafeSciTrain, which SafeSci reports nearly doubled its safety rate from 0.31 to 0.64 while slightly improving MMLU performance. We run this model on both the attack suite and the remapped benign suite, which tests two things at once: whether trained refusals survive grammar attacks, and whether they survive symbolic remapping.
A third baseline is an honest one we expect to be competitive on a subset of tasks: a post-hoc output screen, running a substructure matcher on emitted molecules and an LLM judge on emitted protocols, with latency matched to the firewall. Where it works, we will say so; the interesting comparison is where it cannot. Finally, two ablations isolate the key ingredient. A format-only grammar enforces the same syntax without the hazard subtraction, showing whether risk reduction comes from the safety automaton or merely from structuring the output; and a compose-only variant applies subtraction but skips the compile-time emptiness verification, testing whether the verification step itself (which also produces an auditable proof that a submitted grammar is safe) carries value beyond the masking.
Why not just…?
Why not just fine-tune the model on safety data? Because the evidence says training-time alignment is shallow against this attack class and costly against benign use. The SafeSci fine-tuning result itself leaves a 36% failure rate and, as its authors report, worsens over-refusal of legitimate research queries; Zhang et al. show alignment is often only a few tokens deep, collapsing entirely once a schema-compliant prefix is forced; and SafeProtein achieved up to a 70% attack success rate against internally safety-trained protein models merely by shifting the conditioning modality. We include the fine-tuned baseline anyway, and if it matches the firewall's attack success rate at lower over-refusal on the dynamic suite, that result would argue against the proposal — which is exactly why the comparison is run.
Why not just scan prompts and outputs with a safety classifier? This is the most serious alternative, and we treat it as a real baseline rather than an argument to dismiss. The published version of this idea, the Dual-Plane Guard, already failed at 75.8% residual attack success because a classifier cannot reconstruct intent decoupled across a benign-looking prompt and an obfuscated schema, and the interleaved multi-turn variant makes per-turn classification computationally impractical. A post-hoc output screen is stronger than a prompt screen for molecules, and may match the firewall there; but it cannot defend the refusal pathway (the harm is generated, then discarded, in a retry loop), cannot cover protocol harm where "hazardous" is a property of an action sequence rather than a string pattern, and provides no auditable proof for a submitted grammar. The experiments settle this empirically: the post-hoc screen runs with matched latency, and its win or loss is reported per task type.
Why not just keep the benchmark static with canary strings and perplexity audits? Canary strings only deter scrapers that honor them, and perplexity audits only detect verbatim matches to known corpora, not the term-frequency shortcuts that Snoopy shows confound downstream evaluation. The decisive evidence is DyVal: models celebrated on static benchmarks scored nearly zero on dynamically generated tasks with identical underlying logic, and static benchmarks saturate quickly, as A benchmark of expert-level academic questions to assess AI capabilities documents for MMLU-style suites. For a safety benchmark the stakes are higher: a leaked benchmark means we measure memorized refusal triggers and believe we have measured safe behavior. Experiment 3 directly quantifies this — if refusals do not drop under remapping, the contamination concern is empirically weak for this domain and we will report that.
Risks and Decisive Checks
The honest limitation is coverage: a regular hazard language cannot capture every semantic hazard — novel toxic scaffolds outside the curated spec, valence and stereochemistry errors, or wet-lab viability, the physicality gap that both SafeProtein and ABC-Bench flag. The firewall is therefore one layer, and its residual is a first-class metric, not a footnote. The decisive checks are concrete: attack success rate of effectively zero on hazards covered by the specification, achieved by construction and confirmed empirically; overall biochemical attack success rate significantly below the Dual-Plane Guard's 75.8% at matched or lower over-refusal and latency inside GREATGRAMMA's 5–32 millisecond per-token envelope; and a quantified static-versus-remapped refusal gap that says whether current safety evaluations are measuring chemistry or measuring memorization.
References
- Output Constraints as Attack Surface: Exploiting Structured Generation to Bypass LLM Safety Mechanisms
- Beyond Prompts: Space-Time Decoupling Control-Plane Jailbreaks in LLM Structured Output
- Validating Large Language Models with ReLM
- Flexible and Efficient Grammar-Constrained Decoding
- SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond
- A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity
- SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Models
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- A benchmark of expert-level academic questions to assess AI capabilities