Hearing the Brackets: Prosody-Anchored Choice Points in Grammar-Constrained Speech-to-Structure Decoding

When a speech model turns spoken mathematics into LaTeX, or a spoken request into slot-value pairs, two separate things can go wrong. The output can be malformed, with an unclosed brace or a missing field. Or it can be well-formed but carry the wrong structure: "one over x plus two" written as $\frac{1}{x}+2$ when the speaker meant $\frac{1}{x+2}$. Grammar-constrained decoding fixes the first problem. It is an inference-time filter that blocks any next token a formal grammar would reject. It does nothing about the second. We hypothesize that the second problem is where speech language models are most sensitive to prompt wording. Once format errors are removed, the remaining answer flips across semantically equivalent instructions should cluster at a small number of grammar "choice points", the decoding steps where the grammar allows structurally different continuations. At those steps the text prompt competes with weak acoustic evidence and often wins.

The proposal has two parts. The first is a measurement: split prompt sensitivity in speech-to-structure tasks into a format component and a structure component, and test whether the structure component concentrates at choice points. The second is an intervention. At choice points only, we mix the constrained next-token distribution with a small prompt-independent classifier that reads prosody (pause length, pitch reset, syllable lengthening) from the aligned stretch of audio. Speakers use prosody to mark syntactic grouping. Because this classifier never sees the prompt, it should damp prompt-driven flips exactly where they occur.

Core Claim

Grammar-constrained decoding applied to an end-to-end speech LLM should remove nearly all format-driven prompt sensitivity, but not the structure-driven part. That residue is localized at choice points and can be reduced with a prompt-independent acoustic signal applied only there. The claim is falsifiable at each step. If constrained decoding alone already removes most of the sensitivity, or if the speech encoder already uses prosody well enough that the acoustic term adds nothing, the experiments will show it, and the measurement study stands on its own.

Grounding in Prior Work

Three groups of work motivate the idea. On constrained decoding, Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning masks tokens with incremental context-free parsers. On Penn Treebank, unconstrained LLaMA produced valid trees about 64% of the time, while constrained decoding reached 100%. Flexible and Efficient Grammar-Constrained Decoding makes masking fast (5–32 ms per token) and handles subword tokens that cross grammar boundaries. Both work on clean text only and say nothing about which valid structure gets chosen. We reuse their machinery through the Transformers-CFG implementation.

On prompt sensitivity, ProSA shows that instance-level sensitivity tracks low decoding confidence, and it defines the PromptSensiScore over 12 templates. Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection shows that speech LLMs are prompt-sensitive and fixes this on the input side with a learned prompt projector. On Contact Center, the file reports the worst-versus-best prompt WER gap moving from 13.00% to 11.81%. That work studies free-form transcription only. Our intervention works on the output side and is localized. ProSA's confidence finding predicts our localization hypothesis: masking renormalizes probability over valid tokens, which should sharpen most steps and leave genuinely ambiguous structural steps flat.

On speech-to-structure, Speech-to-LaTeX shows that end-to-end SALMONN (17.5% CER on S2L-equations) beats a Whisper→Qwen cascade (27.2%). Even so, equation CER inside sentences stays around 40%. Its AST parser is used only for offline normalization and never during decoding. Speech-based Slot Filling using Large Language Models enforces JSON on SLURP through prompting alone and never measures variance across prompts. For the acoustic term we borrow two techniques. Learning Alignment for Multimodal Emotion Recognition from Speech shows that soft attention can align acoustic frames to words without forced alignment. Duplex Conversation shows that pitch and intensity separate boundaries from pauses. That paper detects dialogue turns; we redirect the same cue to grouping boundaries inside a parse.

Two strands of work pull against the proposal. Template work (ASPIRO, GAMMA) treats schemas as scaffolds that help. A separate line of results, collected in the file's overview of constrained decoding, reports a "reasoning tax" that degrades reasoning under rigid formats. Our tasks are transcription-like and use no chain-of-thought, so we expect the tax to be small. We measure it rather than assume it.

Research Program

Grammars. We write an EBNF grammar for the LaTeX math subset used in S2L and a slot-annotation grammar for SLURP. The SLURP grammar is input-dependent in the style of Geng et al.: it is restricted to the 55 entity types and to slot values that appear in the N-best hypotheses.

Choice points. A decoding step counts as a choice point when the grammar-valid set contains both a group-closing token (}, end-of-superscript, end-of-slot) and a group-continuing token, and each holds at least τ of the renormalized probability mass. These are the steps that decide fraction scope, exponent extent, and slot span.

Measurement. Using SALMONN and ProSA's 12 templates, we compute three quantities with and without constrained decoding:

  1. instance-level PromptSensiScore;
  2. the share of cross-prompt output disagreements traced to format failures versus structural differences, where structure is compared by AST tree edit distance;
  3. the fraction of structural disagreements whose first divergence falls at a choice point.

Intervention. At choice points only, the final distribution over the valid choices $c$ is

$$P(c) \propto P_{\text{LM}}(c \mid \text{prefix}, \text{prompt})^{1-\lambda}\, P_{\text{ac}}(c \mid a_t)^{\lambda},$$

where:

  • $P_{\text{LM}}$ is the grammar-masked speech-LLM distribution;
  • $a_t$ holds prosodic features (88-dimensional eGeMAPS as in MUSER, plus pause duration) taken from the audio window aligned to step $t$;
  • $\lambda\in[0,1]$ is tuned on a dev split.

The window is located from the LLM's attention over speech-encoder frames. We validate this alignment against Whisper word timestamps on a subset. $P_{\text{ac}}$ is a small classifier trained on the S2L and SLURP training splits, with labels derived from gold trees: close versus continue at each aligned position.

Baselines

The naive reference is unconstrained SALMONN with a single fixed prompt. It is the strongest reported system on S2L-equations, at 17.5% CER (Speech-to-LaTeX). We rerun it under all 12 templates to get both its best-prompt CER and its spread. For S2L-sentences, the reported reference is about 40% equation CER.

The strongest competitor on the sensitivity question is grammar-constrained SALMONN with no acoustic term. It already hears the raw audio, so it may already exploit prosody. It shares weights, prompts, grammar, and decoding settings with our system and differs only in $\lambda=0$. To beat it, we need a statistically significant reduction in structural PromptSensiScore and in choice-point flip rate, with equal or lower AST-normalized CER.

For sensitivity reduction specifically, the strongest competitor is majority voting over all 12 prompts. It costs 12 times the compute. We report it at matched compute (a 12-way vote versus our single pass) and also as a cost-unconstrained ceiling.

The input-side competitor is the prompt projector of Burdisso et al., retrained on S2L with the same data and an equal tuning budget, both alone and combined with constrained decoding.

The cascade competitor is the Whisper→Qwen pipeline (27.2% CER reported) with constrained decoding added.

On SLURP, the references are Vicuna-13B with knowledge injection and LoRA at 63.8 SLU-F1 (Speech-based Slot Filling) and the best pipeline at 70.84 SLU-F1 (SLURP). No reported result found on Lacuna for PromptSensiScore on any speech benchmark, so every sensitivity number here is new.

Three ablations isolate the key ingredients:

  1. replacing $P_{\text{ac}}$ with a classifier over ASR words only, which isolates acoustics from lexical cues;
  2. applying the mixture at every step instead of only at choice points, which tests localization;
  3. using prosody-destroyed audio at inference, with F0 flattened and pauses equalized.

Why not just…?

Why not just use constrained decoding on SALMONN and stop? This is the most serious threat. The speech encoder may already encode prosody, and masking alone may sharpen choice points enough. The $\lambda=0$ baseline settles it directly. If the choice-point flip rate under $\lambda=0$ is already near the floor, we drop the intervention and publish the decomposition, which no paper in the file reports. The prosody-destruction test also tells us whether SALMONN uses prosody at all: if its outputs do not change when prosody is flattened, the encoder ignores the cue.

Why not just ensemble over prompts? Voting reduces variance by brute force, and at 12 times the cost it may match or beat us on PromptSensiScore. It cannot fix a systematic scope error that every prompt shares, because voting only removes disagreement. The scope-accuracy check below separates "consistent" from "correct". We report matched-compute comparisons honestly.

Why not just fine-tune with prompt augmentation or a prompt projector? Input-side fixes changed the worst/best prompt WER gap only from 13.00% to 11.81% in Burdisso et al.. They also add no information the model lacks. If scope is acoustically underdetermined from the model's point of view, no prompt tuning resolves it. The projector-plus-constrained-decoding baseline tests whether the gains are complementary.

Why not just repair the output with a post-hoc AST parser? Repair can close braces, but it has to guess where they go, which is exactly the structural decision at issue. In S2L, AST normalization changed CER by only about 1% (Speech-to-LaTeX).

Decisive Checks

We build a scope-ambiguity subset of S2L. It contains items whose flat spoken word sequence admits more than one parse, identified by comparing the gold tree against the tree a left-to-right flat reading would produce. We split this subset into human-read and TTS audio.

The method is supported if four predictions hold:

  1. Constrained decoding drives the format share of disagreement to near zero.
  2. Most of the remaining structural flips begin at choice points.
  3. The acoustic term raises scope accuracy and lowers flip rate on human audio more than on TTS audio. TTS audio should carry weaker grouping prosody, though that is an assumption we check by inspecting it.
  4. The gain disappears under prosody destruction and under the words-only classifier.

Failure of the third or fourth prediction means prosody is not the operative signal, and we would say so. We also track CER on the full S2L test sets, slot SLU-F1 on SLURP, and per-token latency to rule out hidden costs.