When a language model fills a structured record, such as a JSON object with twenty fields, from a pile of retrieved documents, each field usually depends on only one or two of those documents. The standard pipeline still lets every generated token attend to every document. The evidence for the invoice date therefore acts as distracting noise while the model writes the invoice total, and the reverse is also true. We propose to use the grammar that already enforces the output format as a map of what the model is currently writing. Grammar-constrained decoding restricts the model at each step to tokens that keep the output valid. Its parser always knows which schema field is being filled. At each field boundary, the decoder switches which cached documents it may attend to. Each document is encoded once, in isolation, and its cached attention state (the key-value, or KV, cache) is shared across every field and every request that uses it. What changes from field to field is a cheap attention mask over these shared caches, not a copy of any memory.
The claim is that for structured outputs, the right unit of noise isolation is the schema field, not the whole request, and that the grammar state gives field boundaries for free. Existing noise defenses filter or prune documents once per query. For a multi-field record, a per-query filter has to keep the union of all fields' evidence, and that union is itself the noise for each individual field. If the claim holds, field-level accuracy under request-level pruning should fall as the number of fields with separate evidence grows, while field-scoped attention should stay roughly flat.
What exists and where this sits
Three prior systems each connect two of the three directions, and none makes the connection proposed here. Superposition Prompting supplies the memory layout we reuse. It encodes each retrieved document in its own attention path after a shared preamble, which stops documents from attending to each other. It then prunes low-saliency paths before generating, and reports a 43% accuracy gain over naive concatenation on NaturalQuestions-Open along with up to 93x less compute. Its pruning happens once per query and its outputs are free-form text. We keep the isolated paths but make selection repeat at every field, driven by the grammar.
SGLang shares KV caches across requests through a prefix tree (RadixAttention). It also uses compressed finite-state machines to skip past predictable schema tokens such as field keys. It treats every cached token as equally usable, so a distracting document stays in view for the whole decode. Retrieval-Constrained Decoding shows that moving retrieved candidates out of the prompt and into an output constraint avoids prompt pollution, raising LLaMA-3.1-70B F1 on KAMEL from 8.7% to 32.0%. Its constraint is a flat list of entity strings, not a nested schema, and it never conditions input attention on the output state. That last idea is the one we add.
Two lines of work motivate the proposal by documenting the failure it targets. Magic Mushroom finds that "distracting" noise, meaning topically relevant text with altered facts, hijacks attention in early layers. It reports that LLaMA-3.1-8B correctness falls from 84.1% to 18.0% under full noise. It also finds that active denoisers become overly skeptical when context is clean, which is a constraint any design must respect. UDA shows GPT-4 on FinHybrid falling from 45.9 with retrieval to 0.4 when given whole documents. It also shows that oracle context chunks raise scores by up to 60%, so evidence selection is where the headroom lies.
Grammar engines supply the constraint machinery. SynCode eliminates JSON syntax errors using an incremental parser and precomputed token masks. GREATGRAMMA and Gram2Token make such masks faster and GPU-resident. All three decode a single stream against a fixed context. None exposes its parser state to the attention mechanism, which is the hook this proposal uses.
Core mechanism
Documents are encoded as isolated paths in the style of Superposition Prompting. All paths share one position range after the preamble, so any subset can be attended to without shifting rotary positional encodings. This sidesteps the rotary-embedding complications that LRAgent reports for more aggressive cache sharing.
Decoding runs under a JSON-schema grammar. Whenever the parser enters a new value slot, it emits a field context. This context consists of the key path, such as invoice.line_items[2].amount, the schema description of that field, and the values already generated in the enclosing object.
The field context's tokens are forced anyway and are prefilled in one batched pass, as with SGLang's fast-forwarding. From that pass we read how much attention these tokens place on each document path. The top-scoring documents above a threshold form that field's evidence set. While the value tokens are generated, the decoder attends to the preamble, the JSON generated so far, and the evidence set only.
Two alternative scorers will be compared against attention mass. One is the query-generation likelihood from Distraction-aware Answer Selection. The other is a small cross-encoder.
When no document clears the threshold, the grammar opens a null branch. This mirrors DAS's "unanswerable" instruction and gives the model a legal way to abstain rather than invent a value.
Because the JSON generated so far stays visible, later fields remain conditioned on earlier ones. Examples include a currency code conditioning an amount, or a bridge entity conditioning a multi-hop answer. For arrays of unknown length, selection repeats for every new element. Neither behavior is available to a pipeline that splits the schema in advance.
Memory overhead is one bitmask over document paths per active field per sequence. Document KV caches can be shared across fields, across concurrent requests over the same corpus, and across retries, and they can be managed with a radix-tree cache.
Research Program
The first stage is diagnostic and addresses an open question in its own right: does grammar-constrained decoding help or hurt a distracted model? We will take Magic Mushroom's four noise classes at controlled ratios from 0% to 100%. For each ratio we will measure value accuracy for free-form answers against answers constrained to a JSON schema, holding the documents fixed.
The second stage builds field-scoped attention on SGLang's runtime, with a mask engine in the style of SynCode or GREATGRAMMA, and with per-path attention masks passed to the attention kernel.
The third stage builds evaluation sets where evidence is spread across fields, since no existing benchmark has this property. From UDA's FinHybrid and TatHybrid we will group the expert questions about one document into a single multi-field record, keeping each question's gold evidence chunk as that field's evidence. From HotpotQA we will build two-field records containing the bridge entity and the answer, which have a real dependency between fields. From Magic Mushroom's customizable generator we will build synthetic records with 2 to 32 fields. Each field gets its own gold document plus field-targeted distracting noise, which lets us vary the number of fields and the noise level independently.
Baselines
The naive reference point is linear concatenation of all retrieved documents, followed by JSON-constrained decoding with SynCode masks. It shows how much grammar enforcement alone buys: syntax becomes valid, but it does nothing for distraction. Every method uses the same retriever output, the same model (LLaMA-3.1-8B, plus one larger model), the same schema grammar, and greedy decoding.
The strongest existing method is Superposition Prompting's parallel encoding with query-level path pruning, followed by grammar-constrained decoding over the surviving paths. This is exactly the straightforward composition at the intersection that Gap 1 of the research file describes. Its pruning budget, meaning how many paths are kept, is tuned on a held-out split with the same number of trials we give our threshold.
A second strong baseline is per-field decomposition. Each field is run as a separate Superposition-Prompting query with its own pruning, and the answers are assembled into JSON. Document caches are shared through SGLang prefix caching, so memory is comparable.
A third baseline is reranker filtering with a cross-encoder followed by constrained decoding over the kept chunks. We sweep its threshold so that it gets a fair chance in both the clean and noisy regimes.
For reference numbers, UDA reports 45.9 for GPT-4 with retrieval on FinHybrid, and Magic Mushroom reports 18.0% correctness for vanilla RAG on LLaMA-3.1-8B at full noise. On our field-structured versions of these tasks, no reported result was found on Lacuna for any baseline, so all will be measured by us.
Beating the strongest baseline means higher record-level exact match (all fields correct) at matched or lower latency and KV memory. It also means the gap should widen with the number of fields and with the distracting-noise ratio, while clean-context accuracy stays within one point.
The key ablation keeps everything identical but performs selection once per request, taking the union of per-field scores. This isolates field-level scoping as the active ingredient. A second ablation re-selects documents every fixed number of tokens instead of at grammar boundaries, to test whether field boundaries matter or simply re-selecting often is enough. Random evidence sets of the same size, and oracle gold evidence, bracket the achievable range.
Why not just…?
Why not just run Superposition Prompting and add grammar-constrained decoding on the final decode? This is the obvious combination, and it is our main baseline. Its pruning decides once which documents survive. For an N-field record it must keep roughly the union of N fields' evidence, and each field's evidence is distracting noise for the others. Magic Mushroom's result that distracting noise in particular hijacks attention predicts that this composition degrades as N grows. The field-count sweep tests that prediction directly. If the curve is flat for this baseline, our central claim is wrong.
Why not just split the schema into one query per field? This is a real threat, because it achieves field-level isolation with no new machinery. It fails in two cases. It loses conditioning on earlier generated values, which matters for dependent fields such as HotpotQA bridge-then-answer or currency-then-amount. It also cannot be planned ahead for arrays of unknown length. On flat schemas with independent fields we expect it to roughly tie our method. If it also ties on the dependent-field and variable-array subsets, then the contribution reduces to the empirical finding that field-level isolation matters, together with a cheaper cache-sharing implementation. We will report that result honestly.
Why not just filter with a strong reranker before constrained decoding? Rerankers score documents against the request, not against each field, so they face the same union problem. Magic Mushroom also shows that tuned denoisers discard useful context when noise is low. The reranker threshold sweep across noise ratios shows whether any single threshold works in both regimes.
Why not just use a long-context model with a JSON grammar? UDA's FinHybrid collapse from 45.9 to 0.4 without retrieval indicates this is the weakest option for numeric fields. It is still included in the diagnostic stage at matched token budgets.
Decisive checks
The experiment most likely to change our minds plots record-level and field-level accuracy against the number of fields, from 2 to 32, at a fixed 50% distracting-noise ratio on the synthetic set. The claim predicts that the request-level ablation and Superposition-plus-grammar decline while field-scoped attention stays near its two-field accuracy. A flat curve for the baselines refutes the premise.
A second check is the clean-context regime at 0% noise. There, field-scoped attention must match concatenation to within one point, or it is simply another over-skeptical denoiser.
A third check is dependent fields on HotpotQA records, which separates our method from per-field decomposition. The fixed-interval re-selection ablation tells us whether the grammar's field boundaries do real work.
Throughout, we will confirm that schema validity on JSON-Mode-Eval stays at the 100% level SynCode reports. We will also report throughput and KV memory against SGLang's prefix-caching baseline, so that any accuracy gain is not bought with hidden compute.
References
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- SGLang: Efficient Execution of Structured Language Model Programs
- Retrieval-Constrained Decoding Reveals Underestimated Parametric Knowledge in Language Models
- Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systems
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- SynCode: LLM Generation with Grammar Augmentation
- Flexible and Efficient Grammar-Constrained Decoding
- Gram2Token: Enabling Run-time GPU-Native Grammar-Constrained Decoding for LLMs
- LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering