When a language model fills a structured output from retrieved documents, it emits two kinds of tokens. Some are scaffolding: braces, keys, and keywords that the grammar dictates. Others are facts: an entity name, an API argument, a table identifier. This is the case for a JSON record, an API call, or a list of entities. Retrieval noise only threatens the second kind. Current decoders handle the two problems separately. Grammar-constrained decoding forces the output to obey a formal grammar by hiding every token that would break it. Adaptive contrastive decoding measures, token by token, how much the retrieved text sharpens the model's prediction. It then decides how far to trust that text over what the model already knows. Nobody has combined them. The obvious combination, running both and multiplying their effects, measures the model's confidence over the wrong set of tokens. It also spends its trust decision on scaffolding where there is no decision to make.
We propose a decoder that reads the grammar's current state to tell scaffolding apart from factual slots. Inside a slot, it decides how much to trust retrieval versus memory by measuring uncertainty only over tokens the grammar allows. The same mechanism softens retrieval-constrained decoding, which restricts answers to a prefix tree of retrieved candidate entities. When the gate distrusts retrieval, the model may leave the tree and answer from memory. That removes the hard-trie failure where a retriever that misses the right entity makes the right answer impossible to emit. Because the second forward pass, without retrieved context, is only needed inside slots, the method should also cost less than running adaptive contrastive decoding on every token.
Core Claim
Retrieval-versus-memory gating under a grammar works better when two things hold. The gate is computed on the grammar-admissible token set, renormalized. And it is applied only in grammar states that open a semantic slot. Under retrieval noise, this beats both hard retrieval constraints and naive "adaptive contrastive decoding, then mask" on factual accuracy. It also keeps 100% syntactic validity and runs the unconditioned pass on a fraction of tokens.
The underlying point is simple. Outside slots, the grammar usually leaves one or a handful of admissible tokens, so both distributions collapse and entropy carries no signal about retrieval quality. Inside slots, the model's full-vocabulary distribution may put much of its mass on tokens the grammar forbids, such as a natural-language preamble. Entropy over the full vocabulary then measures "how much does this model want to chat" rather than "does the context resolve the fact."
Research Program
The gated distribution. Let $A_t$ be the set of tokens the grammar admits at step $t$. Let $p^c_t$ and $p^0_t$ be the model's next-token distributions with and without retrieved context. Both are restricted to $A_t$ and renormalized. Let $\tilde H(\cdot)$ denote entropy divided by $\log |A_t|$, so a uniform distribution scores 1 regardless of how many tokens remain. In slot states we use an admissible-set version of the gate from Adaptive Contrastive Decoding:
$$\alpha_t = \frac{\tilde H(p^0_t)}{\tilde H(p^0_t) + \tilde H(p^c_t)}, \qquad \log p_t(v) \propto \log p^0_t(v) + \alpha_t\bigl(\log p^c_t(v) - \log p^0_t(v)\bigr), \quad v \in A_t .$$
When $|A_t| = 1$ the token is emitted with no model call. In non-slot states with $|A_t|>1$, we decode from $p^c_t$ alone. We will also test the Rényi-divergence and peakedness features of CoCoA computed on the same restricted support. The entropy gate is the default because it is the simplest thing the contribution must beat.
Soft retrieval tries. For entity slots, we follow Retrieval-Constrained Decoding, which builds a prefix trie over retrieved knowledge-base surface forms and masks everything else. We keep the trie only for the context branch: $p^c_t$ is restricted to $A_t \cap \text{Trie}_t$. The parametric branch, $p^0_t$, is restricted only by the grammar. When retrieval is good, the context branch is sharp, $\alpha_t \to 1$, and we recover hard-trie behavior. When the trie holds only distractors, the context branch stays flat or conflicts, and the gate shifts mass to the model's own continuation. A small backoff rule handles the moment the generated prefix leaves the trie: after that, the context branch reverts to grammar-only masking.
Slot detection and compute. Slot states are read directly from the grammar. Nonterminals for string values, identifiers, numbers, or entity references are marked once per grammar, which takes minutes of annotation per schema. For masking we build on GREATGRAMMA, whose reported online overhead is 5 to 32 ms per token. Slot boundaries map onto its lexer states.
The unconditioned branch only needs a prefix cache when a slot opens. We therefore prefill it in one batched pass at slot entry, rather than stepping it every token. We will report the fraction of tokens that receive two passes and the resulting wall-clock time, against the every-token doubling that the file attributes to adaptive contrastive decoding.
Analysis. Beyond accuracy, we will log the gate value against whether the gold answer was present in retrieval. That checks whether the gate is detecting retriever failure or merely tracking token frequency. We will also measure how often the admissible-set entropy and the full-vocabulary entropy disagree on the sign of the gate's move. If they rarely disagree, the restricted-support idea is cosmetic.
Baselines
The naive reference point is unconstrained greedy decoding with retrieved documents in the prompt and a formatting instruction. This is the setup used for JSON workflows in Reducing hallucination in structured outputs via RAG. We also run plain grammar-constrained greedy decoding with the same prompt, which isolates what syntactic validity alone buys.
The strongest existing method for entity questions is hard retrieval-constrained decoding. On YAGO-QA it reaches 46.0% F1 with Llama-3.1-70B, against 32.3% for vanilla decoding. On KAMEL it reaches 32.0% versus 8.7% (Retrieval-Constrained Decoding). It is the strongest competitor because it already turns retrieval into a constraint, and it wins precisely when retrieval is clean. We run it with the identical retriever, candidate count, and model. Beating it means matching its F1 within one point on the clean split while exceeding it on splits where the gold entity is removed from the candidate set.
The strongest adaptive-gating method is Adaptive Contrastive Decoding. It reports 42.06 exact match on the noisy TriviaQA subset with Llama 2 7B, versus 24.75 for static context-aware decoding. CoCoA reports an 18.0-point gain over AdaCAD on NQ-swap. No absolute NQ-swap number for it appears on Lacuna, so we will reproduce it rather than quote it. For structured tasks, both are run as "gate over the full vocabulary, then apply the grammar mask," which is the natural composition. No reported result was found on Lacuna for any adaptive contrastive decoder combined with grammar constraints, so these runs are new.
For API calls, Gorilla's fine-tuned model reports 71.68% zero-shot accuracy on HuggingFace APIBench and a 6.98% hallucination rate on TorchHub (Gorilla). RAFT reports 74.00% on HuggingFace (RAFT). These serve as the fine-tuning reference. We also run our decoder on top of the released Gorilla model, since the two approaches are complementary.
Two ablations isolate the key ingredients. The first keeps the slot gating but computes entropy over the full vocabulary. The second keeps restricted-support entropy but gates on every token, ignoring slot state. All decoding methods share the same retriever outputs, prompts, models, and a matched hyperparameter search over the gate temperature on a held-out development split. Fine-tuned models are taken as released, not retrained.
Why not just…?
Why not just run adaptive contrastive decoding and then apply the grammar mask? This is the real threat. If most of the model's probability mass already sits on admissible tokens, full-vocabulary and admissible-set entropy will move together, and the naive composition will match ours. The first ablation tests exactly this. We expect the gap to be largest at slot openings, where models often spread mass over quoting, preamble, and formatting variants the grammar forbids. If the gap is under one point across tasks, we will report the contribution as the soft trie and the compute saving, not the renormalization.
Why not just filter or rerank retrieved documents before generation? Pre-generation filtering, as in extract-then-generate pipelines like Ext2Gen, helps when distractors are separable from the gold passage. It cannot recover an answer the retriever never returned, and that is the failure that makes hard tries brittle. We include a reranker-filtered variant of retrieval-constrained decoding on the gold-removed split. A gain for our method there would show that decoding-time fallback is doing work filtering cannot.
Why not just fine-tune with distractors, as RAFT and Gorilla do? Fine-tuning works, as their APIBench numbers show. But RAFT reports that the best distractor fraction varies by domain and does not adapt when test-time retrieval quality shifts. Grammar-Aligned Decoding found that task-fine-tuned models still suffer grammar-induced distortion under constrained decoding. Our test applies the decoder on top of the fine-tuned Gorilla model while varying the number of injected distractors beyond the training regime. If fine-tuning alone stays flat as noise grows, our method adds nothing there, and we will say so.
Why not just widen the trie with the model's own unconstrained guess? Adding the model's free answer as an extra trie candidate is a cheap fallback. We run it as a baseline. Its weakness is that it requires a full extra generation per slot. It also offers no principled way to choose between the model's guess and retrieved candidates, beyond the same likelihood comparison our gate makes continuously.
Proposed Experiments
Entity recall uses YAGO-QA and KAMEL with the retrieval-constrained setup. We create three retrieval conditions: as retrieved, gold candidate removed from the trie for a controlled fraction of questions, and swapped entities injected in the manner of NQ-swap. Metrics are F1, including multi-answer lists separated by a separator token. The decisive check is the gold-removed split. Hard tries are capped there by construction, and our decoder should recover close to vanilla parametric F1 while keeping hard-trie F1 on clean questions.
Structured outputs use APIBench with AST-match accuracy and hallucination rate, plus NQ and TriviaQA questions reformulated to require a JSON answer object. We add distractors and poisoned passages generated with the corpus-poisoning codebase. Metrics are exact match on the slot value, schema validity, and wall-clock latency per output. Unconstrained TriviaQA noisy and NQ-swap runs confirm that our restricted-support gate does not lose to adaptive contrastive decoding when there is no grammar, since there the two methods should coincide.
The project succeeds if three things hold. Our decoder must beat retrieval-constrained decoding on the noisy splits without losing on clean ones. It must beat "gate then mask" in the first ablation by a margin larger than seed variance. And it must keep schema validity at 100% while calling the unconditioned pass on well under half of tokens. If the first ablation shows no gap, the contribution narrows to the soft trie and the efficiency gain.
References
- Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts
- CoCoA: Confidence- and Context-Aware Adaptive Decoding for Resolving Knowledge Conflicts in Large Language Models
- Retrieval-Constrained Decoding Reveals Underestimated Parametric Knowledge in Language Models
- Flexible and Efficient Grammar-Constrained Decoding
- Reducing hallucination in structured outputs via Retrieval-Augmented Generation
- Gorilla: Large Language Model Connected with Massive APIs
- RAFT: Adapting Language Model to Domain Specific RAG
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Grammar-Aligned Decoding