When a language model fills a structured record, such as a JSON object of entities or a slot-filled intent, from text that looks nothing like its training data, two failures compete. The first is broken structure, where the output will not parse. The second is quiet substitution: the model emits a clean, familiar word in place of the odd string that was actually in the input. A handle like "@dj_kreme" becomes "DJ Kream", or a lab-specific gene alias becomes its textbook name. Grammar-constrained decoding forces the output to follow a formal grammar by blocking every next token that would break it, and it fixes the first failure. A pointer-generator is a decoder that can either generate from its vocabulary or copy directly from the input by following its attention, and it attacks the second failure. No existing system does both at once in a way that makes copying exact and makes the choice of which span to copy depend on the input rather than on the model's prior over familiar words.
The proposal has two parts. The first builds, for each input, a small automaton that accepts exactly the substrings of that input, and plugs it into a context-free grammar as the definition of an entity value. The second uses the automaton's state to restrict a training-free pointer to the input positions consistent with what has been copied so far, and mixes that pointer into the constrained next-token distribution. The claim is specific and falsifiable. Once the grammar already guarantees verbatim copying, the remaining error under domain shift is span selection, and a pointer computed from attention selects spans better than the model's own masked token probabilities, by a margin that grows with how far the input is from the model's training distribution.
Why the obvious combination is not enough
The literature already contains most of the pieces. Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning masks LLaMA's logits with an incremental context-free grammar parser. Its input-dependent grammars, which are grammars rebuilt for each input so that only that input's candidate entities are allowed, lift entity disambiguation accuracy from 54.1% to 80.3%. But the candidates must be pre-extracted from a knowledge base, and the authors note that entities absent from it cannot be produced. The paper also includes no copy mechanism. Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition and its spoken-language-understanding extension End-to-end Spoken Language Understanding with Tree-constrained Pointer Generator do restrict a pointer with a symbolic structure. That structure is a prefix tree over a fixed biasing list rather than a grammar over outputs. The pointer confuses morphological neighbours such as "STRENGTH" and "STRENGTHEN", and turning up the biasing weight trades common-word precision for rare-word recall. LargePiG: Your Large Language Model is Secretly a Pointer Generator shows that an LLM's last-token attention can serve as a pointer with no training, using a copy gate from the divergence between intermediate and final layers, but it has no structural constraints. PG-GSQL combines pointing with schema-guided pruning for cross-domain text-to-SQL, but it copies from previous SQL queries, not from the noisy input, and uses heuristic filters rather than a grammar engine. These papers supply methods: the input-dependent grammar, the structure-restricted pointer, and the training-free pointer. They also motivate the proposal by exhibiting the failure modes.
A reviewer would immediately point out that the grammar-only version is almost trivial. With byte-level tokenizers there is no true out-of-vocabulary word for an LLM, and one could write an input-dependent grammar whose entity rule is "any substring of the input". That version is the central baseline and the central ablation here, and it is the thing the proposal must beat. What the grammar alone cannot do is decide which substring to copy. Within the allowed set, the model still ranks candidates by its own token probabilities, which under domain shift are exactly the probabilities that favour familiar-looking spans. A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization found this substitution toward generic source-domain terms in cross-domain summarization, and it found an explicit pointing mechanism necessary to prevent it. CrossNER documents a typing analogue, the "subcategory trap": specialised target-domain entities get collapsed into generic source-domain categories, and the file's summary of CrossNER reports 84.81% person/artist confusion in the music domain. The contribution is therefore not "add a pointer to a grammar". It is the measured claim that span selection is where domain-shift errors survive grammar constraints, together with a mechanism designed to fix that.
Core mechanism
For each input $x$, we build its suffix automaton. This is a standard linear-size automaton that accepts exactly the substrings of $x$, and each of its states records the set of end positions in $x$ where the string read so far occurs. We expose it as a dynamic terminal, SPAN, inside an otherwise ordinary context-free grammar for the output record. A slot value may be SPAN (copied) or FREE (generated, for abstractive answers).
Subword tokens rarely line up with span boundaries: the copied string may tokenize differently in the output than in the input. We therefore handle alignment at the character level, following Flexible and Efficient Grammar-Constrained Decoding. That work composes the tokenizer's detokenizer with the lexer as a finite-state transducer and precomputes which tokens can realise which terminal sequences. The difference is that one of our terminals is rebuilt for every input, so the precomputation for SPAN must be incremental. Making this fast is a real engineering component, not a detail. We also add an optional word-boundary condition on span ends, which targets the prefix-confusion error reported for the tree-constrained pointer.
While decoding inside a SPAN, let $q$ be the current automaton state and $m_q(t)\in\{0,1\}$ the grammar mask for candidate token $t$. Let $S(q,t)$ be the input positions whose next characters spell $t$ given state $q$. Let $a_i$ be the attention mass the model's last token places on input position $i$, extracted as in LargePiG. The copy-state distribution is
$$P(t \mid q) \propto m_q(t)\left[(1-\lambda)\,P_{LM}(t) + \lambda \sum_{i \in S(q,t)} a_i\right],$$
where $P_{LM}$ is the model's ordinary next-token probability and $\lambda\in[0,1]$ is a single mixing weight. The weight $\lambda$ is tuned on source-domain development data only and never on target-domain data.
This is the same move as the tree-constrained pointer, with a structure that tells the pointer which positions are still live, except that the structure is a substring automaton nested in a grammar rather than a trie of a fixed list. Outside SPAN states the pointer is not used, so structural tokens are governed purely by the grammar.
Whether to copy or generate a slot value is not a hand-tuned gate. It is a grammar alternative chosen by the model's own probability of each branch's first token. This removes the global biasing-strength knob whose precision/recall tension the SLU paper reports. The pointer can only reallocate probability among input spans, so it cannot push common words out of generated text.
Research Program
The work has three stages. The first is the constraint engine: the suffix-automaton terminal, its composition with the token transducer, and the incremental mask computation. It will be built on the Transformers-CFG codebase (epfl-dlab/GCD). The target is per-token overhead within the 5–32 ms range reported for GREATGRAMMA on code grammars.
The second stage is the pointer and the controlled ablations that separate its effect from the constraint's. The third is a domain-shift analysis that tests whether the pointer's benefit tracks distance from the model's training distribution. That is the proposal's actual scientific claim.
All experiments use open-weight models (a LLaMA-family and a Qwen-family model at around 7B and around 30B parameters) with few-shot prompts and no fine-tuning of the main system. This keeps the comparison to training-free grammar-constrained decoding clean.
Baselines
The naive reference point is few-shot prompting with schema instructions and no masking. We report its parse validity alongside extraction scores, because Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning reports only about 64% structural validity for unconstrained LLaMA on Penn Treebank constituency parsing, against 100% with constraints. The same model, prompts, and shots are used everywhere. Any configuration that does not beat this reference on span F1 is uninteresting regardless of validity.
The strongest existing method is grammar-constrained decoding with an input-dependent grammar, in two forms. The first is the published form, with candidates from an external lexicon. The second is our span-only form, in which the entity rule accepts any input substring through the same suffix automaton but $\lambda=0$. The span-only form is the key ablation, because it differs from the proposal in exactly one ingredient, the pointer. Both forms share the grammar, prompts, and decoding budget (greedy decoding plus a beam of width 4 in a secondary run). The closest reported number is on SynthIE-text closed information extraction, where LLaMA-33B with grammar-constrained decoding reaches 36.0 F1 against 34.8 for supervised GenIE T5-base (Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning). We will reproduce that setting as an in-domain sanity check. The proposal must match it there, because the pointer should do no harm on clean text, and it must beat the span-only form on the shifted benchmarks. Beating means a span-F1 gain whose bootstrap 95% interval excludes zero on at least three of the five CrossNER target domains and on TweetQA extractive questions.
The third baseline is LargePiG's unconstrained training-free pointer with its layer-divergence gate (LargePiG). It is run on the same models and prompts, with outputs parsed leniently. It shows what the pointer achieves without the grammar.
The fourth is supervised adaptation. On CrossNER we will reuse the published domain-adaptive pretraining pipeline, which continues pretraining on unlabeled domain text before fine-tuning on the 100–200 labeled target examples. The file reports only its average gain of 2.16 F1 over no adaptation (CrossNER); no absolute per-domain F1 was found on Lacuna, so we rerun it. The training-free proposal is not expected to beat it everywhere. The comparison shows how much of the supervised gain is recovered with zero target labels.
On TweetQA, the reported fine-tuned BERT result is 69.6 BLEU-1 against a human score of 78.2 (TWEETQA). This is our external anchor, although our few-shot, span-or-free setting is not identical to theirs.
Why not just…?
Why not just prompt the model to copy terms verbatim and output JSON? Validity is the smaller problem. The Penn Treebank figure above shows prompting leaves about a third of structured outputs invalid even in-domain. More importantly, prompting gives no guarantee that a value is a real substring of the input. We measure the verbatim-fidelity rate directly, meaning the fraction of predicted values that occur exactly in the input. We expect it to drop most on the noisiest inputs, and that is precisely where the substitution failure lives.
Why not just use a span-only grammar and skip the pointer? This is the real threat, and the proposal is designed so the $\lambda=0$ ablation can refute it. If the grammar alone matches the pointer-augmented system on the shifted benchmarks, the conclusion is that verbatim constraints suffice, and the paper becomes a negative result about pointers plus a useful engine. The pointer should matter only if the model's masked token probabilities are systematically biased toward familiar spans under shift. We test that mechanism directly by measuring, among span-only errors, how often the wrong span has lower model perplexity than the gold span.
Why not generate freely and fuzzy-match outputs back to the input afterward? Post-hoc matching repairs string drift, but it cannot recover a span choice that was wrong to begin with, and it fails on dropped delimiters and invented keys. Validating Large Language Models with ReLM also found in-decoding constraints far cheaper than post-hoc search (15x on extraction). We include a fuzzy-match repair baseline using nearest input span by character edit distance to quantify this.
Why not just adapt the model to the domain? Domain-adaptive pretraining helps modestly (the 2.16 average F1 above), leaves the subcategory trap largely intact per CrossNER, and needs unlabeled domain text and retraining per model. The proposal is complementary, and we report adaptation plus constraints as a combination.
Proposed Experiments
We evaluate on three datasets. CrossNER frames extraction as generating a JSON list of typed entities from CoNLL-style prompts, evaluated on its five target domains. TweetQA uses a slot grammar allowing SPAN or FREE answers, with BLEU-1 and exact-match reported separately on its extractive and abstractive subsets (43% of its questions are abstractive, per the file). SLURP takes transcripts and outputs a grammar-constrained intent-plus-slots record, run on both gold text and ASR hypotheses. Its reported entity SLU-F1 is 78.19 on perfect text and 70.84 for the best pipeline (SLURP), which gives a natural clean-versus-noisy contrast. The spoken-input TCPGen result of 79.5 SLU-F1 overall and 50.2 on held-out unseen slots comes from a different, speech-input setting and is cited only for orientation.
Metrics are entity or slot F1, verbatim fidelity, parse validity, boundary-error rate (predicted span overlapping but not equal to gold), and decoding latency.
The first decisive check is the $\lambda=0$ versus tuned-$\lambda$ comparison under a controlled shift. We perturb the CrossNER test inputs at increasing strength with lowercasing, character noise, and handle or hashtag rewriting of entities, and measure input perplexity under the base model. The claim predicts that the pointer's F1 gain over span-only increases with perplexity and is near zero on clean CoNLL text. A flat or negative trend falsifies it.
The second check targets the boundary errors that motivated word-boundary conditions. We compare boundary-error rates with and without the end-of-word condition, on items where the gold entity is a prefix of a longer input word or vice versa.
The third check targets precision. On SLURP and TweetQA we verify that the pointer does not raise false copies in abstractive answers, confirming that the grammar-branch copy decision avoids the biasing-strength trade-off.
The engine is judged successful if validity is 100%, per-token overhead stays in the tens of milliseconds, and the in-domain SynthIE result is not degraded.
References
- Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
- Tree-Constrained Pointer Generator for End-to-End Contextual Speech Recognition
- End-to-End Spoken Language Understanding with Tree-Constrained Pointer Generator
- LargePiG: Your Large Language Model is Secretly a Pointer Generator
- PG-GSQL: Pointer-Generator Network with Guide Decoding for Cross-Domain Context-Dependent Text-to-SQL Generation
- A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization
- CrossNER: Evaluating Cross-Domain Named Entity Recognition
- Flexible and Efficient Grammar-Constrained Decoding
- TWEETQA: A Social Media Focused Question Answering Dataset
- Validating Large Language Models with ReLM
- SLURP: A Spoken Language Understanding Resource Package