When a language model fills in a JSON record, a SQL query, or a function call, two different things can go wrong. The syntax can break, with a missing bracket or a renamed key. Or the syntax can be fine while the text inside the fields comes out in the wrong language. A Swahili news article might yield an English event description, or a request for a Japanese answer might return an English "answer" string. Grammar-constrained decoding (a technique that, at each step, masks out any next token that would violate a formal grammar) has largely solved the first problem. It does nothing about the second, because a grammar that says "any string goes here" allows English as readily as Japanese. The off-target translation literature has built decoding-time fixes for wrong-language output, but only for free-form text. Applied naively to structured output, those fixes would penalize the braces and English schema keys the grammar requires.
This note proposes to measure and fix language drift inside structured outputs. The core claim has two parts. First, English-heavy scaffolding (keys, delimiters, code keywords) pushes the model toward English in the free-text slots that follow it, so wrapping a task in JSON should raise off-target rates relative to the same task in free form. Second, the incremental parser already used for grammar-constrained decoding knows at every step whether the model is inside a free-text slot or inside structure. That parser state can be used to switch on a language correction only inside slots. Syntax stays guaranteed by the grammar, and language adherence is enforced only where language matters.
What is known and what is missing
On the structure side, SynCode combines an incremental LR parser with a precomputed table of token masks. It removes all JSON syntax errors and 96.07% of Python and Go syntax errors. Its evaluations are high-resource and English-centric, and it never inspects what language the string contents are in. Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning adds input-dependent grammars, whose allowed outputs depend on the input, for example restricting entities to candidates. That idea matters here because it suggests a trivial fix for extractive slots, discussed below. Deco-G argues that tracking format and solving the task compete for the model's capacity. It moves formatting into a separate tractable probabilistic model, but it is evaluated only in English. These three papers supply the machinery we reuse. None of them measures language inside slots.
On the language side, the literature supplies both the motivation and the candidate interventions. Understanding and Mitigating Language Confusion in LLMs shows that drift starts at high-entropy "confusion points" where the model falls back to frequent English tokens. It also finds that instruction tuning makes this worse. Its Language Confusion Benchmark metrics, line-level pass rate (LPR) and word-level pass rate (WPR), are the ones we adapt. Mechanistic Understanding and Mitigation of Language Confusion and ReCoVeR locate the failure in late layers and fix it by editing neurons or adding steering vectors. Language-contrastive decoding subtracts the log-probability the model assigns under a wrong-language instruction. LiBS reranks beams with an external language identifier. All of these operate on free-form text and apply to every token. In structured output that is precisely the problem: most structural tokens are English or ASCII, and a sequence-wide language penalty works against the grammar.
The two papers closest to this intersection leave the question open. KnowCoder-X represents extraction schemas as Python classes and fine-tunes for cross-lingual extraction. It uses no decoding constraints and never asks whether class/def scaffolding pulls slot values toward English. MASSIE evaluates 21 models on extraction in 96 languages. It finds that no model reaches 40 soft-Levenshtein F1 (LF1, a partial-credit F1 based on edit distance) on event extraction, but it cannot say how much of that failure is broken syntax, wrong content, or wrong language.
Core Claim
We claim that in multilingual structured generation, off-target language is a distinct failure mode that grammar constraints cannot catch. We further claim it is concentrated at the first few tokens after a structural boundary, such as the opening quote after "description":. Finally, we claim that a language correction applied only where the parser says the model is inside a free-text slot recovers target-language adherence. It should do so without the syntax breakage of sequence-wide correction, and without the quality loss of hard vocabulary restriction.
Concretely, let $s_t$ be the parser state after prefix $y_{<t}$, let $M(s_t)$ be the set of grammar-legal next tokens, and let $x$ be the prompt with the target-language instruction and $\tilde{x}$ the same prompt with the instruction replaced by an English one. Slot-scoped contrastive decoding scores candidate token $v \in M(s_t)$ as
$$\text{score}(v) = \log P(v \mid x, y_{<t}) - \lambda\,\mathbb{1}[s_t \in \mathcal{S}]\,\log P(v \mid \tilde{x}, y_{<t}),$$
where $\mathcal{S}$ is the set of parser states inside a natural-language string terminal and $\lambda \ge 0$ is the contrast strength. Outside slots the rule reduces to plain grammar-constrained decoding. The contrastive pass is computed only when $s_t \in \mathcal{S}$, reusing the key-value cache of the shared prefix, so the extra cost scales with slot length rather than output length. We test a steering-vector variant in the same way, adding the ReCoVeR vector only at slot positions.
Research Program
The work has three parts.
The first part is a diagnostic benchmark. We will wrap existing multilingual tasks in structured formats while holding the content fixed.
- Flores-200 translations. These become a JSON object with a
"translation"field. - MultiQ questions. These become
{"answer": ..., "explanation": ...}. - Language Confusion Benchmark prompts. These become JSON responses with free-text fields.
- MASSIE event-extraction items. These keep their native argument schema, emitted as JSON and in KnowCoder-X-style Python classes.
Each wrapped task is paired with its free-form original. Each structured version comes in two variants, English keys and keys translated into the target language. This isolates the effect of the scaffolding's language from the effect of structure itself.
Every output is then scored on three separate axes:
- Syntactic validity: does it parse?
- Slot-level language adherence: LPR and WPR computed per slot. We whitelist tokens that appear verbatim in the source, which addresses the loanword problem flagged in the language confusion work.
- Task quality: chrF2 on the translation field, answer accuracy on MultiQ, and LF1 on MASSIE.
This three-way split is exactly what MASSIE lacks.
The second part is a mechanism analysis. At each slot-onset position we will record the entropy of the next-token distribution and the total probability mass on English-script tokens, and compare them with later positions in the same slot and with matched positions in the free-form outputs. If the confusion-point account is right, onset positions after English keys should show elevated entropy and English mass, and translating the keys should reduce both.
The third part is the intervention study. We compare slot-scoped contrastive decoding and slot-scoped steering against the alternatives described below. We use open-weight models, because logits are needed: Llama 3.1 8B Instruct (the model on which ReCoVeR reports results), Gemma 2 2B (used by SynCode), and one larger Llama 3 variant. Grammars are built with SynCode for JSON and Python.
Baselines
The naive reference is unconstrained greedy decoding with the target-language instruction. Greedy decoding is the fair floor because the language confusion study shows that temperature 0 already raises Command R's WPR from 86.5% to 97.2%. Any gain we report must be on top of that.
The first structural baseline is plain grammar-constrained decoding with SynCode. It reports removing all JSON syntax errors on JSON-Mode-Eval, so it should match us on validity. The question is whether it leaves slot drift untouched, and our method must beat it on slot LPR/WPR while matching it on validity.
The strongest language-side competitors are the published interventions, applied sequence-wide on top of the grammar mask.
- Sequence-wide language-contrastive decoding. On Flores-101, combining it with source-contrastive decoding reduced low-quality translations (chrF2 below 10) by 67–83% on average (paper).
- ReCoVeR+ steering. On MultiQ with Llama 3.1 it raised LPR from 33.1% to 91.4% while staying within 0.4 points of baseline MMLU accuracy (paper).
- LiBS beam reranking. On WMT-10 it cut off-target rate from 22.9% to 7.7% with +1.1 BLEU (paper). Here we adapt it by scoring only slot contents with the language identifier.
We found no reported result on Lacuna for any of these methods on structured outputs or slot-level language metrics. The comparison numbers will therefore come from our own runs, and the cited figures serve as sanity anchors for the free-form versions of the tasks.
For fairness, every method uses the same prompts, grammar, models, and beam width. Each receives the same tuning budget: a sweep over its strength parameter ($\lambda$, steering scale, or $\alpha$) on a held-out development split of Flores-200 languages, with test languages kept disjoint.
Beating these methods means one of two outcomes at matched task quality (chrF2, MultiQ accuracy, or LF1 within one point of the best baseline):
- higher slot LPR/WPR than grammar-constrained decoding alone, or
- equal slot adherence to the sequence-wide methods with fewer invalid outputs or less quality loss.
Two ablations isolate the key ingredient, which is the parser-based scoping.
- Random scoping. The contrast is applied at randomly chosen positions with the same count as slot positions. This tests whether where the correction is applied matters, beyond how much correction is applied.
- Hard target-script mask. Inside slots, tokens outside the target script are simply disallowed. This works for non-Latin scripts, and we expect it to damage names, numbers, and loanwords and to be useless for Latin-script languages.
Why not just…?
Why not just constrain slots to copy spans from the input, using an input-dependent grammar? For purely extractive slots such as named-entity strings, this is a real threat, and we expect it to win. If every value must be a substring of the source, it cannot drift into English. Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning already shows that input-dependent grammars raise entity disambiguation accuracy from 54.1% to 80.3%. We therefore include copy-constrained decoding as a baseline on the extractive MASSIE slots and do not claim to beat it there. Our claim is restricted to slots whose content is generated rather than copied: translations, answers, explanations, normalized event descriptions, and cross-lingual extraction where the output language differs from the source. In all of these, copy constraints do not apply.
Why not just translate the schema keys and give target-language few-shot examples? This is cheap, and the language confusion study reports that few-shot prompting nearly removes line-level confusion in base models. Our key-translation variant tests this directly. If translated keys plus few-shot examples close the slot-adherence gap on instruction-tuned models, the proposal reduces to a prompting recommendation, and we will report that. Two reasons suggest it will not fully close the gap. Downstream parsers usually require fixed English keys. And instruction-tuned models, which are the ones deployed in agent pipelines, show more confusion than base models.
Why not just apply ReCoVeR or contrastive decoding to the whole sequence? These are the strongest existing tools, and our method is only a scoping change to them. The open question is whether scoping matters. The grammar mask prevents sequence-wide correction from producing invalid syntax outright. It cannot prevent the correction from warping which legal structural choices the model makes, such as picking an optional field or ending an array early, and those choices affect schema validity and content quality. If sequence-wide steering under the grammar matches slot-scoped steering on every axis, the scoping contribution is null, and the random-scoping ablation will show it.
Why not just detect wrong-language fields afterwards and regenerate? Post-hoc checking with a language identifier is a legitimate baseline, and we run it with up to three retries. We report both adherence and total tokens generated. LiBS already reports 3.5–7.5x latency from language-identifier scoring inside beam search, and full regeneration costs more. Our comparison is at matched compute, which a retry loop may win on quality only by spending several times the budget.
Proposed Experiments
The first decisive check tests the premise. On the Flores-200 and Language Confusion Benchmark items, we compare slot-level WPR under greedy decoding for three versions: free form, JSON with English keys, and JSON with translated keys. If English-key wrapping does not lower adherence relative to free form, the confusion-point hypothesis about scaffolding is wrong. In that case the paper becomes a measurement study of the three failure axes, which MASSIE-style evaluation still lacks.
The second check tests whether the mechanism is located where we think it is. We measure whether English-token probability mass at slot onsets exceeds that at later slot positions, and whether translating the keys reduces it.
The third check is the intervention comparison across 20 or more target languages. These are split into Latin-script and non-Latin-script, and into high- and low-resource. For each method we report syntactic validity, slot LPR/WPR, task quality, and added latency per token. The claim succeeds if slot-scoped contrastive decoding or slot-scoped steering keeps SynCode-level validity, improves slot adherence over grammar-constrained decoding alone, and dominates the sequence-wide versions on at least one axis without losing on another.
The last check tests whether Deco-G's decoupling argument transfers across languages. We run Deco-G with a multilingual event-extraction grammar. The question is whether moving formatting into a separate model reduces slot drift in low-resource languages more than in high-resource ones.
References
- SynCode: LLM Generation with Grammar Augmentation
- Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
- Decoupling Task-Solving and Output Formatting in LLM Generation
- Understanding and Mitigating Language Confusion in LLMs
- Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models
- ReCoVeR the Target Language: Language Steering without Sacrificing Task Performance
- Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding
- Language-Informed Beam Search Decoding for Multilingual Machine Translation
- KnowCoder-X: Boosting Multilingual Information Extraction via Code
- Massively Multilingual Instruction-Following Information Extraction