When the Answer Format Picks the Culture: Distribution-Faithful Structured Elicitation of Moral Judgments

Researchers who measure a language model's cultural values usually force its answer into a fixed format: a multiple-choice letter, a Likert number, or a JSON object with a "verdict" field. The tool that does the forcing is grammar-constrained decoding. At each step it blocks every next token that would break the required format, then rescales the remaining probabilities. This proposal argues that the forcing step is not neutral. When a model would naturally hedge ("it depends on the family's wishes"), refuse, or wander off-format, blocking those continuations and rescaling locally does not give the model's actual conditional opinion. It pushes probability toward whichever valid path was locally favored, and that path is plausibly the most binary, most default-culture answer. If so, part of what the literature reports as models flattening split human opinion, and part of the gap between survey answers and free-text behavior, comes from the measurement procedure rather than from the model.

We propose to test this directly and, if it holds, to fix it. Exact sampling from "the model's distribution, conditioned on the output being valid" is well defined and can be estimated by brute force. We will measure how far standard constrained decoding departs from that target on cultural and moral benchmarks across many languages. We will then build a cheap, reusable estimator that closes the gap. The estimator is a small predictor, trained jointly across many answer formats and languages, of whether a partial answer will end in a valid one. It plugs into existing fast grammar engines as a swappable module.

Core Claim

Formally, let $p(y)$ be the model's probability of a full response $y$ and $L$ the set of format-valid responses. The quantity a cultural survey intends to read out is the conditional distribution

$$p(y \mid y \in L) \propto p(y)\,\mathbb{1}[y \in L].$$

Sampling it token by token requires weighting each candidate token $v$ after prefix $y_{<t}$ by the probability $E(y_{<t}v)$ that an unconstrained continuation of the extended prefix ends in $L$:

$$q(v \mid y_{<t}) \propto p(v \mid y_{<t})\; E(y_{<t} v).$$

Standard constrained decoding sets $E=1$ for every prefix that can still be completed validly. Grammar-Aligned Decoding proves that this distorts the output distribution and proposes ASAp, which refines over-approximations of $E$ from repeated samples. That paper studies program synthesis and constituency parsing, where distortion lowers correctness. Our claim is that in value elicitation the same distortion is systematic rather than random, and that it biases the very quantities cultural-alignment benchmarks report: verdict split ratios, deviation from human preference vectors, and survey-versus-generation consistency. The work has two parts. The first establishes how large this bias is, and in which languages and cultures it is largest. The second makes the correction cheap enough for routine evaluation and deployment.

Relation to Prior Work

Our motivation comes from cultural-alignment evaluations that found symptoms consistent with a format artifact without isolating it. Multilingual Trolley Problems for Language Models runs 19 models over 98,440 Moral Machine vignettes in 107 languages and attributes GPT-4o Mini's outlier misalignment to binarization. ValueBench reports a "Likert paradox," where multiple-choice answers disagree with open-ended generation. DOVE argues that multiple-choice surveys have construct and context gaps and evaluates open-ended text distributions instead. CMoralEval finds most of 26 Chinese models near the 0.33 chance floor on three-option questions. All four treat the elicited answer as the model's opinion. We ask how much of each finding survives once elicitation samples the correct conditional.

Our method draws on three sources. Grammar-Aligned Decoding supplies the target distribution and the ASAp estimator, which we use as an expensive reference. Flexible and Efficient Grammar-Constrained Decoding (GREATGRAMMA) supplies the fast token-mask engine, with 5–32 ms per-token overhead. Our estimator multiplies its masks rather than replacing them. Validating Large Language Models with ReLM, which compiles regular expressions to token-level automata for model testing, constrains our interpretation. Its LAMBADA jump from 41.6% to 71% for GPT-2 XL under a single-word constraint shows that format constraints can change measured capability by tens of points. ReLM reads that change as a gain; we read the analogous change in value measurement as a confound to be quantified.

Two decoding-time alignment methods are adjacent but not competitors on this question. DARC reranks K samples under risk constraints to respect annotator disagreement. MAP navigates value trade-offs via importance sampling over candidates. Both change which answer is chosen. Neither asks whether a format constraint has already changed the distribution being chosen from, and neither operates on token-level masks. Rethinking Machine Ethics shows that theory-guided reasoning prompts improve moral classification, for example GPT-4 with Theory of Dyadic Morality at 88.9% on commonsense morality. It also reports that such prompts make models hallucinate moral risks. We treat reasoning-then-verdict schemas as one of the formats under study, since a forced reasoning field gives local renormalization many more tokens over which to compound.

Research Program

The first component is a distortion audit. For each benchmark item we define a small grammar, either a bare verdict or a reasoning field followed by a verdict, compiled with GREATGRAMMA. We then estimate three distributions over the verdict.

The first is the reference conditional, estimated by rejection sampling. We draw unconstrained samples at temperature 1 and keep those that parse. Rejection sampling yields exact samples of the target, so its error is pure Monte Carlo error. The acceptance rate is itself a key measurement: it is the fraction of the model's mass that the format discards.

The second is standard constrained decoding at temperature 1. The third is the conventional readout: greedy decoding or next-token log-probabilities over option letters.

For every item, language, and model we compute the divergence between the verdict distributions, and how each distribution shifts benchmark scores. The central analysis regresses distortion on acceptance rate, language resource level, and topic. Multilingual Trolley Problems found that alignment quality does not track resource level. A specific prediction follows: distortion should track acceptance rate, which may vary across languages for reasons other than resource level, such as more hedging or refusal in some languages. If so, some of the language-to-language variation in reported misalignment is a format effect.

The second component is an amortized validity estimator. Rejection sampling and ASAp are expensive. Grammar-Aligned Decoding uses up to 2,000 samples to converge, and acceptance can be low exactly where hedging is common. We will train a light probe on the frozen model's hidden states to predict $E(y_{<t}v)$. Its training labels are Monte Carlo rollout outcomes collected during the audit, so the audit data does double duty.

Multi-task learning enters here as the training method. Each combination of schema and language is a task. The probe has a shared trunk and small task heads, trained jointly with balanced sampling. We will measure whether tasks conflict, and apply RotoGrad, which rotates and rescales per-task gradients to reduce conflict, only if joint training underperforms per-task probes. Whether one shared trunk can serve the bare-verdict and reasoning schemas across 107 languages is itself a finding, because it indicates whether "will this end validly" is a format-general property of hidden states.

Modular design is the deployment constraint. The estimator exposes a single interface: given a prefix and a mask, return reweighting factors. The grammar engine, the schema, and the estimator are therefore separately swappable. A new cultural schema reuses the shared trunk and needs only a fresh head. We will report what adding a new schema costs, in rollouts and wall-clock time, against recompiling and rerunning ASAp from scratch.

Baselines

The naive reference point is free generation followed by a parser, with unparseable outputs discarded. This is common practice, and it is exactly rejection sampling with one draw per attempt. Run with enough draws, it is our ground truth for the verdict distribution rather than a competitor. Run with the small draw budgets typical of benchmark papers, it serves as the cheap reference. We run it at matched model calls.

The conventional elicitation baselines are option-letter log-probability scoring and greedy constrained decoding. These are what most cultural benchmarks report. For the strongest reported number on our main benchmark, Multilingual Trolley Problems reports misalignment scores below 0.6 for Llama 3.1 70B, Llama 3 70B and Llama 3 8B, and about 1.45 for GPT-4o Mini, under its own elicitation protocol. We will first reproduce those Llama numbers with that protocol before changing anything. Our method does not "beat" this baseline by lowering the score. A result counts if the corrected elicitation moves the score by more than the run-to-run noise of the original protocol in a consistent direction, and if the corrected verdict distributions are closer to the rejection-sampling reference.

Standard temperature-1 constrained decoding with GREATGRAMMA is the direct comparison for the distortion claim. It uses the same model, the same grammar, and the same temperature as every other arm.

ASAp from Grammar-Aligned Decoding is the strongest existing method for sampling the correct conditional. On SyGuS INV-BV it reports a 38% geometric-mean increase in correct solutions over standard constrained decoding. No reported result was found on Lacuna for ASAp on any moral or cultural benchmark. We run it with a fixed sample budget per item, equal to the rollouts used to train and run our probe.

The amortized probe beats ASAp if it reaches equal or lower divergence to the reference at a fraction of ASAp's per-item cost, with all methods charged the same total model forward passes. Two ablations isolate the key ingredient. The first replaces probe predictions with a constant per schema, which tests whether prefix-dependence matters at all. The second trains per-task probes without sharing, which tests whether the multi-task trunk earns its place.

Why not just…?

Why not just use rejection sampling everywhere? For offline evaluation this is a real threat. If acceptance rates on these benchmarks are high, say above a few tens of percent, rejection sampling is cheap, exact, and makes the estimator unnecessary. In that case the contribution reduces to the audit plus a recommendation, and we will say so. The estimator matters only where acceptance is low, as in long reasoning-then-verdict schemas or hedging-heavy languages, and in deployment, where a single sample is wanted under a latency budget. The audit measures acceptance per language and schema, so it settles this directly.

Why not just prompt the model to answer in the format, without any decoding constraint? Prompting does not escape the problem; it hides it. The parse-failure fraction becomes silent missing data. Discarding those responses is rejection sampling with a non-uniform, prompt-dependent filter, and DOVE documents that prompt phrasing already shifts alignment measurements. We include instruction-only prompting with parse-and-discard as an arm and report both its acceptance rate and how its verdict distribution relates to the reference.

Why not just raise the temperature or average over paraphrases to undo binarization? Temperature changes the target distribution itself, not the bias introduced by masking. A sharper or flatter version of a distorted conditional is still distorted. We run constrained decoding at several temperatures and test whether any temperature matches the reference divergence. If one temperature uniformly does, the simpler fix wins, and we report that.

Why not just fine-tune the model on the format? Grammar-Aligned Decoding reports that QLoRA-fine-tuned models still show distribution distortion under constraints. More fundamentally, fine-tuning changes the model being measured, which defeats the purpose of an evaluation. We do not pursue it except as a cost reference.

Proposed Experiments and Decisive Checks

The primary benchmark is Multilingual Trolley Problems, chosen for its 107 languages and published misalignment scores. Models are Llama 3 8B and 70B and Qwen2.5 7B. The secondary benchmarks are CMoralEval, which tests whether the 0.33 near-chance collapse partly reflects forcing hedged answers into three options, and ValueBench, used to recompute the Likert-versus-generation gap with corrected elicitation. For open-ended consistency we use the DOVE set. Its unbalanced-optimal-transport comparison against human texts in four countries needs, per its authors, at least 500 samples per culture. We check whether corrected verdicts agree better with the cultural profile of the same model's free text. DOVE reports that its open-ended metric correlates at 31.56% with culturally specific hate-speech detection, against 16.20% for the World Values Survey, which is evidence that free text is the more valid reference.

The first decisive check is falsifying. If the divergence between standard constrained decoding and rejection sampling is within Monte Carlo noise on most items, the core claim fails. The project would then report that format constraints are safe for value elicitation, which is itself useful.

The second check is directional. If distortion exists, does it consistently increase verdict concentration and move toward the English-speaking norm profile? Knowledge of cultural moral norms reports that pretrained English models correlate with English norms at about r = 0.80 but with global World Values Survey norms at only 0.20 to 0.35, so English is the natural candidate default. Knowledge of cultural moral norms in large language models provides that comparison point.

The third check is whether the probe, trained on a subset of languages and schemas, holds divergence near the ASAp level on held-out languages and a held-out reasoning schema, at matched forward-pass budgets. Per-token overhead is reported alongside GREATGRAMMA's 5–32 ms baseline.