Choosing Contrastive Decoding Strength Per Question, Without Labels

Several of the cheapest fixes for factual errors in large language models work at decoding time. They rescore the model's next-token probabilities by subtracting a "weaker" opinion, either from an earlier layer of the same network or from a deliberately degraded copy. One number, the contrast strength, controls how hard that subtraction pushes. In every published method it is set once, on a validation set, and then used for every question. The classical literature on regularization says that a single global setting is the wrong target whenever the right amount of correction depends on the input. That literature also provides rules for choosing the setting per problem instance from the data alone, with no ground truth. We propose to carry one such rule into factual decoding: compute the model's answer along a path of contrast strengths, and return the answer that stays the answer over the widest range of strengths.

The rule costs almost nothing. For multiple-choice and likelihood-scored benchmarks, every point on the path comes from the same forward pass. It also yields a principled signal for when to abstain: if no answer is stable along the path, the model should say it does not know. If the idea is right, it gives the flexibility of learned per-token controllers without training one, and it tells us when the validation-tuned constant is leaving accuracy on the table.

Core Claim

Contrastive decoding for factuality is a regularized estimator, and its strength parameter plays the same role as a regularization coefficient $\lambda$. For a candidate answer $y$ to question $x$, write the contrastive score as

$$s_\alpha(y \mid x) = \sum_t \Big[\log p_{\text{final}}(y_t \mid x, y_{<t}) - \alpha \log p_{\text{early}}(y_t \mid x, y_{<t})\Big],$$

where $p_{\text{final}}$ is the last layer's token distribution, $p_{\text{early}}$ is the earlier ("premature") layer or amateur model, and $\alpha \ge 0$ is the contrast strength. The sum runs only over tokens that pass the usual plausibility filter.

Our claim has two parts. First, the $\alpha$ that makes $y$ correct varies substantially across questions, so an oracle that picks $\alpha$ per question beats the best fixed $\alpha$ by a wide margin. Second, a label-free rule recovers much of that gap. Define the persistence of an answer as the length of the interval of $\alpha$ over which it is the top-scoring answer. The rule returns the most persistent answer and abstains when the largest persistence falls below a threshold.

The intuition comes from DoLa's own observation that factual knowledge emerges in late layers. A correct answer should win whether the early-layer prior is discounted mildly or strongly. An answer that wins only in a narrow band of $\alpha$ is being produced by the subtraction rather than by knowledge the model holds. This mirrors the L-curve and discrepancy-style parameter choice rules from inverse problems, which pick $\lambda$ where the solution stops changing rapidly.

Where This Sits

The mechanism we tune comes from three decoding methods. DoLa contrasts the final layer against a dynamically chosen earlier layer, using Jensen-Shannon divergence to pick that layer, and filters implausible tokens. Regularized Contrastive Decoding contrasts against an adversarially weakened amateur model and tunes its subtraction coefficients as static scalars on held-out data. Octopus trains a small decision head with preference optimization to choose among several contrastive strategies per token, but only for vision-language models.

We keep DoLa's layer selection and plausibility filter unchanged, and we train nothing. What changes is that the strength becomes a per-question choice made by a rule, not a constant (as in DoLa and Regularized Contrastive Decoding) and not a learned discrete selector (as in Octopus).

The regularization literature both motivates and constrains the design. Consistency analysis of bilevel data-driven learning shows that a fixed regularization parameter learned from $n$ validation examples converges to the population-optimal fixed value at rate $1/\sqrt{n}$. Validation tuning is therefore not sample-starved. If a fixed $\alpha$ underperforms, the reason must be heterogeneity across questions, and that is exactly what our oracle experiment measures. Classical data-driven selection rules show that per-instance parameters can be chosen without ground truth: generalized cross-validation and Stein's unbiased risk estimate in hyperspectral superresolution, and constraint-driven search over Tikhonov parameters in sound field reproduction. We cannot use those estimators directly, because they need a Gaussian noise model on the observations that token logits do not satisfy. That is why we use a path-stability rule, which assumes only that correct answers are stable.

Evidence from training-time methods points the same way. ε-DPO shows that setting the preference-optimization penalty per training pair beats a static value; for example, the length-controlled AlpacaEval 2 win rate on Mistral-7B rises from 26.8% to 35.6%. CAMP finds that truthfulness and helpfulness conflict in only about 11% of pairs, so uniform pressure is misapplied to most data. The policy mirror descent analysis lists adaptive scheduling of its regularization parameter as open work. All three adapt regularization during training using labels or preference signals. We adapt at inference, per question, using no labels.

One constraint shapes the design. SIREN shows that pushing probability mass toward low-probability tokens in large vocabularies produces nonsense ("entropy explosion"). The path therefore stops at a maximum $\alpha$, and the plausibility filter is applied at every point on it. DoLa's failure on small models tells us to restrict claims to models at 7B parameters and above.

Research Program

The work has three stages. The first measures headroom. On the development splits of TruthfulQA and FACTOR, we compute each question's answer across a grid of about 30 $\alpha$ values between 0 and $\alpha_{\max}$, using DoLa's code (voidism/DoLa). We then compare the best fixed $\alpha$ against the per-question oracle.

For teacher-forced scoring, $p_{\text{final}}$ and $p_{\text{early}}$ at each token do not depend on $\alpha$. The whole path therefore costs one forward pass per candidate answer, the same cost as DoLa itself. For short-answer generation on TriviaQA and Natural Questions, each grid point is a separate greedy decode of a few tokens, run as a batch. We will report the cost of this batch honestly.

The second stage builds and tests the persistence rule and its abstention threshold. We also test two cheap alternative label-free criteria, so that stability is shown to be the operative ingredient rather than any per-question choice. One criterion picks the $\alpha$ where the score margin between the top two answers is largest; the other picks the knee of the path along which the answer distribution changes.

The third stage tests generality by applying the same rule to a second strength parameter: the scaling of the query-specific steering vector in TruthFlow, which adds a learned correction to middle-layer hidden states.

Baselines

The do-nothing reference is greedy decoding, or plain likelihood scoring, with no contrast. For ranking methods only, we also include the best fixed $\alpha$ chosen per task on the development split with the same grid. This is the honest version of the "just grid search" alternative, and it isolates whether per-question selection adds anything.

The strongest existing competitor is DoLa with its published, validation-tuned configuration, because our method is a strict modification of it. DoLa reports gains of 17 absolute points on TruthfulQA for LLaMA-7B and over 12 points for LLaMA-65B. Where we can reproduce it, we also include Regularized Contrastive Decoding, which reports a gain of +19.75 on TruthfulQA MC2 and +12.71 accuracy on FACTOR Expert over base Llama-2. For context among non-decoding methods, Faithful Finetuning combined with TruthX representation editing reports 51.41 MC1 and 74.00 MC2 on TruthfulQA.

For Natural Questions and TriviaQA, both DoLa-family papers report gains, but no reported result on Lacuna gives an exact number for them. We therefore rerun every baseline ourselves. All methods use the same base models (LLaMA-2-7B/13B and Llama-3-8B), the same prompts, the same development split for any tuning, and the same premature-layer candidates.

Beating DoLa means a gain over its tuned fixed $\alpha$ on held-out TruthfulQA MC2 and FACTOR accuracy, with confidence intervals excluding zero across three models. It also means no loss on GSM8K and StrategyQA, which DoLa used as tasks where contrast can hurt.

Three ablations isolate the key ingredient. The first chooses $\alpha$ per question at random from the same grid, which separates the effect of stability from the effect of any variation. The second applies the persistence rule without DoLa's dynamic layer selection. The third drops the plausibility filter.

Why not just…?

Why not just tune one α on a validation set? The bilevel consistency result says this estimate is already accurate with modest data. It will lose only if questions truly differ in the $\alpha$ they need. The oracle experiment settles this before anything else is built. If the per-question oracle beats the best fixed $\alpha$ by less than about two points on TruthfulQA MC2 and FACTOR, we stop and report that fixed tuning suffices.

Why not just sample several answers and take the majority, as in self-consistency? This is the most serious threat, because stability along the $\alpha$ path is itself a kind of consistency check. The differences are cost and signal. For scored tasks our path is free, while sampling multiplies forward passes. HaluEval also shows that models often fail to recognize their own fabrications; ChatGPT detects them at 58.53% accuracy on summarization. That suggests consistency across random samples may be consistently wrong on imitative falsehoods, where the most probable answer is the misconception. We will compare majority voting over $K$ temperature samples against persistence over $K$ $\alpha$ values at equal decode budget on TriviaQA and Natural Questions. If voting matches persistence, we will say so.

Why not just learn a small controller that predicts α from hidden states, like Octopus? A learned controller could beat a fixed rule, but it needs paired training trajectories and may not transfer across tasks. We will train a small regressor on development-split oracle $\alpha$ values, using features that include inter-layer divergence and entropy, and compare it against the rule both in-domain and when transferred from TruthfulQA to FACTOR. If the controller wins in-domain but the rule transfers better, that is a useful, bounded result.

Proposed Experiments

The decisive checks come in order. First, the headroom test described above, run on TruthfulQA, FACTOR (Wiki, News, and Expert subsets), and HaluEval QA. Second, held-out accuracy of the persistence rule against DoLa's fixed setting, including the no-harm tests on GSM8K and StrategyQA.

Third, selective prediction. We plot accuracy against coverage as the abstention threshold varies, and compare with abstaining on low final-layer confidence. On TruthfulQA generation, we use its fine-tuned judge model to report both truthful and informative rates, because abstaining trivially raises truthfulness.

Fourth, the equal-budget comparison with self-consistency on Natural Questions and TriviaQA, measured by exact match. Fifth, the transfer test on TruthFlow's steering strength. On Llama-3-8B, TruthFlow reports raising TruthfulQA's "True" score from 52.32% to 64.79% with a fixed configuration. Success there means matching or exceeding that fixed configuration without per-task tuning.

A negative result at the first check is still informative. It would show that the gains from contrastive decoding come from a single stable correction, not from question-specific regularization.