Several recent methods train language models to "show their real work," meaning to write step-by-step reasoning (a chain of thought) that actually produces the final answer rather than justifying an answer chosen in advance. They build pairs of faithful and unfaithful reasoning chains and fine-tune the model to prefer the faithful one. The labels in those pairs are produced automatically, either by perturbing steps and checking whether the answer flips or by asking a judge model. These labelers are unreliable in a specific way. Their mistakes are not random. They cluster on fluent, confident rationalizations, which are exactly the cases the model is best at defending.
Learning theory has a precise account of why this pattern is dangerous. In agnostic learning, the setting where no classifier fits the data perfectly and an adversary may flip a fraction of labels, standard convex losses fail because a few confidently wrong labels can pull the learned boundary far from the best one. Restricting updates to examples near the current decision boundary repairs this. This proposal carries that repair into preference training for faithfulness. It then tests whether the reported conflict between faithfulness and accuracy is partly an artifact of noisy supervision.
Core Claim
Preference training with a linear reward head is exactly halfspace classification, meaning a classifier that splits the space with a single hyperplane. The features are the difference between the two chains' representations, and the label says which chain is preferred. Labeler errors are then flipped labels. The worst case for this structure is the case that matters here: a fluent rationalization that the labeler marks faithful, and that the model already scores far on the "wrong" side.
Direct Preference Optimization (DPO) minimizes $-\log\sigma(\beta m)$. Here $m$ is the model's implicit reward margin for the labeled-preferred chain over the other, $\beta$ is a temperature, and $\sigma$ is the logistic function. Each pair's gradient is weighted by $\sigma(-\beta m)$, so confidently contradicted pairs get full weight. We propose a banded objective in which a pair contributes only while its margin lies inside a band that shrinks over training:
$$ w_t(m) = \mathbf{1}\left[\,|m - c_t| \le b_t\,\right], $$
where $w_t$ multiplies the DPO gradient at training round $t$, $c_t$ is the band center, and $b_t$ is the band half-width. This is the construction of Learning General Halfspaces with Adversarial Label Noise via Online Gradient Descent. That paper proves that gradients computed with ramp-shaped non-convex losses on points inside a narrow band around the current hyperplane move the solution toward the best one. The result holds under Gaussian inputs, even with an $\epsilon$ fraction of adversarial flips, and gives error at most $C\epsilon$. We test the claim that, at the scale of real language models, the band objective beats DPO on both faithfulness and accuracy when labels come from realistic automated labelers. The advantage should grow with how concentrated the labeler's errors are on high-margin pairs.
Relation to Prior Work
Our premise that the labels are bad comes from three findings. The main automated detector evaluated in FaithCoT-Bench, an expert-annotated benchmark of faithful and unfaithful chains, reaches only 65 to 77 F1, and the benchmark reports that articulate rationalizations are what fool it. Can Aha Moments Be Fake? scores steps by whether perturbing them flips the answer. Its finding that models make correct self-corrections and then ignore them implies such perturbation tests can misfire when computation is redundant or bypassed. A Positive Case for Faithfulness, which measures how well a model's self-explanations help predict its own behavior, finds 5 to 15% of self-explanations egregiously misleading.
The methods we modify are the preference-training pipelines. FRIT labels steps by substituting distractor facts and runs DPO on the resulting pairs. FRODO combines DPO with a counterfactual loss. Both treat their labels as clean. We keep their data pipelines and change only how much each pair is allowed to update the model.
The contradiction we try to explain is this. On the Hardness of Faithful Chain-of-Thought Reasoning reports accuracy drops of up to 20% on TruthfulQA when steering models toward faithfulness, while FRIT and FRODO report accuracy gains. One untested explanation is that faithfulness supervision of different quality produces different effects.
The theory constrains our design. Beyond the halfspace result, Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise shows that vanilla gradient descent with cross-entropy passes through an early checkpoint with error $\tilde O(\sqrt{\mathrm{OPT}})$ before fitting corrupted labels. Here $\mathrm{OPT}$ is the error of the best classifier in the class. This means plain early stopping is a principled competitor, not a strawman. Agnostic Proper Learning of Halfspaces under Gaussian Marginals and the hardness results of Hardness of Learning a Single Neuron with Adversarial Label Noise tell us not to expect guarantees without assumptions on the input distribution. Our guarantees are therefore confined to a linear reward head over approximately Gaussian features, and the full-model results are empirical.
Research Program
The work has four stages. The first is a diagnostic that could kill the project. The FINE-CoT expert annotations from FaithCoT-Bench cover over 1,000 chains, more than 300 of them unfaithful. We will run three automated labelers on these chains: FRIT-style distractor substitution, the necessity test from the True Thinking Score (the Aha-moments paper's measure of how much a step causally affects the answer), and an LLM judge. For each labeler we measure its error rate as a function of the base model's own margin and of fluency.
If the errors are roughly uniform across margins, the noise is benign. In that case ordinary noise-robust tricks should suffice, and we will report that and stop. If the errors concentrate at large margins, the adversarial-noise model applies.
The second stage is the theoretical reduction. We write preference learning with a linear reward head over frozen chain embeddings as halfspace learning on difference vectors. We then state conditions under which the band algorithm's $C\epsilon$ guarantee transfers: approximately Gaussian, whitened difference features, and a labeler flip rate of $\epsilon$. On this frozen-embedding setting we verify the predicted gap between logistic-loss DPO and band training under noise concentrated at large margins.
The third stage scales the objective to LoRA fine-tuning (low-rank adapters) of Mistral-7B and Qwen3-8B on FRIT's pipeline. We follow the original algorithm and anneal $b_t$, set $c_t$ from the running median margin, and search a small grid of band schedules.
The fourth stage revisits the contradiction. We rerun the faithfulness-steering comparison on TruthfulQA with clean expert labels, automated labels, and banded training on automated labels. The question is whether the accuracy drop tracks label quality.
Baselines
The do-nothing reference is the base model with standard chain-of-thought prompting, evaluated with identical decoding.
The strongest existing method is FRIT's DPO pipeline (FRIT). It reports a 7.6-point GSM8K accuracy gain for Mistral-7B and a 6.0-point StrategyQA gain for Qwen3-8B. We rerun it ourselves on identical preference pairs, with the same LoRA rank, the same number of optimizer steps, and the same hyperparameter-search budget as our method. Model selection for all methods uses the same small expert-labeled split. Beating FRIT means higher accuracy and a higher Early Answering score (the area over the curve of answer probability as reasoning steps are revealed, from Tanneru et al.) at equal compute.
FRODO (Making Reasoning Matter) reports a 2 to 3% absolute accuracy gain over standard fine-tuning. It is run under the same budget as a second preference-based competitor.
Early-stopped DPO is the theory-backed competitor from Provable Generalization of SGD-trained Neural Networks. It is DPO with the best checkpoint chosen on the same validation split, with checkpoint search counted against the tuning budget.
Two ablations isolate the key ingredient. The first is a one-sided small-loss filter that discards only the most contradicted pairs, which tests whether dropping already-satisfied pairs and annealing the band matter. The second is a fixed-width band with no annealing.
Why not just…?
Why not just clean the labels with a better judge? The best detector in FaithCoT-Bench reaches 65 to 77 F1, and that benchmark attributes its failures to fluent rationalizations. A better judge therefore reduces the noise but concentrates what remains on the hardest pairs, which is the regime the band objective targets. We will train both methods on judge-filtered labels. If banded training adds nothing on top of filtering, the robustness argument fails.
Why not just stop DPO early? This is a real threat, because theory says early stopping already yields $\tilde O(\sqrt{\mathrm{OPT}})$ error for shallow networks. That bound is weaker than the $C\cdot\mathrm{OPT}$ guarantee of band training for halfspaces. We therefore expect the gap to appear specifically as the concentrated noise rate rises. We test this by sweeping injected noise concentrated at large margins from 0 to 30% while holding validation data fixed. If early stopping matches banded training across the sweep, we will report that as the main finding.
Why not just use a Markovian bottleneck so the answer cannot see the question? Markovian Transformers blocks the answer from attending to the prompt and reaches 57.1% on GSM8K, about 3 to 4 points below an unblocked model. Its authors note that the bottleneck can still carry answers steganographically, meaning the model hides the answer in innocuous-looking reasoning text. The bottleneck and our objective are complementary: the bottleneck changes which models are possible, while we change how training uses labels. We will report banded training both with and without the bottleneck.
Proposed Experiments
The accuracy benchmarks are GSM8K, StrategyQA, and SVAMP, where FRIT reports higher faithfulness but no numeric accuracy. For SVAMP we found no reported numeric result on Lacuna for either FRIT or Markovian Transformers. Faithfulness is measured with the Early Answering score from Tanneru et al. and with True Thinking Scores on held-out chains. We also measure the cue-based counterfactual monitor G-mean from Counterfactual Simulation Training on SNLI and ETHICS.
The first decisive check is the stage-one diagnostic: whether labeler errors on FINE-CoT concentrate at large margins. The second is the injected-noise sweep. The predicted pattern is that DPO and early stopping degrade faster than banded training as concentrated noise grows, and that the three perform about the same under uniform noise. The third is TruthfulQA. If banded training on automated labels recovers most of the accuracy lost to faithfulness steering, the reported accuracy-faithfulness trade-off is at least partly a supervision-noise effect. If it does not, the trade-off is intrinsic to the task and our method only improves faithfulness per unit of noisy data.
References
- Learning General Halfspaces with Adversarial Label Noise via Online Gradient Descent
- FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning
- Can Aha Moments Be Fake? Identifying True and Decorative Thinking Steps in Chain-of-Thought
- A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior
- FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness
- Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning
- On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
- Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise
- Agnostic Proper Learning of Halfspaces under Gaussian Marginals
- Hardness of Learning a Single Neuron with Adversarial Label Noise
- Markovian Transformers for Informative Language Modeling
- Counterfactual Simulation Training for Chain-of-Thought Faithfulness