Trusting Preference Labels Nobody Can Fully Read

Teaching a language model to answer questions that chain several facts across a 30,000-token document needs preference data: pairs of answers where one is marked better. At that length, people cannot produce those labels reliably, and LLM judges struggle too. The labels that do get made are noisy, and the noise is not random. It concentrates on the hardest items: questions needing three or four reasoning steps, with evidence scattered through the document. The tools built to clean noisy preference labels estimate which labels are wrong from the model's own likelihoods, and this reliance is a problem here. On exactly these hard items, a correct answer that uses a rare bridging entity looks less likely to the model than a fluent shortcut answer. The denoiser may then "correct" good labels into bad ones.

This note proposes a different source of trust. A wrong label on a multi-hop answer usually leaves a checkable trace: one of the answer's intermediate claims fails when compared against the short passage it should come from. Checking one claim against one paragraph is a short, easy judgment, unlike judging the whole answer against the whole document. We combine three imperfect signals into a per-pair estimate of how likely the label is flipped: a holistic long-context judge, short claim-by-claim checks, and the policy's own margin. We then train with a noise-corrected version of Direct Preference Optimization (DPO, a method that tunes a model directly on preference pairs without a separate reward model). The correction uses that per-pair flip rate and plugs into a short-to-long training scheme that avoids running full-length preference pairs through the model.

Core Claim

The claim has two parts, and the first can be tested before anything is built.

The diagnostic part is that, in long-context multi-hop preference data, label noise rises with hop count and evidence dispersion. Likelihood-based denoisers, which score each label by how probable the policy finds each answer, lose precision on exactly those items. They do worse than they do on short conversational data.

The method part is that a flip probability per pair, estimated from verification signals whose errors differ in structure, yields better long-context multi-hop models under noise than likelihood-based correction. The estimate comes from agreement between the holistic judge and hop-level short-evidence checks, with the policy margin as a third vote.

We expect the largest gains on three- and four-hop questions. If the gains are uniform across hop counts, the mechanism story is wrong even if the method helps.

What Exists and What It Misses

The training scheme we build on comes from SoLoPO. SoLoPO runs preference optimization on short evidence-only contexts built from MuSiQue. It adds a chosen-only term that makes the model's implicit reward for the chosen answer match whether it sees the snippet or the full distractor-filled document. SoLoPO beat vanilla preference optimization in 28 of 32 settings and raised trainable length from 9,000 to 19,000 tokens. It assumes its labels are clean, though. Because the alignment term is applied only to the chosen answer, a flipped label is actively propagated: the model learns to reward the wrong answer consistently across long contexts. We keep SoLoPO's structure and change what counts as "chosen" and how much each pair is trusted.

The noise-robust methods split into two families, and both constrain our design.

Provably Robust DPO gives an unbiased loss correction for a known flip rate. It matched clean performance on IMDb at 40% noise (mean reward 0.99 against 0.88 for DPO), but it assumes one global flip rate. We reuse its estimator with a per-pair rate instead.

PLC-DPO and PerpCorrect infer flips from policy margins or perplexity differences, and are strong on conversational data. PLC-DPO reports a 60.5% mean win rate over 57 cells against 55.5% for rDPO. PerpCorrect reports 94.92% reward accuracy at 40% noise on Golden HH against 53.15% for DPO. Both rest on the premise that the better answer is the one the model already finds likelier. That premise is what the diagnostic part of this proposal tests in the long multi-hop regime.

Uncertainty-Penalized DPO down-weights ambiguous pairs using an ensemble of five reward heads. It was tested only at GPT-2-Medium scale because the ensemble is costly over long sequences, and that cost is a reason we avoid ensembles.

Two papers motivate the use of short, local checks. LongReward observes that verifying responses against 100,000-token documents is nearly impossible for humans, and How Reliable Is Human Feedback documents fatigue-driven flips in forced-choice labels. Both point to the cause: judging a whole answer against a whole document is the hard step.

Two works show that localized checking can give dense, reliable signal. LongRLVR shows that rewarding only final answers starves evidence localization of gradient in 64,000-token inputs, and that chunk-level evidence rewards fix it. It relies on teacher-generated chunk labels and on full reinforcement learning. Step-DPO shows that locating the first wrong step makes preference training sample-efficient, but only for short math problems with programmatic checkers.

We borrow the idea of localized verification from both. We use it neither as a reward nor as the label itself, but as one noisy witness about whether the holistic label is right.

Research Program

The data substrate follows SoLoPO's construction. MuSiQue questions, which compose two to four single-hop questions and filter out shortcut-answerable ones (MuSiQue), have their supporting paragraphs embedded among distractor documents. The resulting contexts span 16,000 to 32,000 tokens.

Candidate answers are sampled from the policy itself, following Step-DPO's finding that in-distribution pairs train better. Answers are required to cite the chunks they rely on. MuSiQue's gold answers and decompositions give clean labels that are hidden from training and used only for measurement. This lets us study realistic noise with ground truth available.

We study two noise regimes. In the synthetic regime, labels are flipped at a uniform rate. In the realistic regime, labels come from an open-weight LLM judge that reads the full long context and picks the better answer, as a long-context pipeline without gold would do.

Each pair receives three signals.

  • Signal A is the holistic judge's verdict.
  • Signal B comes from claim-level checking. The judge model splits each answer into bridge claims. It checks each claim only against the short chunk the answer cited, or against the best-matching chunk if none was cited. It then scores the answer by the fraction of its claims that are supported. Each check sees a few hundred tokens rather than tens of thousands.
  • Signal C is the reference-normalized policy margin, the quantity PLC-DPO uses.

We fit a Dawid–Skene model. This is the standard latent-class model for combining noisy annotators, which learns each annotator's confusion rates without gold labels. Three conditionally independent annotators make it identifiable. We give it separate confusion matrices per stratum of estimated hop count and context length, so a signal can be trusted on two-hop items and distrusted on four-hop items. Its output is a posterior flip probability $\epsilon_i$ for each pair $i$. A 200-pair audited set, with labels checked against gold, is used only to check calibration, never to fit.

Training plugs $\epsilon_i$ into the rDPO estimator. Write $\mathcal{L}(y_w, y_l)$ for the standard DPO loss on the short context. The short-context term becomes

$$\mathcal{L}_i^{\text{short}} = \frac{1-\epsilon_i}{1-2\epsilon_i}\,\mathcal{L}(y_w, y_l) - \frac{\epsilon_i}{1-2\epsilon_i}\,\mathcal{L}(y_l, y_w).$$

To avoid the variance blow-up of this estimator near $\epsilon_i = 1/2$, we clip $\epsilon_i \le 0.4$. Pairs whose posterior is near a tie are dropped from this term.

The long-context alignment term is applied to the posterior-preferred answer $\hat y_i$ and weighted by confidence:

$$\mathcal{L}_i^{\text{RA}} = |1 - 2\epsilon_i|\,\big(r_\theta(x^{\text{long}}_i, \hat y_i) - r_\theta(x^{\text{short}}_i, \hat y_i)\big)^2.$$

Here $r_\theta(x,y) = \beta \log \frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ is DPO's implicit reward: $\pi_\theta$ is the policy being trained, $\pi_{\text{ref}}$ is the frozen reference model, and $\beta$ is the KL-regularization strength. The weight $|1-2\epsilon_i|$ ensures that uncertain pairs do not anchor long-context behavior.

The extra cost over SoLoPO is the offline claim checks, which are short forward passes, plus a negligible EM fit. No extra long-sequence passes occur during training.

Baselines

The do-nothing reference is plain SoLoPO with DPO, trained on the noisy labels as given. Beside it we report an oracle: SoLoPO trained on gold labels. The gap between them is the damage noise does, and the fraction of that gap each method closes is our headline measure.

The strongest existing competitor is SoLoPO with its short-context term replaced by PLC-DPO. PLC-DPO is the best-performing denoiser in the file, and it adds almost no compute because it reuses the DPO forward pass. That makes it the method our approach must beat on equal footing.

We also run SoLoPO with PerpCorrect preprocessing, given the same 50 clean anchors it requires. We run SoLoPO with rDPO at a single global $\epsilon$, tuned on the audited set.

All methods use the same base model (Qwen2.5-7B-Instruct), the same pairs, and the same number of optimizer steps. Each gets the same tuning grid over $\beta$ and SoLoPO's balancing weight, chosen on a held-out MuSiQue dev split.

On the conversational benchmarks where it was evaluated, PLC-DPO's reported number is a 60.5% mean win rate. No reported result was found on Lacuna for PLC-DPO, PerpCorrect, or rDPO on long-context multi-hop QA, and none for SoLoPO under label noise. These cells are therefore new.

Beating PLC-DPO means two things. Our method must close at least half of the remaining gap to the oracle on MuSiQue-long F1. Its lead must also be concentrated on three- and four-hop items, with non-overlapping confidence intervals across three seeds.

As an external reference from a different training paradigm, LongRLVR reports LongBench v2 accuracy of 46.5 for Qwen2.5-14B-1M, against 39.8 for naive verifiable-reward RL. The LongBench v2 paper reports 39.4% for Qwen2.5-72B-Instruct. We report LongBench v2 to place our 7B model against these numbers, not to claim a like-for-like win.

Three ablations isolate the key ingredient. The first removes Signal B, leaving Dawid–Skene over the judge and the margin only. The second removes stratification, using one confusion matrix for all items. The third keeps $\epsilon_i$ in the short-context term but restores SoLoPO's original chosen-only alignment term, which measures how much of the gain comes from no longer propagating flipped labels into long contexts.

Why not just…?

Why not just plug PLC-DPO into SoLoPO and be done? This is the obvious composition and we run it as the main baseline. Our objection is specific: its routing depends on policy and reference margins. The reference models in this regime score under 40% on LongBench v2, and PerpCorrect's own authors flag that valid technical reasoning can carry higher perplexity than distractor text. Experiment 1 below measures PLC-DPO's flip-detection precision by hop count. If its precision does not degrade on three- and four-hop items, this proposal's premise fails, and the right paper is the simple composition.

Why not just use the claim checks as the label, or keep only pairs where the judge and checks agree? This is a real threat. If claim-level verification were near-perfect, using it directly would match our method at lower complexity. It is not near-perfect. It cannot see whether the answer's facts combine correctly, and claim splitting introduces its own errors. Agreement filtering also discards data in a regime already starved of long examples: Scaling Long Context Training Data by Long-Distance Referrals estimates long documents at under 0.1% of web data, and PLC-DPO reports that correcting labels beats discarding them. We run both alternatives as baselines, "checks-as-label" and "agreement filter". If either matches our method within noise, we will report the contribution as the diagnostic plus the simpler recipe.

Why not skip preferences and do LongRLVR-style reinforcement learning with chunk rewards? LongRLVR's chunk rewards depend on teacher-assigned evidence labels, which carry the same attribution noise we model, and it needs on-policy long rollouts. The scarcity literature names exactly these rollouts as the main cause of out-of-memory failures on research hardware. Our method stays offline and closed-form. We do not claim it beats LongRLVR's absolute numbers at 14B. We compare at 7B on RULER-QA only if compute allows a faithful LongRLVR reproduction, and otherwise state the comparison as open.

Why not just train on gold MuSiQue labels, since they exist? We do, as the oracle. Gold exists only because MuSiQue was built by composing known single-hop questions. The regime we care about is real long documents with LLM-judged labels, and MuSiQue is our controlled proxy for that regime. Experiment 4 tests whether the learned noise structure transfers to documents without gold.

Proposed Experiments

Experiment 1 (the diagnostic). We generate realistic judge labels on 20,000 long MuSiQue pairs. We measure the true flip rate by hop count and context length. We then measure the precision and recall of PerpCorrect, PLC-DPO's posterior, and our Dawid–Skene posterior at detecting flips, both overall and by stratum. We also split by whether the wrong answer is reachable through a single-paragraph shortcut. The predicted signature has three parts: flip rate rises with hops; likelihood-based precision falls with hops and is worst on shortcut-reachable pairs; the combined posterior holds steady.

Experiment 2 (synthetic uniform noise). We train every method at 10%, 20%, 30%, and 40% uniform flips. This regime favors rDPO's assumptions, so we expect parity there. A large win for us would itself be suspicious.

Experiment 3 (realistic judge noise). We train under the judge's natural labels. We evaluate on held-out MuSiQue-long (answer F1, broken down by hop count), RULER-QA, and LongBench v2. HotpotQA and 2WikiMultiHopQA in long-distractor form test transfer. Short-context Open LLM Leaderboard tasks check for regressions, as SoLoPO did.

Experiment 4 (transfer). We fit the noise model on MuSiQue. We apply it to judge-labeled pairs over real long documents built from LongBench v2-style sources, and audit 200 pairs by hand.

The decisive check is Experiment 3's hop-stratified gap. The proposal succeeds if our method beats SoLoPO with PLC-DPO mainly on three- and four-hop questions under realistic noise, while the agreement-filter and checks-as-label baselines fall short. It fails if PLC-DPO's detection precision holds up across hops in Experiment 1, or if agreement filtering matches us in Experiment 3.