Robust training objectives with a "redescending" influence function treat a training example the way a careful juror treats an implausible witness. They listen to mild disagreement, but once an example contradicts the model's consensus strongly enough, its pull on the parameters falls back to zero. Recent work turned this property into an unsupervised data-quality tool for language-model alignment. It trains on pairs of responses where one is labeled preferred, with no clean reference labels, and it both estimates what fraction of those pairs are corrupted and ranks individual pairs by how trustworthy they are. Meanwhile, alignment pipelines increasingly get these preference pairs from other models rather than from people. Pairs produced this way fail in a structured way. They carry systematic style artifacts that downstream models learn to exploit instead of learning safety.
Our central observation is that these two developments collide. A redescending objective finds outliers relative to the consensus of the data it is trained on. When the corruption is the consensus, as it is for systematic synthetic artifacts, the audit should report the data as clean. This is the robust-statistics version of the "generative data leakage" problem: evaluating synthetic data against itself is over-optimistic. We propose to fix it the way that problem was fixed in tabular synthetic data, by evaluating across independent sources. Each pair from one synthetic source is scored by redescending models trained on other sources, and the scores are combined with a rule that a single like-minded source cannot dominate. The result is two things: an audit score for a synthetic preference corpus that should predict real jailbreak vulnerability, and a filter that should recover the safety of conservative pipelines without throwing away multi-model data.
Core Claim
Scalable Valuation of Human Feedback through Provably Robust Model Alignment introduced Hölder-DPO, a preference objective derived from Hölder divergence whose per-pair gradient vanishes once the model strongly disagrees with a pair's label. It shows the objective survives 40% injected label flips with an average reward of about 0.94. It also derives a closed-form estimate of the contamination ratio and reports roughly 25% noise in Anthropic HH. That paper supplies our method. It also states our starting limitation: it assumes the label errors are independent and identically distributed, and says it does not handle structured noise.
More is Less supplies the motivating failure. It found that synthetic safety preference data built with a stronger external model makes Llama-2-7B far easier to jailbreak. The attack success rate (ASR, the fraction of adversarial prompts that elicit harmful output) on AdvBench is 0.53 for GPT-4o+Self data, against 0.10 for Self+RM data and 0.22 for human-labeled data. The authors trace this to chosen and rejected responses being linearly separable on surface features such as length and formatting.
Combining the two leads to a specific, falsifiable claim. Shortcut pairs are ones the model fits easily, so the model strongly agrees with them. A self-referential redescending audit should therefore assign them full weight and estimate low contamination for exactly the most dangerous corpora. A cross-source audit should separate the regimes in the order their ASR does.
Research Program
Suppose we have preference corpora from $K$ sources: self-generated with a reward model, peer-model, GPT-4o-mixed, human-curated mixed, and human-labeled, all over the same safety prompts. For each source $j$ we train an anchor policy with Hölder-DPO on that source alone. Running anchor $j$ on a pair $i$ from another source $k$ yields a redescending weight $w_j(i) \in [0,1]$. This is the factor by which the objective shrinks that pair's gradient: it is near 1 when the anchor finds the label plausible and near 0 when the anchor rejects it as an outlier. We define the cross-source trust of a pair and the cross-source contamination of a corpus as
$$ s(i) = \operatorname{median}_{j \neq k} \, w_j(i), \qquad \hat\epsilon_k = 1 - \frac{1}{n_k}\sum_{i \in k} s(i), $$
where $n_k$ is the number of pairs in source $k$. We use the median, not the mean, on purpose. Synthetic Data, Real Errors leaves open how to handle a generator whose errors are shared across sources. The median makes the aggregate itself robust, so that one anchor sharing source $k$'s bias (for instance a peer model with the same formatting habits) cannot rescue a pair.
The program has three parts. The first is building the leave-one-source-out audit and comparing $\hat\epsilon_k$ against the self-referential estimate from the original Hölder-DPO recipe. The second is using $s(i)$ to weight or filter the pooled multi-model corpus before ordinary DPO training. The third is analyzing which pairs get rejected.
That analysis matters because of a constraint from the synthetic-data literature. On the Equivalency, Substitutability, and Flexibility of Synthetic Data found that the most useful synthetic data is the most behaviorally complex. Multi-group synthetic scenes raised DanceTrack HOTA from 68.8 to 73.5, while simple scenes hurt. Any consensus-based filter risks discarding rare but valuable examples, here refusals to unusual attack styles. So we will check whether rejected pairs concentrate on rare harm categories rather than on artifact-driven ones.
A further limitation must be stated plainly. Per-pair reweighting cannot break a confound present in every pair. If every chosen response is GPT-4o-styled, filtering removes only the pairs whose label is unsupported by content. It does not remove the style correlation itself. The experiments below are built to show how much of the ASR gap lies in the part that filtering can reach.
Baselines
The naive reference is standard DPO on the pooled GPT-4o+Self corpus, as in More is Less. The reported Llama-2-7B AdvBench ASR is 0.53. We rerun it under our prompts and seeds rather than relying only on the reported value. The conservative target is Self+RM DPO, with a reported ASR of 0.10. Matching that number while keeping the general-capability gains of multi-model data is what would count as success.
The strongest existing method on the question of noisy preferences is self-referential Hölder-DPO from Scalable Valuation of Human Feedback. This means training on the pooled corpus, then either using the robust objective directly or applying its recipe of filtering the lowest-ranked 25% of pairs and retraining. It is the most direct competitor because it already does unsupervised valuation with a redescending objective. Its reported result is an average reward of about 0.94 under 40% injected noise. No reported result was found on Lacuna for its ASR on synthetic multi-model safety data. The fair comparison gives it the same Llama-2-7B base, the same pooled data, and the same $\gamma$-tuning budget as our anchors. Beating it means a lower ASR at equal retained data and equal general-benchmark scores.
A shortcut-probe filter tests whether the robust machinery is needed at all. We train a logistic regression on surface features (length difference, formatting markers, source identity) to predict chosen versus rejected, then drop the pairs it predicts most confidently, removing the same fraction of pairs as our method.
A frontier-judge filter asks a strong LLM whether each chosen response is actually safer, and drops the pairs it rejects.
Two ablations isolate the key ingredients. In the first, cross-source anchors are trained with plain DPO, and their margin is used with a fixed threshold instead of a redescending weight. This tests whether redescending matters or only the cross-source anchoring does. In the second, the median aggregate is replaced by the mean, which tests the robust aggregation.
All methods get the same base model, the same number of DPO steps, the same tuning budget, and three seeds.
Why not just…?
Why not just use Self+RM data, which already gets ASR 0.10? This is a real threat. More is Less itself notes that restricting pipelines to self-generated data gives up the richer diversity that multi-model data provides on general benchmarks. Our proposal only earns its place if filtered multi-model data matches Self+RM on ASR and beats it on UltraFeedback, ARC, and HellaSwag. If it matches only on ASR, the honest conclusion is that Self+RM suffices. In that case the cross-source audit's value reduces to diagnosis, to predicting which corpora are unsafe before training.
Why not just filter by surface-feature separability? More is Less shows that separability correlates with vulnerability, so a probe filter may capture most of the effect. We expect it to fall short in two places. It cannot see content-level errors that carry no surface signature. It also removes pairs where the safer response happens to be longer, which is a real signal. The controlled injection experiment below separates these cases. If the probe matches our filter there and on AdvBench, we will report that the cheap probe wins.
Why not just run Hölder-DPO, or a bounded-loss robust DPO, on the pooled data? Our core prediction is that both fail for the same structural reason. They define outliers relative to a consensus that the artifacts have already captured. The original paper flags structured noise as outside its scope, and bounded-influence variants already plateau below it under independent flips. The first decisive check tests this directly.
Why not just ask a frontier LLM to verify each pair? For safety labels a judge may do better than it does on math. Still, MathClean found that models such as GPT-o1-preview and DeepSeek-R1 score below 80% at detecting errors in synthetic answers, and a judge from the GPT-4o family may share the very style preferences we are trying to remove. We include it as a baseline and also as one anchor, so that its correlation with the GPT-4o source can be measured.
Proposed Experiments
The first decisive check is a controlled injection experiment on Anthropic HH-RLHF harmless-base and SafeRLHF prompts. We flip labels on a known fraction of pairs and mark the flipped chosen responses with a consistent artifact: added length and a formatting signature. We then also run independent flips at the same rate. Self-referential Hölder-DPO should recover the independent rate but underestimate the structured one. The cross-source $\hat\epsilon$ should recover both. Recall of the true flipped pairs is the metric. This mirrors the gap reported in Synthetic Data, Real Errors, where a single-generator evaluation predicted AUC 0.841 against a real AUC of 0.738 on Mexican Covid-19 data.
The second check is predictive validity. Across the five generation regimes of More is Less, we compute self-referential and cross-source contamination and correlate each with measured AdvBench ASR. The claim fails if cross-source $\hat\epsilon$ does not order Self+RM below Peer+RM below the GPT-4o and HC mixes. The reported ASRs are 0.10, 0.39, 0.51, and 0.53.
The third check is the training comparison against every baseline above. Primary outcome is AdvBench ASR. Capability is measured on UltraFeedback, ARC, and HellaSwag, toxicity on TOXIGEN, and helpfulness by win rate on Golden HH. Each method is evaluated with and without length-matched rejected responses, to measure how much of the remaining ASR comes from a corpus-wide confound that no per-pair weighting can reach.
The final check is the rejected-set analysis. We test whether pairs with low $s(i)$ concentrate in rare harm categories, which would indicate the filter is losing useful complex data, or in artifact-heavy pairs, which is what it should remove.
References
- Scalable Valuation of Human Feedback through Provably Robust Model Alignment
- More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment
- Synthetic Data, Real Errors: How (Not) to Publish and Use Synthetic Data
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- MathClean: A Benchmark for Synthetic Mathematical Data Cleaning