Is Low-Agreement Data Toxic or Just Mislabeled? A Dose-Response Audit of Consensus Filtering

When several people label the same example and disagree, the usual practice is to throw the example away. Training sets are often kept to unanimous or two-thirds-majority items, on the theory that a cleaner signal beats a larger one. The best evidence that disagreement is harmful comes from a histopathology study. There, adding the most contested images (4 of 7 pathologists agreeing) to training lowered accuracy after the moderately contested ones had helped. The authors called this a "complexity ceiling." But that observation mixes three things that normally move together. Lowering the agreement cutoff adds more data, adds examples whose majority label is wrong, and adds examples that are genuinely ambiguous even when the majority label is right. It also turns each ambiguous example into a single confident label. Nobody has separated these effects, so nobody knows whether contested examples should be discarded, relabeled, or kept with a softer training target.

We propose to borrow the study design of clinical dose-response epidemiology and apply it to annotator agreement. In those studies, a continuous exposure such as blood glucose is split into ordered groups. Each group is compared against a reference group while other factors are held fixed, and flexible spline curves locate the point where risk starts to rise. Here the exposure is how strongly annotators agreed on an example. The outcome is the change in held-out performance from adding a fixed number of examples at that agreement level to a fixed high-consensus core. Because the number of added examples is held constant, the volume confound disappears. Each agreement stratum is then crossed with how the label is encoded (majority label versus the full vote distribution) and, where ground truth exists, with whether the majority was correct. The result is a measured marginal-value curve for disagreement. We expect it to show that most of the reported harm comes from majority-wrong items and from hard label encoding, and that ambiguity itself is not the cause.

Core Claim

The claim is that the "purity over volume" rule is a confounded conclusion. Measured at matched volume, low-agreement examples whose majority label is correct have positive marginal value, and examples whose majority label is wrong have negative value. When contested examples are trained on their vote distribution instead of a hard majority label, the aggregate marginal value of the lowest-agreement stratum will be non-negative on at least one of our benchmarks. That would overturn the complexity ceiling as a property of the data. We also expect the sign of the effect to differ by class, as a subgroup effect. This mirrors how aggregate glucose quartiles in the extracorporeal-circulation cohort hid a reversed risk among diabetic patients (HR 0.56) in The relationship between blood glucose and clinical outcomes after extracorporeal circulation. If this holds, a single global agreement cutoff is the wrong design, and filtering should be driven by an estimate of majority error per stratum and class.

The claim can fail cleanly. If contested examples still hurt at matched volume, under soft labels, and within the majority-correct subset, then disagreement really is toxic in the sense Learn like a Pathologist proposed. The purity literature would then be vindicated on firmer ground.

Relation to Prior Work

Our motivation comes from work reporting that consensus predicts usefulness. RICT at the NTCIR-14 QALab-PoliInfo Task found that training on a smaller unanimous subset beat larger majority-vote training. Doan reached the same conclusion for syndromic surveillance, and Kowtha et al. found that consensus predicted performance in speech emotion recognition. All three compare datasets of different sizes and label quality at once, which is exactly the confound we remove.

The closest work is Learn like a Pathologist. It adds agreement tiers cumulatively in a curriculum on 3,152 colorectal polyp patches and reports the Stage 3 peak. That paper supplies our main dataset and the phenomenon we test. It differs from our design in three ways. Its tiers are confounded with training order, because the hardest items always arrive last. Its tiers are also confounded with cumulative volume. And it always uses hard majority labels.

HelpSteer 2 discarded about half of its annotations to reach κ = 0.79 without measuring intermediate retention levels. We treat that pruning as one point on a curve we will trace. Label Consistency-based Worker Filtering removes whole annotators using a genetic algorithm. It acts on workers rather than instances, and its authors note that it fails when workers share errors.

Two papers set constraints on the design. Learning with Noisy Labels Revisited, which introduced CIFAR-N, shows that human label noise depends on the instance and concentrates on ambiguous images. That is why agreement tier and majority correctness can come apart, and it is why we need CIFAR-10N's clean labels to measure the split. Mitigating Memorization in Sample Selection shows that systematic class confusion ("dominant noise") produces confident, low-loss wrong labels. By analogy, unanimous-but-wrong examples should exist, and no agreement filter can catch them. We will measure how common they are.

Epidemiology supplies the method. Serum Magnesium Levels in T2DM contributes quintile stratification against a reference group, adjusted comparisons, and spline-located thresholds. Gu et al. isolate a zero-consumption group before forming quintiles, and we reuse that hybrid scheme to treat unanimous examples as a distinct reference group before splitting the rest by agreement.

Research Program

The central object is a stratum marginal-utility estimate. We fix a core set C, drawn from the highest-agreement items and held constant across all runs. Let S_k be agreement stratum k, and let n be a batch size small enough that every stratum can supply it. We define

$$ U_k(e) = \mathbb{E}\left[\,M\big(C \cup B_k^{(n)},\, e\big) - M\big(C \cup R^{(n)},\, e\big)\right], $$

where $B_k^{(n)}$ is a random size-$n$ sample from $S_k$, $R^{(n)}$ is a size-$n$ sample from the reference (unanimous) stratum, $e$ is the label encoding (hard majority or soft vote distribution), $M$ is held-out performance, and the expectation is over seeds and resamples.

Positive $U_k$ means that, example for example, stratum $k$ is worth more than more unanimous data. Because the comparison is against equal-size unanimous batches, volume is controlled by construction. Training order is controlled because all examples are shuffled together.

Next, we decompose $U_k$. On CIFAR-10N, where clean labels exist, each stratum splits into majority-correct and majority-wrong subsets, and we estimate $U$ for each. The stratum's aggregate value is roughly the error-rate-weighted mix of the two. This tells us whether a negative $U_k$ reflects wrong labels or ambiguity. On the polyp and HelpSteer2 data, which lack independent ground truth, we instead cross strata with the $e$ factor. We also fit class-by-stratum interaction terms, to test whether contested examples of some classes help while others hurt.

Where agreement is closer to continuous, we fit a restricted cubic spline, meaning a smooth piecewise-cubic curve constrained to be linear at the ends. The spline is fitted to $U$ as a function of agreement to locate the crossing where $U = 0$. This applies to HelpSteer2's Likert ratings, where the spread across 3–4 raters varies finely. We report this crossing with bootstrap intervals over seeds.

The practical output is a stratum-and-class-specific retention and encoding rule derived from the curves. We evaluate that rule as an ordinary training recipe against the baselines below.

Baselines

The naive reference is vanilla training on all data with majority labels. On the polyp data, Learn like a Pathologist reports an AUC of 83.7% for this setting. On CIFAR-10N, the clean-label cross-entropy accuracy is about 93% and the Worst-tier accuracy is about 77%, per Learning with Noisy Labels Revisited. A no-filtering majority-vote aggregate has been reported only on Music_genre: 71.43% before worker filtering, from Label Consistency-based Worker Filtering.

On the polyp data, the strongest agreement-aware competitor is the four-stage agreement curriculum. It reports AUC 88.2% and κ 0.473, against 0.450 for practicing pathologists, in Learn like a Pathologist. It is the strongest competitor because it already uses exactly the annotator signal we use. We will reimplement it on identical splits, with the same backbone, epochs, and hyperparameter search budget, and confirm that we reproduce roughly 88% before comparing. Our derived rule beats it if it exceeds the reproduced curriculum AUC by more than the across-seed standard deviation over at least five seeds, and matches or exceeds its κ.

The second strong competitor is fixed-threshold consensus filtering, specifically the best of unanimous-only, ≥6/7, and ≥5/7. Its cutoff will be tuned on the same validation split and budget our method receives.

For methods that select examples by the model's own loss, the strongest is DivideMix (DivideMix). DivideMix fits a two-component mixture to per-sample losses and treats high-loss examples as unlabeled. It is the natural competitor because it needs no annotator information at all. We found no reported DivideMix result on Lacuna for CIFAR-10N Aggregate, Random, or Worst specifically; the file only states that robust methods lose 1–5% on human versus synthetic noise. We will therefore run it ourselves with its published settings on the same splits. We will also run Criteria_ALL (Mitigating Memorization in Sample Selection), since the dominant-noise argument makes it the relevant competitor for unanimous-but-wrong items.

On HelpSteer2, the reference is the released filtered data. HelpSteer 2 reports 88.8% on Reward-Bench for Llama-3-70B and 92.0% for Nemotron-4-340B. For compute reasons we will train a smaller base model. We found no reported result on Lacuna for small-base reward models on HelpSteer2, so our comparison is between our rule and the published 50%-pruning rule on the same small base.

Two ablations isolate the key ingredients. The first removes volume matching and adds strata cumulatively, as the curriculum does. If that alone recreates the Stage 4 dip, volume and order explain the ceiling. The second applies soft labels to all data with no stratification. This tests whether the encoding change does all the work.

Why not just…?

Why not just tune the agreement cutoff on a validation set? With four agreement levels on the polyp data, a grid search over cutoffs costs four runs and will find the best global threshold. We do not claim to beat it at that task. A cutoff answers "where do I cut?" but cannot tell whether a stratum is harmful because its majority labels are wrong or because it is ambiguous. That distinction decides whether to buy more annotations per item or more items. A per-class cutoff grid also grows combinatorially with the number of classes. If the class-by-stratum interactions turn out to be null and our derived rule reduces to the tuned global cutoff, we will report that as the finding.

Why not just train on the soft vote distribution for everything? This is the most serious threat, because it may dissolve the ceiling by itself. It is our second ablation. If soft labels on all data match our stratum-specific rule on every benchmark, the practical conclusion is simply "stop filtering, use soft labels," and the curves serve as the explanation rather than as a method. The case where stratification still matters is the majority-wrong subset on CIFAR-10N. There the vote distribution can still put most of its mass on the wrong class, and we can measure directly whether soft labels mitigate or merely dilute that harm.

Why not just use DivideMix and ignore annotators? Loss-based selection assumes networks learn clean patterns first. Mitigating Memorization in Sample Selection shows this breaks under structured noise: MentorNet reaches 17.6% error against 12.2% for class-penalized selection on 40% pair-flip CIFAR-10. Instance-dependent human noise is memorized faster still (Learning with Noisy Labels Revisited). Even so, this is an empirical question on the specific benchmarks, and the head-to-head CIFAR-10N runs will settle it. We will also test DivideMix combined with our rule, since the two use different signals.

Why not just weight each example by its agreement? Agreement weighting presumes a monotone relationship between agreement and value, which is the hypothesis in question. If majority-correct contested items have positive value, down-weighting them is wrong. We include agreement-proportional weighting as a baseline, with the weighting exponent tuned on the same validation budget.

Proposed Experiments and Decisive Checks

On the polyp dataset (4 strata), CIFAR-10N (agreement 3/3 versus 2/3, with clean labels), Music_genre, and HelpSteer2 (continuous rating spread), we estimate $U_k$ under both encodings over at least five seeds and several resamples of each batch. We report stratum effects against the unanimous reference with bootstrap intervals, following the reference-group logic of the magnesium quintile study.

The first decisive check is the polyp Stage 4 test. The complexity ceiling is an artifact if $U_{4/7}$ becomes non-negative once volume is matched and soft labels are used. It is real if $U_{4/7}$ stays negative under both. The second is the CIFAR-10N decomposition. We predict $U > 0$ for majority-correct 2/3 items and $U < 0$ for majority-wrong ones, and we will report the frequency and harm of unanimous-but-wrong items as a limit on any consensus filter. The third is effect modification: a class-by-stratum interaction significant after correction would show that a global cutoff is mis-specified. The fourth is the HelpSteer2 spline. We check whether the $U = 0$ crossing falls near the retention level implied by the published 50% pruning, or whether the pruning discarded net-useful ratings.

Finally, the derived rule is compared with the reproduced curriculum, the tuned cutoff, DivideMix, Criteria_ALL, and soft-labels-everywhere under identical compute and tuning budgets. The proposal succeeds if the mechanism checks come out clearly one way or the other, whether or not the derived rule wins.