Collection-Time Metadata as a Map of Label Noise

Methods that correct biased labels need to know how the labels went wrong. They need an estimate of how often each true class gets recorded as each observed class, for each demographic group. Such an estimate is called a noise transition matrix. In practice these matrices are either assumed known or guessed from the same noisy data they are meant to fix. We propose getting them from a different source: the circumstances under which each label was made. That means which annotator labeled the item, which version of the instructions they saw, which scanner produced the image, and whether the label came from a rule-based text parser or a radiologist. These facts are cheap to log at collection time and cannot be recovered afterward. Our bet is that, once you condition on them, label errors become simple enough to estimate from a small, targeted audit.

The core claim is testable. Label noise that looks instance-dependent across a whole dataset, where the chance of an error depends on complicated features of each item, is often approximately class-conditional within strata defined by collection-time metadata. If that holds, three things follow. A proactive documentation schema that logs the right fields turns an intractable correction problem into many small tractable ones. A fixed budget of expert relabeling is better spent stratified by those fields than spread at random or by demographic group. And the correction itself does not need sensitive attributes at training time, because the strata come from provenance rather than demographics.

Where this sits

Two groups of prior work motivate the gap. On the label-bias side, On Group Sufficiency Under Label Bias proves that calibrating across groups against biased labels guarantees miscalibration against the true labels. It introduces CMI-REG, a mutual-information regularizer with backward correction through a group-specific transition matrix, and it assumes noise depends only on class and group. Bias-Aware Mislabeling Detection via Decoupled Confident Learning and Mitigating Label Bias via Decoupled Confident Learning introduce DeCoLe, which trains a separate classifier per demographic group to set group-specific noise thresholds, and therefore requires those demographic labels.

Neither line asks where a better partition of the data could come from. Two further papers suggest where. Don't Blame the Annotator shows that shallow patterns in annotation instructions (72.7% of instruction examples) propagate into 61% of collected data, which means the instructions themselves are a variable that drives error. Exploring the interplay of label bias with subgroup size and separability shows that scanner manufacturer shapes how label bias spreads through learned representations on the EMBED mammography dataset. It also shows that thresholds tuned on biased validation data collapse the majority group's true-positive rate from 0.898 to 0.518.

The documentation side supplies the method for capturing such variables. Datasheets for datasets and the CheXpert datasheet record labeling pipelines only as prose. The CheXpert datasheet, for instance, describes the automated report labeler, the "Uncertain" label, and the eight-radiologist test consensus, but offers no computable link to training. Mobus shows that provenance can be a machine-checked invariant: it writes cell-level transformation manifests and refuses to run on unverified inputs. Mobus tracks licenses and units, not labeling conditions.

We borrow Mobus's "fail loud" manifest idea and apply it to label provenance. We borrow the planning-stage discipline of From Planning Stage Towards FAIR Data to fix the logged fields before collection starts.

One piece of work constrains the design. When More Data Lead Us Astray shows that uncertainty-driven acquisition under label bias pulls in exactly the most prejudiced labels. It produces an apparent fairness improvement while the true gap widens. Our audit allocation therefore must not be driven by model uncertainty. It is driven by stratum statistics.

Research Program

The program has three components.

A label-provenance manifest. Each label is stored with a fixed set of collection fields: annotator identifier, instruction version, labeling source (human or rule-based), acquisition device and site, and time spent on the item. A stratum $s$ is a cell in a coarsening of these fields. We choose the coarsening by merging cells whose audited error patterns do not differ significantly, so strata stay large enough to estimate. Following Mobus, training refuses to proceed if any stratum holding more than a set fraction of the data has no audited items. This turns silent under-coverage into an explicit error.

Stratified audit and estimation. Given an expert relabeling budget $B$, we first spend a small pilot share uniformly. We then allocate the rest across strata in proportion to $n_s \hat\sigma_s$, where $n_s$ is the number of items in stratum $s$ and $\hat\sigma_s$ is the pilot estimate of the standard deviation of that stratum's error indicator. This is the Neyman allocation from survey sampling, which minimizes the variance of a stratified estimate for a fixed total sample. From the audited items we estimate each stratum's transition matrix $\hat T_s$, with entries $\hat T_s[i,j] = \hat P(\tilde Y = j \mid Y^* = i, S = s)$. Here $\tilde Y$ is the observed label and $Y^*$ the true one. We shrink $\hat T_s$ toward a pooled estimate when a stratum is small.

Stratum-wise correction. Training uses backward-corrected loss per stratum:

$$\mathcal{L} = \frac{1}{n}\sum_{k=1}^{n} \Big[\hat T_{s_k}^{-1}\,\boldsymbol{\ell}(f(x_k))\Big]_{\tilde y_k},$$

where $\boldsymbol{\ell}(f(x_k))$ is the vector of losses for predicting each possible class on item $x_k$, and $s_k$, $\tilde y_k$ are that item's stratum and observed label. When the evaluation target is group calibration, we add CMI-REG's regularizer unchanged, feeding it $\hat T_{s}$ in place of $T_g$. Demographic attributes are needed only on the evaluation set.

The analysis component tests the central assumption directly. On audited items within each stratum, we fit a classifier that predicts "this label is wrong" from input features. We report how much of its predictive power remains after conditioning on stratum, compared with conditioning on demographic group alone. If within-stratum error is still strongly feature-predictable, the class-conditional assumption fails for that data. That result is itself worth reporting.

Benchmarks

We need data where true labels are recoverable for some items and collection metadata exists or can be simulated faithfully. We use four settings.

CheXpert. Training labels come from the rule-based report labeler, and the radiologist-consensus validation and test sets serve as the true labels. Available strata are labeler source, "Uncertain" flags, and site or view fields.

EMBED. We use manufacturer as the provenance field and repeat the 30% flip protocol of Exploring the interplay. We extend it so that flip rates also vary within a manufacturer by a simulated reader identity, which makes the noise instance-dependent unless reader identity is logged.

Hate speech. We use the Kennedy et al. data used by DeCoLe, with its Rasch-measurement gold labels as $Y^*$, and annotator-level strata to the extent annotator identifiers are released.

Prospective crowdsourcing study. This is the only setting that genuinely tests the proactive claim. We collect an NLU labeling task modeled on Don't Blame the Annotator. Instruction versions with controlled example diversity are randomized across workers, and every item is also expert-labeled, so the full $Y^*$ is known and the budget $B$ is simulated by subsampling.

Baselines

The naive reference point is ordinary training on observed labels, with thresholds tuned on a biased validation split. This reproduces the collapse regime reported on EMBED.

The strongest existing method for the group-calibration objective is CMI-REG from On Group Sufficiency Under Label Bias. That paper reports reducing the sufficiency gap by up to 7.2% across eight datasets with no significant loss in AUROC. We run it with its transition matrices estimated from a random audit of the same budget $B$, on the same backbone, with the same hyperparameter search budget. Beating it means a lower sufficiency gap measured against $Y^*$ at equal $B$, without lower AUROC against $Y^*$.

For detecting mislabeled items, the strongest competitor is DeCoLe, run with true demographic labels, which our method does not use. The paper reports 3–4× higher recall on disadvantaged groups than Confident Learning and Co-Teaching on the hate speech data. The file gives no scalar value; no reported result found on Lacuna for DeCoLe's absolute recall. We compare precision and recall of flagged items per group against $Y^*$.

Three ablations isolate the key ingredient, which is the metadata-defined partition:

  1. Replace strata with demographic groups while keeping stratified audit and correction. This tests whether provenance beats demographics as a partition.
  2. Keep the strata but allocate the audit uniformly. This tests whether targeting matters.
  3. Use DeCoLe with strata substituted for groups. This tests whether the partition helps an existing detector without our correction.

All methods see identical data, budget, and tuning.

Why not just…?

Why not just audit a random subset and estimate one matrix per group? This is the real threat, and it is ablation 1 combined with the CMI-REG baseline. If collection metadata does not explain noise beyond what demographics do, random group-level auditing will match us, and the proposal's value reduces to the manifest's bookkeeping. The within-stratum predictability analysis and the prospective study, where we control how instruction versions induce errors, decide this. We will report the budget $B$ at which the two converge.

Why not just feed the metadata to the model as extra input features? Conditioning predictions on annotator or scanner identity teaches the model to reproduce each annotator's bias, not to remove it. A model given the annotator ID learns $P(\tilde Y \mid x, s)$, not $P(Y^* \mid x)$, and the metadata is unavailable at deployment anyway. We include this as a baseline, expecting good observed-label accuracy but no gain against $Y^*$.

Why not just collect more data, or acquire labels actively? Is Underrepresentation overemphasized? finds that systematic label error harms group fairness far more than small sample size does. When More Data Lead Us Astray shows that uncertainty sampling makes the true gap worse while the observed gap shrinks. We include uncertainty-driven audit allocation as a comparison and report both the observed and true gaps, so that this failure mode is visible.

Why not just use post-hoc detectors like influence functions? Detecting labeling bias using influence functions finds nearly 90% of flips at under 1% false positives on MNIST49. On CheXpert, however, separation becomes "less distinct" because of last-layer Hessian approximations, and CheXpert is our setting. We include it as a detection baseline there.

Decisive Checks

The first check is whether the proposal holds up against its most dangerous competitor. On the prospective study and on EMBED with simulated reader noise, stratified correction must beat random-audit CMI-REG on the sufficiency gap against $Y^*$ at small budgets, around 1–5% of items audited. If it does not, the central claim fails.

The second check is the assumption test: within-stratum flip predictability should fall sharply compared with group-only conditioning. Where it does not, as we suspect may happen for CheXpert's report labeler, we expect the method to lose its advantage, and we would say so.

The third check concerns the observed-versus-true gap. Throughout, we report the difference between the fairness gap measured on observed labels and the gap measured on $Y^*$. A method that shrinks only the observed gap has not fixed anything.