Narrow Subnetworks as Built-In Label Auditors

A width-elastic ("slimmable") network is a single set of weights that can run at several widths, for example a quarter, half, three-quarters, or all of its channels. Separately, benchmark datasets contain many wrong labels. ImageNet's validation set is roughly 6% wrong, TACRED's labels are 23.9% wrong, and wide networks are expressive enough to memorize these errors. This proposal asks whether a slimmable network can find those wrong labels by comparing its own widths. The intuition is that a narrow subnetwork lacks the spare capacity to memorize an idiosyncratic wrong label, while the full-width network memorizes it. The examples where the full width agrees with the given label but the narrow widths confidently prefer another class should then be disproportionately label errors. If this holds, a single training run yields the per-example "does the label look wrong?" probabilities that label-cleaning methods currently buy with k-fold cross-validation or with two separately trained peer networks.

There is a catch, and handling it is the new part of the idea. Standard slimmable training shares weights so that the wide network teaches the narrow one. That sharing may leak memorized errors into the narrow widths and erase the very disagreement we want to use. We therefore compare ordinary sharing against a one-way variant. In the one-way variant, the narrow channels are updated only by the narrow-width loss. The wide network can read the narrow network's features but cannot push gradients into them. The same model also lets us test, within one family of weights, the "model reversal" that earlier work reported only across separately trained architectures: larger models look better on noisy test labels and worse on corrected ones.

Core Claim

We claim two things. First, the gap between full-width and narrow-width confidence in the given label is a competitive label-error score. On human-verified error sets it should match cross-validated Confident Learning at roughly half the training compute. Second, whether this works depends on how weights are shared: one-way sharing keeps the narrow widths from memorizing errors, while standard slimmable sharing partly does not. Concretely, let $p_w(y \mid x)$ be the model's predicted probability of class $y$ for input $x$ when run at width fraction $w$, and let $\tilde y$ be the given, possibly wrong, label. The score is

$$ s(x,\tilde y) = p_{1.0}(\tilde y \mid x) - \frac{1}{|W|}\sum_{w \in W} p_w(\tilde y \mid x), $$

where $W$ is the set of narrow widths, for example $\{0.25, 0.5\}$. A large positive $s$ means the full width supports a label that the capacity-limited widths reject. As a second use of the same model, we feed the narrow-width probabilities $p_w$ into the class-thresholding and counting steps of Confident Learning. There they replace the cross-validated, out-of-sample probabilities that the method normally requires.

Relation to Prior Work

Two papers motivate the premise that capacity limits resist memorization. Can network pruning benefit deep learning under label noise? finds that very high sparsity (above 95%) restores robustness to label noise, but intermediate sparsity hurts. Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noise reports a "final ascent", meaning test loss worsens again at large widths under noise. It also finds that lowering density helps more than lowering width. Both studies train separate static models on synthetic symmetric noise. We instead use capacity differences inside one shared model and evaluate against human-verified real errors. The second paper also constrains the design: if width reduction is a weak lever, narrow subnetworks may memorize nearly as much as wide ones. That is exactly the failure our decisive checks look for.

On the scaling side, Slimmable Neural Networks supplies the method. It shares weights across widths, keeps private batch-normalization statistics per width, and shows that the wide network implicitly teaches the narrow one; a 0.25× MobileNet v1 is 3.3% more accurate than a standalone 0.25× model. That implicit teaching is what we fear transfers memorized errors, and it is what our one-way variant removes. Bayesian Nested Neural Networks for Uncertainty Calibration and Adaptive Compression shows that narrow nested subnetworks are better calibrated and handles ambiguous radiologist masks on LIDC-IDRI. However, it never scores discrete label errors, and it needs Bayesian sampling. Switchable Representation Learning Framework with Self-compatibility estimates per-width uncertainty only to balance losses on clean re-identification data. We reuse its private-normalization setup but none of its gradient surgery.

On the noise side, Confident Learning: Estimating Uncertainty in Dataset Labels is the detector we aim to match more cheaply. It needs out-of-sample probabilities, and the file notes that producing them by k-fold cross-validation is prohibitive for large models. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks supplies the ground truth: human-verified errors in ten test sets, published through labelerrors.com and the cleanlab tooling. It also documents model reversal across separately trained architectures. Are we done with ImageNet? supplies the ReaL relabeling of ImageNet, and Re-TACRED: Addressing Shortcomings of the TACRED Dataset supplies corrected relation labels.

Two methods use disagreement between models, and they are the competitors whose cost we aim to cut. How does Disagreement Help Generalization against Label Corruption? (Co-teaching+) trains two peer networks that update each other on examples where they disagree. CrossSplit: Mitigating Label Noise Memorization through Data Splitting trains two networks on disjoint halves of the data. Both pay for two full networks. Neither asks whether capacity differences inside one network supply the needed disagreement.

Research Program

The program has three parts.

One-way width sharing. We build a slimmable network with private normalization per width. In the one-way variant, gradients from the width-$w$ loss reach only the channels that are new at width $w$. The narrowest block is therefore trained purely as a small model, and each wider width adds channels that can read, but not rewrite, narrower features. The variant costs no more than standard slimmable training. For convolutions with widths $\{0.25, 0.5, 0.75, 1\}$, the training cost is about $0.25^2 + 0.5^2 + 0.75^2 + 1 \approx 1.9$ full-model passes, since convolution cost scales with the square of width. Five-fold cross-validation costs about four full trainings.

Measuring memorization per width. On training sets with known error locations, we track for each width how quickly it comes to fit the wrong labels and how quickly it fits the correct ones, for both sharing schemes. This directly answers whether wide-to-narrow teaching transfers memorized errors.

Measuring reversal inside one model. On test sets with corrected labels, we evaluate every width of one model against the original labels and against the corrected ones. The question is whether the width ranking flips inside a single set of weights, as it does across separately trained architectures in the Pervasive Label Errors study.

A secondary arm repeats the analysis along depth. We train a transformer relation classifier with LayerDrop (Reducing Transformer Depth on Demand with Structured Dropout) and use shallow-versus-full disagreement as the score. This asks whether depth behaves like width, which the final-ascent study left open.

Baselines

Do nothing. The naive reference is the full-width model's own in-sample confidence in the given label, $1 - p_{1.0}(\tilde y \mid x)$. It measures how much the narrow widths add beyond an ordinary network that has already partly memorized the errors.

Strongest existing detector. This is Confident Learning run through cleanlab with 5-fold cross-validated probabilities from the same architecture at full width, using the same augmentation, epochs, and hyperparameter search budget as our model. The file reports no detection AUROC for this detector. It reports only that Confident Learning cut manual review by about 90% in the Pervasive Label Errors audit, and that on CIFAR-10 with 40% noise it beat seven robust methods, with no accuracy value given. We will therefore compute its numbers ourselves. Beating it means equal or higher precision among the top-$k$ flagged examples, where $k$ is the number of verified errors, and equal or higher area under the precision-recall curve. It also means doing so at no more than half its training compute. Matching it at half the compute also counts as a success.

Two standalone models of different widths. A standalone 0.25× model and a standalone 1.0× model are trained independently and scored with the same gap formula. This asks whether weight sharing adds anything beyond having a small model and a large model. Co-teaching+ and CrossSplit are the peer-network baselines for the downstream clean-and-retrain task. There, the file's reported numbers are CrossSplit's 52.40% on CIFAR-100 with 90% symmetric noise and 78.48% on mini-WebVision, and Co-teaching+'s 80.95% on CIFAR-10 with open-set noise from SVHN, against 56.44% for a standard network. Beating these means matching or exceeding the accuracy while training one model instead of two.

Key ablation. Standard slimmable training with full two-way sharing, at equal compute, isolates the one-way ingredient. If it detects errors as well as the one-way variant, our claim about error transfer through shared weights is wrong.

Why not just…?

Why not just run Confident Learning with cross-validation? It is the right accuracy target, and we may not beat it on accuracy. The claim is about cost: roughly 1.9 versus 4 full trainings on CIFAR-scale convolutional networks. If the narrow-width probabilities, fed into Confident Learning's thresholds, fall clearly below cross-validated probabilities in top-$k$ precision, the proposal reduces to "cheaper but worse". We will report that result as it stands.

Why not just train a small model and a large model separately? This is the most serious threat. The one-way variant is close to that pair, with the narrow model embedded inside the wide one. If two standalone models match the one-way score, the only remaining advantage is memory and deployment: one set of weights instead of two, and a detector that runs as a by-product of an elastic model someone already wants. We will state that plainly rather than claim a detection gain.

Why not just use early-training snapshots of one full model? Early in training, networks fit clean examples before they fit errors. A snapshot of the full model from an early epoch might therefore give the same signal at no extra cost. We include this as a baseline, choosing the epoch on a held-out slice with injected noise. We then test whether the narrow-width signal is complementary, meaning that combining the two scores beats either one alone.

Why not just clean the benchmark by hand? Re-TACRED re-annotated 106,264 sentences and could release its fixes only as patch files, because of licensing. ReaL relabeled only the validation set. Human review does not scale to training corpora, so a cheap ranking that decides which examples humans should look at is the practical need.

Decisive Experiments

The primary evaluation is on real, verified errors.

TACRED. We train a width-elastic SpanBERT-style encoder (facebookresearch/SpanBERT) on the original TACRED training labels. We score each example and treat a disagreement between the TACRED and Re-TACRED labels, on the sentences retained in both, as ground truth.

ImageNet, CIFAR-10, and CIFAR-100. We train slimmable ResNets and MobileNets on the training sets. We score the test sets against the labelerrors.com verified errors and, for ImageNet, against ReaL.

Synthetic and web noise. We add CIFAR-100 with symmetric and pair noise and mini-WebVision for the downstream task: remove the flagged examples, then retrain.

Metrics. We report top-$k$ precision, area under the precision-recall curve, training compute in full-model equivalents, and, downstream, test accuracy on clean labels.

The decisive checks are four. First, under two-way sharing, the narrow widths should fit wrong labels measurably faster than under one-way sharing, which would show that wide-to-narrow teaching transfers errors. Second, the one-way gap score should match cross-validated Confident Learning's top-$k$ precision at no more than half its compute. Third, it should beat two standalone models of different widths, or at least tie them. Fourth, on corrected test labels, the narrow widths of one slimmable model should overtake the full width, reproducing model reversal within a single set of weights.

If the narrow widths memorize nearly as much as the full width, the idea fails. The width-and-density study gives some reason to expect this, since it finds width a weak lever. In that case the fallback, reported as the result, is to replace width reduction with density reduction inside the elastic model.