When people label images on crowdsourcing platforms, their mistakes are not random. They mislabel the images that genuinely look like another class, such as a cat photographed like a dog or a truck that resembles an automobile. Deep networks learn these human mistakes much faster than they learn random label flips, because the network can fit a wrong label using real visual features that the image shares with the wrong class. Nearly every method for learning from crowds intervenes after the features are computed. Some change the loss, some aggregate labels, and some add per-annotator confusion layers at the output. The feature extractor itself is left standard. This proposal tests whether the problem should also be fixed inside the feature extractor, using divisive normalization. Divisive normalization is a computation found in sensory neurons: each feature's response is divided by the pooled activity of similar features. The effect is to suppress redundant, shared evidence and amplify what is distinctive.
The specific claim is narrow and can be falsified. If we replace standard normalization in a convolutional network with divisive normalization whose suppression pools follow learned feature similarity, the network should take longer to memorize instance-dependent human noise. It should shrink the accuracy gap between human noise and synthetic noise of matched rate on CIFAR-10N and CIFAR-100N. It should also make loss-based clean/noisy sample selection more reliable, so it adds to existing robust training methods rather than replacing them. A general accuracy gain from a better architecture would not support the claim. The claim is that the gain should be selectively larger under human noise than under synthetic noise.
Why this connection, and what is already known
On the crowd side, Learning with Noisy Labels Revisited introduced CIFAR-N and showed that real human noise is instance-dependent. Networks memorize it faster than synthetic noise, robust methods lose 1–5% when moving from synthetic to human noise, and plain cross-entropy reaches only about 77% on CIFAR-10N "Worst" versus about 93% on clean labels. That paper motivates this proposal: it locates the failure in shared visual features that make wrong labels easy to fit.
Output-side crowd methods supply the regime we work in but not the mechanism. Deep Learning from Crowds adds per-worker confusion matrices on top of a standard backbone. ALCrowd makes embeddings invariant to worker identity. DivideMix splits data into clean and noisy subsets by fitting a two-component mixture to per-sample losses. All three treat the backbone as fixed. DivideMix in particular depends on noisy samples still having high loss when selection happens, and early memorization of human noise breaks that assumption. To Aggregate or Not? constrains our training protocol: with three annotators per image, training on each annotator's label separately beats majority aggregation. We therefore train on separated labels by default.
On the normalization side, Simoncelli et al. showed that dividing filter outputs by their neighbors' pooled amplitude removes higher-order dependencies between features. This is the property we hypothesize matters, since human noise is fit through features that co-activate across confusable classes. Two papers supply the method and its known risks. Divisive Feature Normalization in AlexNet inserted learnable divisive normalization after ReLUs and reported sparser activations, cleaner deep-layer representations, and robustness to input corruption (61.33% vs 59.58% ImageNet top-1 over batch normalization). It also reported that the layer often fails to converge without batch normalization, and its pools follow a hand-made one-dimensional channel ordering. Bio-inspired learnable divisive normalization reported a 1.8% ImageNet-100 gain with lateral excitatory and inhibitory connections. Both evaluated only clean labels and input noise; neither examined label noise. Normalizing the Normalizers shows that batch and layer normalization are special cases of divisive normalization with different pools, and that adding a learnable smoothing term plus an L1 sparsity penalty alone helps. That result is the main threat to our claim, because it suggests sparsity, not similarity-weighted suppression, could do the work. The research file found no paper applying divisive normalization to label noise of any kind.
Core Claim and Mechanism
We use the standard form, in which the response $y_i$ of channel $i$ at a spatial location is
$$y_i = \frac{x_i^\gamma}{\sigma^\gamma + \sum_j w_{ij}\, x_j^\gamma},$$
where $x_i \ge 0$ is the rectified input, $\gamma$ is a power (we test 1 and 2), $\sigma$ is a learnable semi-saturation constant, and $w_{ij} \ge 0$ sets how strongly channel $j$ suppresses channel $i$.
The ingredient we add is the source of $w_{ij}$. Prior work orders channels on a line. We instead parameterize $w_{ij}$ from the similarity of channels' filter weights or running activation correlations, kept non-negative and row-normalized, and updated during training. The hypothesis follows from the redundancy-reduction view. A wrong human label is fit easily when the features supporting the wrong class are the same features that co-activate on the true class. Suppression pooled over similar channels reduces the gain of exactly these jointly active features, so fitting the wrong label requires more instance-specific evidence, and that takes longer to accumulate. For stability we follow the AlexNet study and keep batch normalization before the divisive stage, with matching initialization.
Research Program
We first build divisive-normalization variants of a ResNet-18 for CIFAR-scale data: one with line-topology pools reproducing prior work, one with learned similarity pools, and one with pools restricted to spatial neighbors only. All variants are matched in parameter count to the batch-normalization baseline.
We then measure memorization directly. On CIFAR-10N "Worst" and on CIFAR-100N fine labels, we know which training labels disagree with the clean CIFAR labels. We track three quantities by epoch. The first is the fraction of mislabeled examples whose given wrong label the network predicts. The second is the epoch at which this fraction crosses 50%. The third is how well the training loss separates clean from mislabeled examples, measured as AUC.
The central control is synthetic noise with matched statistics. From CIFAR-N we estimate the class-to-class transition matrix and flip clean labels with that matrix at the same rate. This produces noise that is class-dependent but not instance-dependent. The quantity of interest is the human-minus-synthetic accuracy gap for each architecture. Our claim predicts that divisive normalization shrinks this gap, not just raises both numbers.
Finally, we analyze mechanism. For mislabeled images, we measure the cosine overlap between penultimate features and the wrong-class mean, with and without divisive normalization, and check whether lower overlap predicts which examples are memorized later.
Baselines
The naive reference point is a ResNet-18 with batch normalization trained with cross-entropy on separated CIFAR-N labels, following To Aggregate or Not?. We also run a majority-vote variant. Its anchor is the roughly 77% cross-entropy accuracy on CIFAR-10N Worst reported by Learning with Noisy Labels Revisited. Our reproduction must land near that figure before any comparison counts.
The strongest existing method is DivideMix, chosen because it is the most widely used sample-selection method and its weakness is precisely early memorization. The research file reports 74.76% on Clothing1M and 76.0% on CIFAR-10 with 90% symmetric noise. The file contains no reported DivideMix result on CIFAR-10N or CIFAR-100N, so we run it ourselves. We use its published schedule and the same backbone, epochs, augmentation, and a 20-trial hyperparameter search shared equally by every method. Beating it means one of two things. Either DivideMix with a divisive-normalization backbone exceeds DivideMix with batch normalization by more than the seed standard deviation over five seeds, or the divisive backbone with plain cross-entropy closes a meaningful fraction of the gap to DivideMix while yielding higher clean/noisy loss AUC at the warm-up cutoff.
For crowd-specific comparison on LabelMe, we use the Crowd Layer, which reported 83.15% versus 76.74% for majority vote. We run it with and without a divisive backbone.
Two ablations isolate the key ingredient. The first is BN* from Normalizing the Normalizers, which adds learnable smoothing and an L1 penalty with no cross-channel suppression; it tests whether sparsity alone explains any effect. The second is line-topology divisive normalization from the AlexNet study; it tests whether learned similarity pools matter. Both receive the same tuning budget, including a sweep of the L1 weight.
Why not just…?
Why not just use DivideMix or another robust loss and leave the backbone alone? These methods are strong, but they only see the loss, and the loss stops separating clean from noisy examples once human noise is memorized. That is why they still lose 1–5% under human noise according to Learning with Noisy Labels Revisited. We do not propose replacing them. The DivideMix-plus-divisive-backbone run tests whether the two combine. If divisive normalization adds nothing on top of DivideMix, the practical case for it collapses, and we will report that.
Why not just add sparsity or stronger regularization? This is the most serious alternative. Normalizing the Normalizers gets clean-label gains from smoothing plus L1 alone (51.56% vs 48.14% on CIFAR-100), and the AlexNet study links divisive normalization's benefits to sparsity. The BN* ablation with a tuned L1 weight, plus a weight-decay sweep, settles it. If these match learned-pool divisive normalization on the human-minus-synthetic gap, we conclude that sparsity is the mechanism and drop the claim about similarity-weighted suppression.
Why not just train on soft labels from the three annotators? Annotator vote distributions already encode which images are confusable, and that might prevent overconfident fitting of wrong labels more cheaply. We include soft-label training as a baseline with both normalization types. If divisive normalization helps only with hard labels and not with soft labels, its role is redundant with label-side information. That would be a real, publishable negative result, though it would weaken the case for the architecture.
Why not just early-stop? Early stopping exploits the delay between learning clean and noisy examples, but human noise shrinks that delay. Our memorization curves directly measure whether divisive normalization lengthens it. We report early-stopped accuracy, using a held-out noisy validation split, for every architecture so the comparison is fair.
Decisive Checks
The idea succeeds only if three things hold together. First, learned-pool divisive normalization reduces the CIFAR-10N Worst human-minus-synthetic gap relative to batch normalization by more than the seed standard deviation, while showing no comparable gain on matched synthetic noise. Second, it delays the 50% memorization epoch for mislabeled examples. Third, it beats both the BN*-sparsity and line-topology ablations on these two measures. Replication on CIFAR-100N, where fine-label noise is about 40%, and on LabelMe with the Crowd Layer tests whether the effect holds beyond one dataset.
If divisive normalization raises clean and noisy accuracy equally, the result is an architecture gain and says nothing about crowds. If the sparsity ablation matches it, the mechanism is sparsity rather than similarity-weighted suppression. Either outcome is informative, and the experiments are designed to tell them apart.
References
- Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
- Deep Learning from Crowds
- Adversarial Learning for Chinese NER From Crowd Annotations
- DivideMix: Learning with Noisy Labels as Semi-supervised Learning
- To Aggregate or Not? Learning with Separate Noisy Labels
- Modeling Surround Suppression in V1 Neurons with a Statistically Derived Normalization Model
- Divisive Feature Normalization Improves Image Recognition Performance in AlexNet
- Bio-inspired learnable divisive normalization for ANNs
- Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes