Reading Clean Labels Out of Early Layers: Symmetric-Loss Readout Selection in Dense Networks

Neural networks trained on mislabeled data eventually memorize the wrong labels. Two research lines have attacked this from different ends. Noise-robust losses, which cap how hard any single example can pull on the model, are one end. Dense feature reuse is the other: an architecture in which every layer's output is concatenated and passed unchanged to all later layers, so the final classifier sees early and late features side by side. This note proposes a single mechanism that joins them. In a densely connected network, the final prediction is a sum of contributions from each layer's features. That makes it cheap to refit the classifier so that it reads mostly from the layers that stayed clean. A symmetric loss, one whose value summed over all possible labels is constant, gives a way to decide which layers those are using only noisy held-out data, because such losses rank models the same way under uniform label noise as under clean labels.

In two sentences: train a DenseNet with a noise-robust loss, then refit only its last linear layer with a penalty that lets it drop whole groups of channels coming from particular layers. Choose how much to drop by measuring a symmetric loss on a noisy held-out split. The question is whether this recovers accuracy that late-layer memorization destroyed, and whether that guarantee survives realistic web noise, where it is not supposed to hold.

Core Claim

The research file frames a tempting hypothesis: dense connectivity's short gradient paths might cure the underfitting of bounded losses such as mean absolute error (MAE) without the logit bias of Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning. We expect this to be false, and we treat it as a falsification target rather than the contribution. Staats et al. trace the underfitting to the output itself. With many classes, the starting predicted probability of the true class sits in a region where the bounded loss's gradient with respect to the logits is nearly zero. Every backbone, dense or residual, multiplies its backward signal by that same small factor. Dense connections fix gradient decay across depth, as Densely Connected Convolutional Networks shows, but they do not fix saturation at the softmax.

What dense reuse does offer is structural. If $x_l$ is the globally pooled output of layer $l$ and $W_l$ is the slice of the final linear layer that reads it, the logit vector is exactly

$$z(x) = \sum_{l=0}^{L} W_l\, x_l + b.$$

Each source contributes additively and is never overwritten by later layers. A residual network instead sums features into a shared stream, so early information cannot be separated afterward. Our claim is that under label noise, memorization concentrates in the late-layer slices $W_l$, while early and middle sources keep class information that is still mostly correct.

We therefore propose refitting $\{W_l\}$ on frozen features by minimizing the robust training loss plus a group penalty $\lambda \sum_l \omega_l \lVert W_l \rVert_F$, where the weight $\omega_l$ grows with depth. The strength $\lambda$ is chosen by MAE measured on a noisy held-out split. The symmetry result of Robust Loss Functions under Label Noise for Deep Neural Networks shows that, for uniform noise below a threshold, minimizing a symmetric loss on noisy labels gives the same solution as minimizing it on clean labels. In that regime, noisy-holdout MAE is a model-selection criterion that needs no clean labels. For image-dependent noise the guarantee does not hold, and testing how far it degrades is half the point.

Relation to Prior Work

The loss side supplies the training objectives we reuse: generalized cross-entropy, which interpolates between MAE and cross-entropy (Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels); symmetric cross-entropy, which pairs cross-entropy with a reversed term (Symmetric Cross Entropy for Robust Learning with Noisy Labels); and MAE with a logit bias (MAE*) from Staats et al. All three change the loss and leave the trained network's readout untouched. We add a post-hoc, label-free readout selection that exploits architecture.

Symmetric cross-entropy also motivates the work. Its finding that noisy training under-learns subtle "hard" classes suggests that fine-grained early features exist but are not used, which is the kind of signal a source-aware readout could recover.

Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels constrains the design in two ways. It shows that methods tuned on synthetic label flips often fail on real web noise. It also shows that under web noise networks lack the clean "early peak then collapse" curve, which weakens early stopping as a fix. It tested ResNet and Inception models, never DenseNet, and it did not isolate robust losses. Its Red Mini-ImageNet and Red Stanford Cars benchmarks are our main test of whether the symmetric-loss selector breaks under image-dependent noise.

Learning from massive noisy labeled data for image classification models image-dependent noise explicitly but uses clean labels to do it. We use none.

Residual dense blocks from super-resolution (Residual Dense Network for Image Super-Resolution) are a hybrid with only local concatenation. We include them as an intermediate architecture, since their outer residual path re-blends sources and should weaken separability.

Research Program

The work has three parts.

The first is a factorial training study. We cross architecture (DenseNet-BC, a residual network matched in parameters and FLOPs, and a residual-dense-block classifier), loss (cross-entropy, GCE, symmetric cross-entropy, MAE), and noise (symmetric and class-asymmetric flips at 20 to 80 percent, plus controlled web noise). This directly tests the falsification target: if DenseNet trained with plain MAE matches MAE without a logit bias at 100 classes, our output-saturation argument is wrong and we report it.

The second is a source-level memorization analysis. For each trained DenseNet, we fit a separate linear probe on each source $x_l$ and measure accuracy on the clean test set and on the training set's clean and corrupted subsets. Synthetic noise makes those subsets known. This tells us where in depth memorization of wrong labels begins.

The third is the readout refit and its label-free selector. We report how well noisy-holdout MAE predicts clean test accuracy across settings of $\lambda$, measured by rank correlation, separately for synthetic and web noise.

Baselines

The naive reference point is DenseNet-BC trained with cross-entropy and evaluated at the final epoch. It shows how much noise costs with no intervention.

The strongest loss-based competitor is MAE from Staats et al., since it is the most recent fix for bounded-loss underfitting at many classes. No reported result was found on Lacuna for MAE top-1 accuracy on CIFAR-100 or WebVision, so we rerun it ourselves on every backbone using their class-count rule for the bias.

Symmetric cross-entropy is the best-documented competitor in the file. Its paper reports 85.34% accuracy at 40% noise on CIFAR-10, against 78.70% for cross-entropy, and 71.02% on Clothing1M. Those numbers come from that paper's own backbones, so we treat them as anchors and compare only against our rerun on identical backbones.

On web noise, the strongest non-loss method is MentorMix. No reported per-noise-level numbers were found on Lacuna for Red Mini-ImageNet, so we rerun it.

On Clothing1M, the 78.24% of Xiao et al. uses clean labels and is reported as an upper reference, not a fair competitor.

All methods share data, augmentation, epochs, and a fixed per-method budget of hyperparameter trials. All trials are selected on the same noisy holdout, never on clean test labels.

The key ablation applies the identical group-penalized readout to the matched residual network, using pooled outputs from each of its blocks concatenated together. This isolates whether dense training produces cleaner early sources, or whether any multi-layer readout would do. Beating the field means a consistent gain over the same robust loss without refitting, on the same backbone, at 40% or more noise. It also means beating the residual-readout ablation by more than seed variance on at least CIFAR-100 and Red Mini-ImageNet.

Why not just…?

Why not just use MAE* on a ResNet? It may be enough, and the factorial study measures exactly that. Two observations leave room for our approach. Staats et al. note that the class-count rule can overestimate the bias and erode robustness. And our refit sits on top of any loss, MAE included. If refitting adds nothing on top of MAE, the proposal fails on its own terms.

Why not just probe intermediate ResNet layers? This is the most serious threat, and it is the main ablation. A residual stream still contains early information, only blended with later updates. If residual multi-layer readout matches dense readout, then the contribution reduces to label-free readout selection with a symmetric loss, independent of architecture. That would still be a useful finding, but a narrower one, and we would say so.

Why not just stop training early using a noisy validation set? Early stopping truncates in time, while our refit truncates in depth. The two can be combined, and we report both separately. Jiang et al. found that web noise lacks a sharp early-learning peak, which suggests time-based truncation is weaker exactly where our method is tested hardest.

Why not just use sample selection such as MentorMix or Co-teaching? These methods need curricula or paired networks, and Jiang et al. report that separating clean from noisy samples by their losses is harder under web noise. Our method is a single linear refit after training. MentorMix is run as a baseline, and combining the two is a reported variant, not a claim of superiority.

Decisive Checks

Three results decide the project. The first is whether plain MAE on DenseNet still underfits CIFAR-100 without a logit bias. We predict it does, which would settle the tempting hypothesis negatively. The second is whether per-source probes show corrupted-label fitting starting late in depth for DenseNet but not for the matched residual network. The third is whether the noisy-holdout MAE selector picks a near-oracle $\lambda$ under synthetic noise and degrades under red noise, with the degradation tracking the noise level. If the second check fails, the readout has nothing to recover and the project ends at the factorial study.