When some training labels are wrong, a classifier trained the usual way slowly memorizes the errors. One classical defense is a symmetric loss: a loss whose values, summed over all possible labels for an example, give the same constant regardless of the model's prediction. Mean absolute error on softmax outputs is the standard example. Under uniformly random label flips, such a loss sees noise only as a constant offset plus a rescaling of the clean objective. Nobody has asked what this does to a Bayesian posterior over network weights. The answer is short. If we turn a symmetric loss into a likelihood, the posterior from noisy labels is, in expectation, exactly the posterior from clean labels at a higher temperature. Label noise does not change the shape of the posterior. It only flattens it by a known factor.
This proposal builds that sampler and tests what the identity buys. It targets a gap in the symmetric-loss literature known as probability blindness: these losses find the right decision boundary but say nothing reliable about confidence. The proposal asks whether averaging predictions over posterior samples, followed by a single scalar temperature correction, gives calibrated, noise-robust confidence. The useful symmetric losses are non-smooth, so the sampler is a proximal Langevin method. Ordinary Langevin sampling moves by gradient steps plus Gaussian noise. The proximal version replaces the gradient step on the non-smooth part with a small exact optimization step, which avoids the oscillation that plain gradient samplers show at kinks and walls.
Core Claim
Take a symmetric loss $L(k, f(x))$ with $\sum_{k=1}^K L(k, f(x)) = C$, where $f$ is the network with weights $\theta$. Define the tempered posterior
$$\pi_\lambda(\theta \mid y_{1:n}) \propto p(\theta)\exp\Big(-\lambda \sum_{i=1}^n L(y_i, f_\theta(x_i))\Big),$$
where $p(\theta)$ is the prior and $\lambda$ is an inverse temperature. Suppose each label is kept with probability $1-\eta$ and otherwise replaced uniformly by one of the other $K-1$ classes. Then the expected noisy loss is $(1 - \tfrac{K\eta}{K-1})L(y, f) + \text{const}$. This follows in one line from the symmetry condition, the same algebra On Symmetric Losses for Learning from Corrupted Labels and Robust Loss Functions under Label Noise for Deep Neural Networks use for point estimates. It follows that the noisy-label posterior at temperature $\lambda$ matches the clean-label posterior at $\lambda(1 - \tfrac{K\eta}{K-1})$, up to a finite-sample fluctuation term of order $\sqrt{n}$ that does depend on $\theta$.
The claim is stronger than "the minimizer is unchanged." The whole posterior is unchanged up to temperature, so a single scalar recovers clean-label uncertainty. A cross-entropy posterior has no such property: noisy labels distort its shape and pull samples toward weights that memorize the errors. The open empirical questions are three. How large is the fluctuation term at realistic $n$? Does the identity degrade gracefully under class-conditional and human noise, where it does not hold exactly? Does averaging over posterior samples actually repair probability blindness?
Why the Sampler Must Be Proximal
A symmetric loss that is non-negative cannot be convex. On Symmetric Losses for Learning from Corrupted Labels handled this with the barrier hinge loss. The barrier hinge is linear inside a margin band $[-r, r]$, where it equals $r - z$ and is exactly symmetric. Outside the band it has steep walls of slope $b$. For Bayesian use, this structure matters for a specific reason. The linear "unhinged" part alone gives a potential that is unbounded below, so the posterior is improper. The walls are what make it proper. As $b \to \infty$, the barrier hinge posterior becomes the linear-loss posterior restricted to the set where every margin lies in $[-r, r]$. A hard constraint like this is exactly the non-smooth case proximal samplers were built for, and its proximal step is a projection.
The method comes from three papers.
- Proximal MCMC / MYULA, Durmus et al. replaces a non-smooth potential with its Moreau–Yoshida envelope, a smoothed version with a closed-form gradient. It shows that this samples a slightly biased surrogate of the target far faster than proximal MALA, a Metropolis-corrected proximal Langevin method.
- Stochastic Proximal Langevin Algorithm: Potential Splitting and Nonasymptotic Rates takes forward stochastic gradient steps on the smooth part and stochastic proximal steps on many non-smooth terms. It gives KL-divergence rates for convex potentials.
- From stability of Langevin diffusion to convergence of proximal MCMC for non-log-concave sampling extends proximal stochastic gradient Langevin (PSGLA) to non-convex potentials under strong convexity at infinity.
All three were developed for imaging and sparse-regression priors. None has been applied to a loss that encodes noise robustness.
The two pieces connect in a way the earlier papers could not exploit. The Moreau–Yoshida smoothing bias, a known open issue in Durmus et al., has a concrete meaning here. Smoothing the barrier hinge only changes the loss within a distance of order $\gamma b$ from the walls, where $\gamma$ is the smoothing parameter. The band where symmetry holds exactly therefore shrinks by a controllable, computable amount. The sampler's bias knob becomes a symmetry-violation knob, and we can measure how much noise robustness each unit of smoothing costs.
Research Program
The work proceeds in two regimes chosen so that the theory applies cleanly in the first and is stress-tested in the second.
Regime 1: last-layer posterior on fixed features. The backbone is frozen. To avoid confounding by noisy labels, it is trained self-supervised, or with the same symmetric loss by SGD. The posterior is over the final linear layer. Here the multiclass linear symmetric loss $L(k, f) = -f_k + \tfrac{1}{K}\sum_j f_j$ is linear in the weights, so the posterior is log-concave given a Gaussian prior and the margin box constraint. Each per-example box constraint has a closed-form projection along that example's feature direction.
We implement two samplers. The first is SPLA-style, applying stochastic proximal projections to a random minibatch of constraints. This sidesteps the sequential-projection bottleneck noted in Salim et al.. The second is MYULA on the smoothed barrier. Both come with convergence guarantees from that paper. In this regime we verify the temperature identity directly. We sample from noisy labels at $\lambda$ and from clean labels at $\lambda(1 - \tfrac{K\eta}{K-1})$, then compare the two posterior predictive distributions on held-out data.
Regime 2: full network. We run PSGLA. It takes forward Langevin steps on the whole network using the smooth linear part of the loss, and applies the barrier proximal step in output-margin space, linearized through the last layer. The theory in From stability of Langevin diffusion… covers this case only loosely, because its operator assumptions are hard to verify for deep networks. This regime is empirical.
Prediction. A test example's confidence is the posterior vote fraction or the average clipped margin. Temperature is selected on a small clean validation set, or set from a known or estimated $\eta$ using the identity.
Noise beyond the uniform case. For class-conditional pair flips and CIFAR-N human noise the identity does not hold exactly. We predict a measurable change in posterior shape. Its size, rather than accuracy alone, is the diagnostic we report.
Baselines
Naive reference. The do-nothing point of comparison is cross-entropy trained by SGD. Its noisy CIFAR-10 accuracy is reported at 78.70% at 40% symmetric noise in Symmetric Cross Entropy for Robust Learning with Noisy Labels. Learning with Noisy Labels Revisited reports about 77% on CIFAR-10N "worst" labels.
Cross-entropy Langevin sampler. The matching Bayesian reference is cross-entropy posterior sampling with the symmetric minibatch splitting sampler of Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamics. That paper reports 89.79% accuracy and good calibration on clean CIFAR-10. It is the right test of whether the symmetric loss, rather than sampling itself, is what helps under noise.
Point-estimate symmetric and hybrid losses. - Barrier hinge and MAE trained by SGD (Charoenphakdee et al., Ghosh et al.). These isolate the value of sampling over optimizing the same objective. On twonorm with 45% corruption, the barrier hinge is reported at about 96% against about 80% for logistic loss. - Symmetric Cross Entropy is the strongest single-network loss baseline in the file, at 85.34% on CIFAR-10 with 40% noise and 71.02% on Clothing1M.
Multi-network accuracy ceiling. DivideMix (DivideMix, 76.0% on CIFAR-10 at 90% noise) and Co-teaching (Co-teaching, 72.62% at 45% pair noise) set the accuracy ceiling. We expect not to beat DivideMix on accuracy and do not claim to.
Ablations. - Subgradient Langevin on the barrier hinge, to isolate the proximal step. - Smooth bounded symmetric loss under plain stochastic gradient Langevin, to isolate the barrier structure. - MYULA across a range of $\gamma$, to trace the symmetry-violation trade-off.
Fairness and success criteria. All methods share the same architecture, data splits, and epoch-equivalent compute. Samplers are charged for every gradient and projection evaluation. Each method gets an equal hyperparameter search budget on the same clean validation split. "Beating" a baseline means one of two things:
- Matching Symmetric Cross Entropy accuracy within one point while cutting clean-test expected calibration error (ECE) and negative log-likelihood (NLL) clearly below both Symmetric Cross Entropy and the cross-entropy sampler.
- Beating the cross-entropy sampler on accuracy at 40% or more symmetric noise.
The file reports no calibration numbers under label noise for any of these methods; I found no reported result on Lacuna for ECE or NLL under noisy labels, so we will produce those baselines ourselves.
Why not just…?
Why not just use SAM or an explicit Jacobian penalty? Why is SAM Robust to Label Noise? shows that sharpness-aware minimization slows memorization through early-phase regularization of the network's Jacobian. A plain $L_2$ penalty on activations and weights lifts SGD from about 52% to 60.8% peak accuracy at 30% noise, against 69.47% for 1-SAM. These methods still give a single point estimate, rely on early stopping, and produce no posterior. They compete on accuracy but not on the calibration and mislabel-detection claims. We include SAM as a baseline and also evaluate it at the final epoch, because the temperature identity predicts that our sampler does not need early stopping.
Why not just run DivideMix or Co-teaching? They are more accurate in the file's numbers and probably will remain so. They need two networks, noise-rate-dependent drop schedules, and mixture-model thresholds, and they lose 1–5% on human noise in CIFAR-N (Wei et al.). Our claim is narrower: one network that yields calibrated uncertainty with one interpretable scalar. We will report the accuracy gap honestly.
Why not just use Symmetric Cross Entropy and then temperature-scale it? This is a real threat to the calibration claim. Post-hoc temperature scaling on a clean validation set may fix much of a point estimate's miscalibration. Our argument is that symmetric-loss point estimates push margins toward the band edges $\pm r$, so a single rescaling has little per-example signal to work with. Posterior vote fractions vary across examples. The test is direct: give Symmetric Cross Entropy, barrier hinge SGD, and our sampler the same clean validation set and the same one-parameter temperature fit, then compare ECE, NLL, and selective-prediction accuracy. If temperature-scaled Symmetric Cross Entropy matches us, the sampling claim fails.
Why not just sample a smooth bounded symmetric loss with plain Langevin and skip the proximal machinery? This is the most serious alternative. A bounded output such as $r\tanh(g)$ with the linear loss is smooth, symmetric, and gives a proper posterior. Its gradients vanish at saturation, which is the same mechanism behind MAE's underfitting (Ghosh et al., Wang et al.). The barrier keeps a constant gradient inside the band and acts only at the walls. We run this alternative as an ablation under matched compute on CIFAR-10 and CIFAR-100. If it matches our accuracy and calibration, we would drop the proximal component and keep the temperature identity as the contribution.
Why not just use subgradient Langevin on the barrier hinge? Salim et al. show that subgradient Langevin oscillates and misses the sharp mode even on a one-dimensional Laplace potential. PnP-PSGLA needs 10,000 iterations where unadjusted Langevin needs 1,000,000 for a 30.81 versus 27.89 PSNR result in imaging. Steep barrier walls of slope $b$ should be worse still. We confirm this with the ablation by tracking margin-constraint violations and effective sample size per gradient evaluation.
Decisive Experiments
Identity check. We run Regime 1 on twonorm, MNIST, and CIFAR-10 features at symmetric noise rates from 0 to 80%. If the predictive distributions from noisy and temperature-shifted clean posteriors diverge by more than the gap between two independent clean chains, the core claim is false. As a direct test of Robust Classification with Noisy Labels Based on Posterior Maximization, we apply the same temperature-shift test to a cross-entropy posterior. That paper argues symmetry is unnecessary for robustness, but only for point estimates. Our prediction is that cross-entropy fails the test, which would show that symmetry matters at the posterior level even if not at the minimizer.
End-to-end comparison. We run Regime 2 on CIFAR-10 (symmetric 20/40/80%, pair 45%), CIFAR-10N (aggregate, random, worst), and CIFAR-100N. We report:
- clean-test accuracy, ECE, and NLL;
- AUROC for flagging mislabeled training examples by posterior disagreement with the given label;
- compute per method.
Failure-mode measurement. On pair-flip and human noise, the size of the posterior shape distortion should predict the accuracy loss relative to the uniform-noise case. If it does not, the identity is the wrong lens for realistic noise, and that negative result is worth stating plainly.
References
- On Symmetric Losses for Learning from Corrupted Labels
- Robust Loss Functions under Label Noise for Deep Neural Networks
- Efficient Bayesian computation by proximal Markov chain Monte Carlo: when Langevin meets Moreau
- Stochastic Proximal Langevin Algorithm: Potential Splitting and Nonasymptotic Rates
- From stability of Langevin diffusion to convergence of proximal MCMC for non-log-concave sampling
- Symmetric Cross Entropy for Robust Learning With Noisy Labels
- Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
- Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamics
- DivideMix: Learning with Noisy Labels as Semi-supervised Learning
- Co-teaching: Robust training of deep neural networks with extremely noisy labels
- Why is SAM Robust to Label Noise?
- Robust Classification with Noisy Labels Based on Posterior Maximization