Bayesian neural networks average the predictions of many plausible weight settings. The idea is appealing, but in practice it runs into two walls. The first is storage and speed: keeping and evaluating dozens of dense networks is expensive. The second is the quality of the samples: standard samplers such as stochastic gradient Langevin dynamics (SGLD, which is gradient descent plus carefully scaled Gaussian noise so that the iterates wander over the posterior rather than converge to one point) have no preference for flat regions of the loss, and flat regions are the ones associated with small gaps between training and test error. Recent work fixes the second problem: flatness-aware SGLD evaluates the gradient at a randomly perturbed copy of the weights, which quietly adds a penalty on the Hessian trace, a measure of how sharply the loss curves around the current weights (Flatness-Aware Stochastic Gradient Langevin Dynamics). That method assumes a loss that is four times differentiable, however, and it cannot handle a sparsity prior such as an $L_1$ penalty. The authors list the non-smooth case as open.
We propose a sampler that produces weight samples that are both flat and exactly sparse. It does this by giving each source of non-smoothness its own tool. Gaussian perturbation smooths the network loss, which is only piecewise smooth because of ReLU units. A proximal step handles the sparsity prior. The proximal step is the "soft-threshold" operation from convex optimization, which sets small weights to exactly zero. The central claim is that this split matters. If the Gaussian perturbation is also applied to the prior, the kink in $|w|$ is rounded off: exact zeros disappear, and the sparsity penalty picks up its own bias. If the proximal step is used without flatness perturbation, the zeros are exact but there is no bias toward flat regions. Only the split sampler gets both.
Core Claim
Let $L(\theta)$ be the training loss of a network with weights $\theta$, let $g(\theta)=\lambda\|\theta\|_1$ (or a group-lasso penalty over channels) be the sparsity prior, and let $\beta$ be the inverse temperature. The update we propose is
$$\theta_{k+1}=\mathrm{prox}_{\gamma g}\Big(\theta_k-\gamma\,\widehat{\nabla L}(\theta_k+\sigma\varepsilon_k)+\sqrt{2\gamma/\beta}\,\xi_k\Big).$$
Here $\gamma$ is the step size, $\varepsilon_k,\xi_k$ are independent standard Gaussians, $\sigma$ is the perturbation scale, $\widehat{\nabla L}$ is a minibatch gradient, and $\mathrm{prox}_{\gamma g}$ is soft-thresholding. We conjecture that this chain targets
$$\pi_\sigma(\theta)\propto\exp\!\big(-\beta\,[\,\mathbb{E}_\varepsilon L(\theta+\sigma\varepsilon)+g(\theta)\,]\big),$$
up to a discretization bias. The first term expands as $L+\tfrac{\sigma^2}{2}\mathrm{Tr}\nabla^2 L$, which is where the flatness bias comes from. The Gaussian-smoothed loss is infinitely differentiable whenever $L$ is merely Lipschitz, with gradient Lipschitz constant of order $G/\sigma$. This removes the need for the four-times-differentiability assumption in fSGLD, so ReLU networks fall inside the theory rather than outside it. The prior stays unsmoothed, so every sample has exact zeros and the sparsity pattern varies across samples according to the posterior.
Relation to Prior Work
Three lines of work supply the ingredients, and none combines them. fSGLD supplies the flatness mechanism, including its coupling between $\sigma$ and $\beta$, and reports the lowest negative log-likelihood (NLL) on CIFAR-10/100 and strong label-noise results. It is restricted to smooth potentials with no prior structure (Flatness-Aware Stochastic Gradient Langevin Dynamics).
Proximal Langevin methods supply the handling of the prior. Moreau-envelope sampling replaces a kinked potential with a smooth surrogate (Efficient Bayesian Computation by Proximal Markov Chain Monte Carlo: When Langevin Meets Moreau). The stochastic proximal Langevin algorithm splits a smooth gradient step from proximal steps and gives rates, but only in the log-concave case (Stochastic Proximal Langevin Algorithm: Potential Splitting and Nonasymptotic Rates). PSGLA extends this splitting to non-convex smooth parts using a stability argument under weak dissipativity (From stability of Langevin diffusion to convergence of proximal MCMC for non-log-concave sampling). That analysis is the template we will adapt, with the randomized-smoothed loss as the drift. All of these target the plain Gibbs measure and never ask about curvature or generalization.
SAFE shows that sparse and flat minima generalize markedly better under label noise, outperforming standard pruning by up to 30% in accuracy (SAFE: Finding Sparse and Flat Minima to Improve Pruning). SAFE is a deterministic ADMM optimizer with an inner maximization in the style of sharpness-aware minimization (SAM), so it returns one network, costs two gradient evaluations per step, and gives no uncertainty estimate. SAFE motivates the proposal, since it provides evidence that flatness and sparsity together matter. fSGLD and PSGLA supply the method.
Two results constrain the design. First, Moreau smoothing shifts modes and rounds peaks (The forward-backward envelope for sampling with the overdamped Langevin algorithm). This is the reason we apply the exact proximal step to $g$ rather than a smoothed gradient of $g$. Second, the active-trace analysis shows Moreau-Yosida sampling stays efficient because trajectories rarely sit near the kinks (Active-Trace Complexity Bounds for Moreau--Yosida Unadjusted Langevin Sampling). That result is proven only when the smooth part is strongly convex. Whether it holds with a non-convex network loss is a question we measure directly rather than assume.
Research Program
The theoretical part adapts the PSGLA stability argument to the smoothed drift. The aim is a Wasserstein bound between the chain and $\pi_\sigma$ under Lipschitz $L$ and dissipativity, with $\sigma$ entering both the flatness bias and the drift's Lipschitz constant. This trade-off should explain how to choose $\sigma$ relative to $\gamma$.
The empirical part first verifies the target on low-dimensional problems where $\pi_\sigma$ can be computed by numerical integration. One such problem is a two-dimensional loss with one sharp and one wide basin of equal depth, times a Laplace prior. On this problem we check three things: that sample histograms match $\pi_\sigma$, that mass shifts toward the wide basin as $\sigma$ grows, and that the exact-zero mass is preserved. After that, the sampler is run on image classification.
We will also log a network analogue of the active trace: the fraction of weights within $\gamma\lambda$ of the soft-threshold boundary at each step. This tests whether the curvature-tube picture survives non-convexity. A second diagnostic, suggested by the "deceptive flatness" finding that low loss variation can coexist with erratic gradients (Beyond Deceptive Flatness: Dual-Order Solution for Strengthening Adversarial Transferability), is to record gradient variance under weight perturbation alongside the Hessian trace. This checks whether the samples are genuinely flat or only flat in loss value.
Baselines
The naive reference point is plain SGLD and ordinary SGD training of the dense network at the same epoch budget. It tells us how large the generalization gap is before any intervention.
The strongest dense competitor is fSGLD itself (Flatness-Aware Stochastic Gradient Langevin Dynamics). It shares our flatness mechanism and cost profile of one gradient per step, and it is reported to give the best NLL on CIFAR-10/100 and strong label-noise accuracy. Its summary on Lacuna gives no raw numbers, so no reported result was found on Lacuna for fSGLD accuracy or NLL, and we will rerun it with the authors' coupling of $\sigma$ and $\beta$. For the sparse regime we give fSGLD its best chance by magnitude-pruning each sample to the same sparsity our sampler reaches, then recalibrating batch-norm statistics.
The strongest sparse competitor is SAFE (SAFE: Finding Sparse and Flat Minima to Improve Pruning). Its reported numbers on Lacuna are a sharpness of 0.09 at its flat minimum and gains of up to 30% accuracy over standard pruning under label noise; no raw CIFAR accuracy was found on Lacuna. Because SAFE yields one network, we also build an ensemble of independently seeded SAFE runs matched to our number of samples. This baseline costs several times more compute than a single chain, and we will report that cost openly.
The ablations isolate the split. The first is proximal SGLD, which is PSGLA-style sampling with $\sigma=0$ and tests whether flatness matters. The second, "fully smoothed", applies the Gaussian perturbation to $L+g$ with a subgradient for $g$ and tests whether the exact proximal step matters. The third replaces the proximal step with the Moreau-envelope gradient of $g$, as in MYULA, and tests whether a smoothed prior suffices.
All methods get the same architectures (ResNet-18 and VGG-16), the same epochs, the same number of retained samples, and an equal hyperparameter grid of about twenty configurations. We count total gradient evaluations and report wall-clock time.
We will count the proposal as successful if one condition holds at matched sparsity of 90% and 98%. Either the sampler must beat pruned fSGLD on NLL and on the train-test gap under CIFAR-10N/100N label noise, or it must match the SAFE ensemble's accuracy at a fraction of its compute while also giving better out-of-distribution AUROC.
Why not just…?
Why not just run fSGLD and prune the samples afterwards? This is the most serious threat. Post-hoc pruning may land in weights that are nearly as good, since flat regions tolerate perturbation. The case against it is that pruning moves each sample off the distribution the chain was sampling, and the removed weights were never chosen with the prior in mind. SAFE's large gains over standard pruning suggest that sparsity and flatness chosen jointly beat sparsity imposed afterwards. The matched-sparsity comparison above settles the question. If pruned fSGLD ties us at 90% sparsity, we expect the gap to open only at 98%, and if it does not, the claim fails.
Why not just add a proximal step to SAM, or use SAFE? SAM-style methods double the cost of every step and return a point estimate. SAM's own mechanism also changes between penalizing the top eigenvalue in full-batch training and the trace in stochastic training (How Sharpness-Aware Minimization Minimizes Sharpness?). The seeded SAFE ensemble is the fair version of this alternative. If it wins on both NLL and cost-adjusted accuracy, sampling adds nothing here.
Why not just smooth everything, or use a smooth $L_1$ surrogate? Smoothing the prior yields no exact zeros, so it gives no storage savings. It also moves the modes, as the forward-backward envelope work documents for Moreau smoothing (The forward-backward envelope for sampling with the overdamped Langevin algorithm). The fully smoothed and Moreau ablations measure how much accuracy and calibration this costs at the same thresholded sparsity.
Why not just use SWAG or high-learning-rate checkpoint collection? SWAG is used in Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examples and checkpoint collection in LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinity. Both explore wide regions cheaply, but neither has a defined target distribution or any way to enforce exact sparsity. We include SWAG as a secondary dense baseline for NLL and out-of-distribution detection.
Proposed Experiments
The testbeds follow fSGLD and SAFE. For generalization and calibration we use CIFAR-10 and CIFAR-100, reporting accuracy, NLL, and expected calibration error. For label noise we use CIFAR-10N, CIFAR-100N, and a WebVision subset, reporting the train-test gap. For out-of-distribution detection we train on CIFAR and test on SVHN, reporting AUROC and AUPR. On every run we measure the Hessian trace and top eigenvalue of retained samples, the per-sample sparsity, the overlap of sparsity patterns across samples, and ensemble inference FLOPs.
Three checks are decisive. The first is the synthetic test, which confirms the chain targets $\pi_\sigma$ and not the unsmoothed Gibbs measure. The second is the matched-sparsity comparison against pruned fSGLD under label noise, which shows whether the split is worth having. The third is the near-kink fraction over training, which should stay small and shrink with $\gamma$ if the curvature-tube behavior carries over to non-convex losses. If that fraction instead stays large, the step size will need to shrink with $\lambda$, and the method loses its cost advantage. That would be a useful negative result for proximal sampling in deep networks.
References
- Flatness-Aware Stochastic Gradient Langevin Dynamics
- Efficient Bayesian computation by proximal Markov chain Monte Carlo: when Langevin meets Moreau
- Stochastic Proximal Langevin Algorithm: Potential Splitting and Nonasymptotic Rates
- From stability of Langevin diffusion to convergence of proximal MCMC for non-log-concave sampling
- SAFE: Finding Sparse and Flat Minima to Improve Pruning
- The forward-backward envelope for sampling with the overdamped Langevin algorithm
- Active-Trace Complexity Bounds for Moreau--Yosida Unadjusted Langevin Sampling
- Beyond deceptive flatness: Dual-order solution for strengthening adversarial transferability
- How Sharpness-Aware Minimization Minimizes Sharpness?
- Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examples
- LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinity