Sharpness-Aware Private Training Through the Normalization Layers That Privacy Allows

Private deep learning that starts from random weights, with no public data to pre-train on, trains badly. Two facts in the literature point to a specific reason and a specific fix that nobody has tried. First, the standard private training algorithm (DP-SGD, which clips each example's gradient and adds Gaussian noise so that no single record can move the model much) makes slower progress the sharper the loss surface is. Second, the cheapest known way to make training seek flatter regions works through the normalization layers, but it works through BatchNorm. BatchNorm computes statistics across all examples in a batch, so it leaks information between examples and is banned from private training. We propose to apply sharpness-aware updates only to the per-example normalization layers that privacy permits (GroupNorm, LayerNorm, KernelNorm). The perturbation direction is taken from the previous step's already-privatized gradient, so the method costs no extra privacy and almost no extra compute.

The question is whether this lets a private model escape the sharp early region where noise stalls it, the point where it settles prematurely at a poor solution. It is also whether this can be done without public data and without bringing back the memorization that flatness-seeking is known to cause. Cross-sample leakage is the constraint that removed BatchNorm and with it the known lever. Premature convergence is the failure we are trying to fix.

Core Claim

Pre-training Differentially Private Models with Limited Public Data analyzes per-step loss improvement under DP. It finds a slowdown term, $\sigma^2 \mathrm{tr}(H)/(Bc^2)$, where $\sigma$ is the noise multiplier, $\mathrm{tr}(H)$ is the Hessian trace (a measure of sharpness), $B$ is the batch size and $c$ is the clipping threshold. The authors escape the sharp early region by pre-training without privacy on public data. Separately, Perturbing BatchNorm and Only BatchNorm Benefits Sharpness-Aware Minimization shows that sharpness-aware minimization (SAM) works better when it perturbs only BatchNorm's scale and shift parameters than when it perturbs all weights. SAM is an optimizer that takes the gradient at a slightly worsened nearby point.

Our claim is that this "normalization-only" effect carries over to sample-independent normalization. If it does, it lowers $\mathrm{tr}(H)$ early enough in private training to shrink the slowdown term. The result would be measurably better accuracy at a fixed privacy budget. The gain should be largest exactly where the formula predicts, at small $\varepsilon$ (large $\sigma$) and small batches.

Neither ingredient alone answers the question. Mueller and Hein never test non-BatchNorm layers or privacy, and they offer no theory for why the effect exists. Bu et al. never touch normalization or the optimizer's geometry, and they need public data. That makes this a test of a transfer, and that test could fail.

Research Program

The update is as follows. Let $\phi$ denote the normalization scale and shift parameters and $\theta$ all other weights. Let $\tilde g_{t-1}^{\phi}$ be the $\phi$-block of the noisy, clipped gradient released at step $t-1$. At step $t$ we compute per-example gradients at the perturbed point:

$$\phi_t' = \phi_t + \rho\, \tilde g_{t-1}^{\phi} / \|\tilde g_{t-1}^{\phi}\|_2 .$$

We clip and noise these gradients exactly as in DP-SGD and apply the update to the unperturbed $(\theta_t, \phi_t)$. Here $\rho$ is the perturbation radius. The perturbation is a function only of already-released noisy quantities, so privacy accounting is identical to plain DP-SGD by post-processing. The per-example gradient pass is the same single pass that Opacus already computes. The scale and shift parameters are a tiny fraction of the model, so the overhead is negligible.

We will build this for GroupNorm, LayerNorm and KernelNorm networks. KernelNorm comes from Kernel Normalized Convolutional Networks, which normalizes each example within the convolution window and is the strongest batch-independent normalization in the file under DP.

The analysis tracks three things over training. The first is $\mathrm{tr}(H)$, estimated with random-vector Hessian-trace probes on training data, for diagnosis only and never released. The second is the realized slowdown ratio. The third is accuracy, so we can see whether improvements coincide with early curvature reduction, as the mechanism requires.

Because flatness-seeking has a known privacy cost, we also measure empirical membership leakage at fixed $\varepsilon$. On Memorization and Privacy risks of Sharpness Aware Minimization and Membership Privacy Risks of Sharpness Aware Minimization show this cost without privacy: SAM memorizes atypical examples and shrinks prediction variance on non-members. Their theory covers only the interpolating linear regime. DP caps worst-case leakage regardless, but whether the empirical gap they report survives under noise is open. The answer constrains how this method can be marketed.

Baselines

The naive reference point is DP-SGD on the same GroupNorm network with a cosine learning-rate schedule. This is what a practitioner gets by swapping BatchNorm out to satisfy the Opacus validator. It sets the floor.

The strongest existing method is a KernelNorm ResNet-18 trained with DP-SGD as in Kernel Normalized Convolutional Networks. That paper reports 22.01% accuracy on ImageNet32x32 under DP, beating LayerNorm and GroupNorm. The file does not state the $\varepsilon$ for that number, so we will first reproduce it under the paper's protocol before comparing. For supervised DP training from scratch on CIFAR-10 and CIFAR-100, no reported result was found on Lacuna. We will establish those numbers ourselves with the same tuning budget for every arm: an equal grid over learning rate, clipping threshold and batch size, and for our method a small grid over $\rho$. All arms use identical architectures, epochs, privacy accountant and seeds, and report means over at least three seeds.

Beating the strongest baseline means a gain at matched $\varepsilon$ and matched wall-clock that exceeds the seed spread. The gain must also grow as $\varepsilon$ shrinks. A constant offset would suggest a tuning artifact rather than the slowdown mechanism.

Three ablations isolate the key ingredient. The first is full-parameter SAM with the same stale privatized direction, which asks whether restricting the perturbation to the normalization parameters matters. The second is normalization-only SAM with a fresh ascent direction from an extra privatized gradient that spends part of the budget, which asks whether staleness costs anything. The third is a random perturbation of the same radius, which asks whether the gradient direction matters at all or noise alone suffices.

An oracle arm, public ImageNet warm-start following Bu et al., is reported for context only. Under our premise it is not available.

Why not just…?

Why not just use a bigger batch? The slowdown term has $B$ in the denominator, so larger batches are the cheapest way to shrink it. This is a real threat. However, at fixed $\varepsilon$ and epochs, larger batches mean fewer steps and require more noise per step through accounting. Every arm searches over batch size, so our method only wins if it helps at each arm's best batch size. We will also report the gain as a function of batch size. If the gain disappears at large batches, the honest conclusion is that the method only helps in compute-limited regimes.

Why not just average the weights (an exponential moving average)? Weight averaging is a cheap, widely used flatness proxy in private training and costs no privacy. It is the alternative most likely to match us. We include it both as a competitor and combined with our method. If averaging alone matches our accuracy and curvature curves, the specific normalization-layer mechanism is unnecessary and we would say so.

Why not just run standard SAM on all weights, or simply swap in GroupNorm? Swapping in GroupNorm is our naive baseline. Bu et al.'s analysis predicts it still suffers the curvature slowdown, and KernelNorm's DP results show GroupNorm lags. Full-parameter SAM is our first ablation. Mueller and Hein found that full perturbation underperforms normalization-only perturbation with BatchNorm, but this is unverified for other normalizations. The experiment settles it directly.

Why not just warm-start on public data? The target setting has none: rare clinical cohorts and proprietary records. The oracle arm quantifies how much of the public-data gap we close.

Proposed Experiments

We train from scratch on CIFAR-10, CIFAR-100 and ImageNet32x32 at $\varepsilon \in \{1, 3, 8\}$ with $\delta = 10^{-5}$, using GroupNorm and KernelNorm ResNet-18 backbones. We report test accuracy alongside the curvature and slowdown-ratio curves.

The first decisive check is mechanistic. Our method must lower $\mathrm{tr}(H)$ within the first portion of training relative to DP-SGD. Its accuracy advantage must widen as $\varepsilon$ falls. If accuracy improves without a curvature drop, the explanation is wrong even if the method works.

The second check is transfer. Normalization-only must beat full-parameter perturbation for at least one batch-independent normalization. If it does not, Mueller and Hein's effect is specific to BatchNorm, which is itself a publishable negative result on why their effect exists.

The third check is privacy cost. Following Kim et al.'s protocol, we run LiRA (a likelihood-ratio membership inference attack using shadow models) and report true-positive rate at 0.1% false-positive rate on CIFAR-10 at matched $\varepsilon$, against plain DP-SGD. For reference, their non-private numbers are 12.5% for SAM versus 8.8% for SGD. A widened gap under DP would show that flatness still buys accuracy partly through memorization even within the formal bound. That would be an important caveat on Bu et al.'s prescription of flat basins as well. A closed gap would show that noise removes the variance-shrinkage channel they identified.