Private deep learning with DP-SGD (differentially private stochastic gradient descent, which clips each example's gradient to a maximum length $C$ and adds Gaussian noise before averaging) forces a choice. A small $C$ distorts the gradient, and a large $C$ drowns it in noise. Robust statistics faced a similar choice between robustness and efficiency: estimators that resisted outliers wasted information on clean data. That field partly resolved it with a simple classical device, the one-step estimator. It starts from a reasonable pilot guess and applies a single bounded-influence correction around that guess. The resulting estimator is robust, and it is nearly as efficient as the sample mean when the data are well behaved.
Our observation is that DP-SGD's clipped mean is exactly a one-step Huber-type estimator. Its pilot guess is fixed at zero, which is about the worst pilot available. We propose to re-center it on the optimizer's momentum. The momentum is already computed from past privatized releases, so using it costs no privacy budget and no extra memory. Once centered, the clipping threshold only has to cover how much per-example gradients vary around the current update. It no longer has to cover the gradient's full length. At equal privacy, this should reduce both clipping bias and the injected noise. The size of the gain should be predictable in advance from a measurable signal-to-spread ratio of per-example gradients.
Core Claim
Anchored clipping replaces the DP-SGD aggregate with
$$\hat g_t = m_{t-1} + \frac{1}{B}\Big(\sum_{i \in B_t} \mathrm{clip}_{C_t}\big(g_i - m_{t-1}\big) + \mathcal N(0, \sigma^2 C_t^2 I_d)\Big),$$
where $g_i$ is example $i$'s gradient, $B_t$ is the Poisson-subsampled batch of expected size $B$, and $m_{t-1}$ is the momentum built only from previously released $\hat g$'s. The term $\mathrm{clip}_{C}(v)=v\min(1, C/\|v\|_2)$ is the clipping operation, and $\sigma$ is the usual noise multiplier. The anchor is post-processing of earlier private outputs, so the sensitivity of the summed term is still $C_t$. Privacy accounting is therefore identical to DP-SGD with the same $(\sigma, C_t, B)$.
The substantive claim concerns the ratio of spread to signal. Define $\rho_t = \|\nabla f(w_t)\|^2 / \mathbb E\|g_i - \nabla f(w_t)\|^2$, where $f$ is the training loss and $w_t$ the parameters at step $t$. Zero-centered clipping must choose $C$ relative to $\mathbb E\|g_i\|$, and $\mathbb E\|g_i\|^2 = \mathbb{E}\|g_i-\nabla f\|^2(1+\rho_t)$. The anchored version only needs $C$ relative to the spread term plus the staleness error $\|m_{t-1}-\nabla f(w_t)\|$. We predict the accuracy gain at fixed $\varepsilon$ grows with $\rho_t$ and with the lag-adjusted quality of the anchor. When $\rho_t \ll 1$ we predict no gain.
This is a falsifiable prediction rather than a promise. In large models trained from scratch, per-example gradients are nearly orthogonal and $\rho_t$ may be tiny. In low-dimensional fine-tuning, early training, and regression under heavy-tailed noise, it may not be.
The threshold schedule borrows from Adaptive Huber Regression. That paper shows a Huber threshold that grows with effective sample size at a moment-dependent rate removes truncation bias. For heavy-tailed data with $(1+\delta)$ moments it attains minimax rates $\delta/(1+\delta)$. Here the effective sample size is set by the noise budget, and $C_t$ trades the residual-tail bias against the noise term $C_t\sqrt d/(B)$ times $\sigma$. We estimate the residual spread at zero budget cost by packing slack coordinates, following SlaClip. The difference is that we apply the slack trick to residual norms $\|g_i - m_{t-1}\|$ rather than raw norms.
Relation to Prior Work
Two papers motivate this proposal by showing that Huber-style soft truncation around an estimate beats hard clipping around a fixed point. A Huber Loss Minimization Approach to Mean Estimation under User-level Differential Privacy iterates a Weiszfeld-type solver to a Huber location for a static mean, using smooth sensitivity. It explicitly leaves the DP-SGD case open. Perturbed M-Estimation links bounded influence to bounded sensitivity through a smooth loss, but only under convex objective perturbation. We cannot iterate to a fixed point inside each step, because each data access costs budget. Instead we take the classical one-step route: a single correction per step, with the optimizer's own trajectory serving as the pilot. This mirrors the two-stage "coarse anchor, then refine" design of All-Purpose Mean Estimation over R, which attains Gaussian-optimal constants while remaining robust.
One paper constrains the claim. Revisiting Gradient Clipping proves clipped SGD has an irreducible floor $\Omega(\min\{\sigma, \sigma^2/c\})$ under asymmetric or heavy-tailed noise, including under DP noise. Centering does not repeal this floor, because asymmetric residual tails still bias a clipped mean. What centering changes is which quantity the floor is measured against. The error from clipping away the mean gradient itself disappears. The remaining error depends only on the tails of the residuals, and the growing $C_t$ schedule targets that residual bias. We will state the result this way and test it on the paper's $\chi^2$-noise quadratics.
The closest competitor in effect is DiceSGD. It removes the constant clipping bias by storing and re-injecting a clipped error buffer. That costs roughly triple the optimizer memory and a slightly larger noise multiplier, and its analysis assumes sub-Gaussian gradients. Anchored clipping compensates through the momentum state that DP-SGD with momentum or Adam already stores. It adds no buffer and keeps the standard noise multiplier.
Adaptive-threshold methods, including DC-SGD, quantile clipping and AdaptDP-FedAvg, tune the size of $C$ but always clip around zero. They are complementary to our change of center and are ablated below.
Research Program
The first step is diagnostic and precedes any method claim. We will log $\rho_t$ and the anchor error $\|m_{t-1} - \nabla f(w_t)\|/\mathbb{E}\|g_i-\nabla f\|$ along non-private training trajectories for every benchmark below. Private runs cannot expose these quantities without spending budget, so we measure them non-privately purely for analysis. If $\rho_t$ is negligible everywhere, the proposal narrows to heavy-tailed low-dimensional problems, and we will report that.
The second step is analysis in the smooth, possibly nonconvex setting. We will bound the bias of $\hat g_t$ by two terms. One is a staleness term in $\|m_{t-1}-\nabla f(w_t)\|$, controlled by the momentum parameter and step size. The other is a residual-tail term of order $\mathbb E[(\|g_i-\nabla f\|-C_t)_+]$, which under finite $p$-th residual moment scales like $\sigma_r^p/C_t^{p-1}$. Balancing this against the noise term yields the $C_t$ schedule. We will prove convergence to stationarity without sub-Gaussian assumptions and compare the resulting rate with the $\tilde O(T^{-1/3})$ of quantile clipping.
The third step is the implementation. It is a few lines on top of Opacus-style per-example gradients: subtract the anchor before clipping, add it back after noising, and append slack coordinates for the residual-norm histogram.
Baselines
The naive reference is standard DP-SGD with momentum and a fixed, grid-searched $C$ in Opacus. We give it the same data, privacy accountant, epochs and batch size, and the same hyperparameter budget of trials as our method. This matters because a well-tuned fixed $C$ is often hard to beat. The non-private optimizer run on the same model provides the ceiling, and on synthetic data it gives the "efficiency parity" target borrowed from robust statistics.
The strongest existing method is DiceSGD (Differentially Private SGD Without Clipping Bias). It reports 86.5% versus 78.9% for standard DPSGD-GC on CIFAR-100 with a pre-trained ViT-small at $\varepsilon=2$. We will run it with its published clipped-feedback settings, retune its learning rate and $C$ within the same trial budget, and report peak memory alongside accuracy. Beating it means matching or exceeding its accuracy at $\varepsilon=2$ within noise across three seeds, while using no additional optimizer memory beyond DP-SGD with momentum. Matching its accuracy at one third the extra memory counts as a win on the stated question. Falling clearly below it counts as a loss.
The strongest adaptive-threshold method is DC-SGD-E (DC-SGD), which reports a 10.62% accuracy improvement over DP-SGD on CIFAR-10 with ResNet34 at $\varepsilon=2$. It spends budget on private histograms, so we will charge that cost within the same total $\varepsilon$. On MNIST at $\varepsilon=1$, SlaClip reports 94.15% against 92.23% for vanilla DP-SGD (SlaClip), and we include it as the zero-budget adaptive reference.
Three ablations isolate the key ingredient. The first is anchored clipping with a fixed, tuned $C$, which removes the adaptive schedule. The second is the adaptive residual-norm schedule without centering, which is anchor $=0$. The third centers on the previous single noisy gradient instead of momentum, which tests anchor quality. If the uncentered adaptive variant matches the full method, centering is not the source of the gain and the claim fails.
Why not just…?
Why not just use DiceSGD? It is the real threat, because error feedback and anchoring both return clipped-away signal to the update. The concrete difference is resource cost and assumptions: DiceSGD stores an extra error buffer and uses a larger noise multiplier, and its guarantee assumes sub-Gaussian gradients. If DiceSGD dominates at equal memory on CIFAR-100 and GPT-2 fine-tuning, anchoring is merely a cheaper approximation, and we will report it as such. The head-to-head run and the memory measurements in the baselines settle this.
Why not just tune or adapt the threshold better? Methods like DC-SGD and quantile clipping choose the size of $C$ well, but any threshold around zero must trade off against $\|\nabla f\|$ whenever $\rho_t$ is non-negligible. The uncentered-adaptive ablation is designed to answer this question directly. The prediction that the gap between centered and uncentered versions tracks the measured $\rho_t$ makes the answer quantitative rather than anecdotal.
Why not just apply momentum after ordinary clipping? Standard DP-SGD with momentum already averages noise over time, but it changes nothing about what gets clipped. Each example's gradient is still shrunk toward zero, so the bias enters the momentum and persists. Our naive baseline is exactly this configuration, so any gain we report is over momentum-after-clipping.
Why not normalize gradients, as heavy-tailed optimization suggests? From Gradient Clipping to Normalization for Heavy Tailed SGD analyzes normalized SGD only without privacy. With privacy, normalization is zero-centered clipping at an infinitesimal threshold, and the added noise scales with $\sqrt d$ relative to a unit-norm signal. It is included implicitly as the small-$C$ end of the naive baseline's grid.
Proposed Experiments
The synthetic stage uses linear regression with Student-$t$, log-normal and Lomax noise, plus clean Gaussian noise, across dimensions from 10 to 1,000. The non-private reference points come from Adaptive Huber Regression, which reports mean error 0.806 versus 1.355 for OLS under Student-$t$ noise and 3.917 versus 8.529 under log-normal noise, with almost no efficiency loss on normal data. The decisive robustness-efficiency check has two parts. On Gaussian data, anchored DP-SGD should match the best-tuned DP-SGD error at every $\varepsilon$, meaning no efficiency penalty. On heavy-tailed data, it should reduce error, and the reduction should track $\rho_t$. We also rerun the $\chi^2$-noise quadratics of Revisiting Gradient Clipping. There we check whether the error floor scales with residual tails rather than gradient magnitude, and whether the growing $C_t$ drives it down.
The deep-learning stage covers four settings, all at $\varepsilon \in \{1, 2, 8\}$ with three seeds and matched trial budgets. The first is MNIST and FashionMNIST at $\varepsilon=1$. The second is CIFAR-10 with ResNet34 at $\varepsilon=2$, against DC-SGD. The third is CIFAR-100 with ViT-small fine-tuning at $\varepsilon=2$, against DiceSGD. The fourth is GPT-2 fine-tuning on E2E NLG, reporting BLEU and ROUGE-L. For E2E NLG, no reported numeric result was found on Lacuna for DiceSGD; the file states only that it outperformed the baseline. For each run we report accuracy, peak memory, wall-clock time, and the logged non-private $\rho_t$.
The single most informative outcome is the scatter of accuracy gain against mean $\rho_t$ across all tasks. A clear positive relationship confirms that anchoring works for the stated reason. A flat relationship means any gains come from somewhere else and the mechanism claim is wrong.
References
- Adaptive Huber Regression
- SlaClip: Gradient Norm Slacks can be Indicator for Adaptive Clipping in DP-SGD
- A Huber Loss Minimization Approach to Mean Estimation under User-level Differential Privacy
- Perturbed M-Estimation: A Further Investigation of Robust Statistics for Differential Privacy
- All-Purpose Mean Estimation over R: Optimal Sub-Gaussianity with Outlier Robustness and Low Moments Performance
- Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach
- DC-SGD: Differentially Private SGD With Dynamic Clipping Through Gradient Norm Distribution Estimation
- On the Convergence of DP-SGD with Adaptive Clipping
- Adaptive Clipping for Differential Private Federated Learning in Interpolation Regimes
- From Gradient Clipping to Normalization for Heavy Tailed SGD