Gradient clipping caps the length of each update step and is used almost everywhere in deep learning: language models, recurrent networks, and private training. Theory says that when the gradient is noisy, clipping does not just shorten the step. It tilts the average step away from the true downhill direction, and this tilt leaves training stuck some distance from a solution. Practice mostly shrugs. The same authors who build costly fixes for this bias find that the cheap, "biased" version works just as well on their own benchmarks. Nobody has measured the bias directly inside real deep-network training and asked whether its size predicts the damage.
This note proposes that measurement. At checkpoints during ordinary training runs, we estimate how much of clipping's distortion is harmless shrinkage and how much is a real change of direction. We then test whether that directional part predicts how much a de-biasing method gains over plain clipped training. The result would be an empirical rule for when bias-correcting clipping methods earn their extra memory and compute, checked against the simpler rules a practitioner would try first.
Core Claim
Only one part of clipping bias should matter. When clipping sits inside a noisy gradient estimate, the average clipped step can be written as a positive multiple of the true gradient plus a leftover vector perpendicular to it. The multiple is just a smaller effective learning rate, which tuning absorbs. The perpendicular leftover is what drags the optimizer to the wrong resting point.
Let $g$ be a mini-batch gradient with mean $\nabla f(x)$, let $\mathrm{clip}_c(g)=g\min(1,c/\|g\|)$ be clipping at threshold $c$, and write
$$\mathbb{E}[\mathrm{clip}_c(g)] = \alpha\,\nabla f(x) + b_\perp,\qquad b_\perp \perp \nabla f(x),$$
where $\alpha>0$ is the shrinkage factor and $b_\perp$ is the directional bias. Our hypothesis is that the ratio $\rho = \|b_\perp\| / (\alpha\|\nabla f(x)\|)$ predicts how much a de-biasing method improves final loss and accuracy.
The hypothesis is falsifiable. We expect $\rho$ to stay small in standard non-private training with batch-level clipping, which would explain why biased single-sample clipping works fine in practice. We expect $\rho$ to become large under per-sample clipping with added privacy noise, which would explain the large reported gains of error-feedback methods there.
Where This Sits
Two theory papers motivate the question.
Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees proves that clipped SGD cannot get closer to a stationary point than roughly $\min\{\sigma,\sigma^2/c\}$, where $\sigma$ is the gradient noise level. It shows this floor does not depend on the step-size schedule. Its experiments stop at logistic regression on w1a and synthetic quadratics.
Convergence of Clipped SGD on Convex (L0,L1)-Smooth Functions introduces double sampling to remove bias in its proofs. Double sampling draws one mini-batch to decide the clipping factor and a second, independent one to take the step. Yet that paper finds single sampling "just as well, if not better" on California Housing and Parkinsons data.
We take these two results as a contradiction to resolve, not as a settled fact. Our proposal differs from both by measuring the bias vector itself in non-convex networks, rather than bounding it or inferring it from final loss.
Three methods papers supply the de-biasing tools we will compare, and set the bar.
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (DiceSGD) keeps a buffer of clipped-off gradient mass and adds it back into later steps. It reports 86.5% versus 78.9% for standard private clipped SGD on CIFAR-100 with a ViT-small at privacy budget $\epsilon=2$.
- Clip21: Error Feedback for Gradient Clipping applies the same idea across machines in distributed training.
- On the Convergence of DP-SGD with Adaptive Clipping removes bias by slowly raising the clipping quantile toward one.
We do not propose another de-biasing algorithm. We ask when any of them is needed.
One paper constrains the design. Understanding Gradient Clipping In Incremental Gradient Methods shows that the angle between mini-batch gradients and the true gradient governs how severe the bias is. That makes angular variance a competing predictor we must beat or match.
The reproducibility study [Re] Can gradient clipping mitigate label noise? is a warning. There, clipping-based losses that worked in synthetic settings lost to tuned cross-entropy on CIFAR-100 and were very sensitive to threshold choice. Every comparison here therefore uses equal tuning budgets.
Research Program
The measurement tool. At about twenty checkpoints per run, we freeze the weights and draw $K$ independent mini-batches at the training batch size, with $K$ around 256. From these we compute:
- the average raw gradient, as an estimate of $\nabla f$;
- the average clipped gradient;
- the per-batch noise level $\sigma^2$.
Projecting the average clipped gradient onto the estimated true gradient gives $\alpha$; the remainder gives $b_\perp$. A split-half resampling estimate subtracts the noise in $b_\perp$ caused by using finite $K$. For per-sample clipping in private training, we apply the same procedure to the per-sample clipped average before privacy noise is added.
This costs a few hundred extra gradient evaluations per checkpoint. It is used only as a diagnostic, never as part of the optimizer.
The main study. We sweep a grid in each setting over clipping threshold, batch size, learning rate, and, in private runs, noise multiplier. Each configuration is trained four ways:
- plain clipped SGD;
- clipped SGD with double sampling;
- the relevant error-feedback method (DiceSGD for private training, Clip21 for the distributed setting);
- a shrinkage-matched control, described in the Baselines section.
For each configuration we record the de-biasing gain, defined as the improvement in final training loss and in test accuracy over plain clipped SGD, and we record $\rho$ over the course of training.
The central analysis. We compare how well each candidate predictor ranks the de-biasing gain across configurations. The candidates are $\rho$, the fraction of steps that were clipped, the theoretical floor $\sigma^2/c$ from Koloskova et al., and the angular variance of Qian et al. We use rank correlation, and hold-out prediction: fit a threshold on one setting and apply it unchanged to another. We also check whether the gradient norm at which training plateaus scales with $\sigma^2/c$ as the theory predicts. This gives a direct test, on deep networks, of a bound so far checked only on convex models.
Baselines
The naive reference points are two cheap runs. One is SGD without clipping at a tuned learning rate. The other is standard single-sample clipped SGD, tuned over threshold and learning rate with the same budget as every other method.
On CIFAR-10 with ResNet-18, Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness reports clipped SGD at approximately 95% test accuracy, above Adam. Any claim that bias harms non-private image training must therefore show a de-biased method clearly exceeding a reproduced run near that level.
The strongest existing competitor for private training is DiceSGD, at 86.5% versus 78.9% for standard private clipped SGD on CIFAR-100 with ViT-small at $\epsilon=2$. We rerun both at matched privacy budget. We use DiceSGD's own larger noise multiplier and record peak memory and wall-clock time. "Beating" DiceSGD is not the goal. Success means our predictor, measured on a cheap pilot run, correctly flags this setting as one where de-biasing pays off.
For GPT-2 on E2E, the file reports only that DiceSGD beats the baseline on BLEU, ROUGE-L and CIDEr, with no reported numbers found on Lacuna. We will report our own. For Clip21 on CIFAR-10 with VGG11, the paper reports training loss more than three orders of magnitude below clipped SGD at equal epochs. This makes it the natural strongest method for the distributed setting.
The ablation that isolates our key ingredient is a shrinkage-matched control. It replaces per-step clipping with a single global rescaling of the raw gradient, chosen so that its average step length equals $\alpha$ times the true gradient. This removes the directional distortion while keeping the shrinkage. If clipped SGD matches this control, the direction change is harmless. If it falls behind, the gap is exactly the damage done by $b_\perp$.
Double sampling serves as a second bias-free control at twice the gradient cost.
Why not just…?
Why not just increase the batch size? Qian et al. show that larger batches shrink the angular variance that drives bias, and our grid includes batch size for exactly this reason. But under per-sample clipping with privacy noise, the bias comes from each individual example's gradient being clipped, so a bigger batch does not remove it. Batch size is also bounded by memory in streaming and on-device settings. The study will show directly whether $\rho$ falls with batch size in non-private runs and stays high in private ones. If it falls everywhere, we will report batch scaling as the sufficient fix.
Why not just use Adam or tune the learning-rate schedule? Koloskova et al. prove the floor is independent of the step-size schedule for fixed thresholds. The Stability paper finds clipped SGD beating Adam on ResNet-18. Both claims come from narrow settings, so we include Adam and a tuned cosine schedule as baselines at equal budget rather than assuming the answer.
Why not just log the fraction of clipped steps? This is the most serious threat. The clip fraction is free, and it may predict de-biasing gains as well as $\rho$ does. We treat it as a head-to-head competitor in the central analysis. If it matches $\rho$ on held-out settings, the practical recommendation becomes "watch the clip fraction", which is a useful and simpler result. We would then report $\rho$ only as the explanation of why that works.
Why not just run an error-feedback method everywhere? DiceSGD keeps an extra buffer the size of the model and needs a larger privacy noise multiplier. Clip21's guarantees are mainly for deterministic distributed gradients. Measured memory and wall-clock time decide whether always-on de-biasing costs more than it returns when $\rho$ is small.
Proposed Experiments
Sanity replications. We first reproduce the convex findings: the w1a floor from Koloskova et al., and the single-versus-double sampling result on California Housing from Gaash et al. We then measure $\rho$ there. Our prediction is that California Housing shows small $\rho$, which would explain why single sampling suffices.
Deep-network settings. We then run the full grid on:
- CIFAR-10 with ResNet-18 and non-private batch clipping;
- CIFAR-10 with VGG11 in the distributed setting with Clip21;
- CIFAR-100 private fine-tuning of ViT-small with DiceSGD;
- E2E private fine-tuning of GPT-2.
The decisive checks. There are three.
- Across all deep configurations, the gap between clipped SGD and the shrinkage-matched control should rise with $\rho$. If it does not, directional bias is not the mechanism, and the core claim fails.
- A $\rho$ threshold fitted on CIFAR-10 should correctly predict, before full training, that CIFAR-100 private fine-tuning benefits from DiceSGD while non-private ResNet-18 does not.
- In deep networks, plateau gradient norms should follow $\sigma^2/c$ as the threshold $c$ varies, or visibly depart from it. Either outcome is informative about whether the convex theory carries over.
References
- Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees
- Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach
- Clip21: Error Feedback for Gradient Clipping
- On the Convergence of DP-SGD with Adaptive Clipping
- Understanding Gradient Clipping In Incremental Gradient Methods
- [Re] Can gradient clipping mitigate label noise?
- Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness