Measured-Constant Clipping: Setting Non-Euclidean Trust Regions from Sharp Bounds and Online Curvature Estimates

Gradient clipping caps how far a training step can move. For many years it was treated as a stabilizing trick. Recent work shows it is something more principled: under a geometry chosen per layer, such as the spectral norm for weight matrices or the max-coordinate norm for embeddings, clipping is what you get from taking the steepest step inside a ball of fixed radius. The theory that justifies it is called $(L_0,L_1)$-smoothness. It assumes that local curvature grows at most linearly with the gradient's size, instead of being bounded by one global constant. In practice, though, the ball radius is still a hand-tuned hyperparameter, and the published bounds hide their constants inside big-O notation. The theory therefore explains why clipping helps but cannot say how much to clip.

This note proposes to close that loop and to test whether the closed loop works. We derive the exact one-step-optimal clipping radius under non-Euclidean $(L_0,L_1)$-smoothness, with no slack constants. We estimate the two curvature parameters online, per layer, in each layer's own norm. We plug the estimates into the formula and train without tuning the radius. The scientific question is whether the theory, once its constants are made sharp and its parameters measured, predicts the radius that a grid search finds. If it does, the theory becomes a usable tool. If it does not, we learn whether the problem lies in the constant, in the curvature model, or in the noise.

Core Claim

Let $\|\cdot\|$ be the norm chosen for a layer and $\|\cdot\|_*$ its dual. Assume the Hessian, measured as an operator from the norm to its dual, satisfies $\|\nabla^2 f(x)\| \le L_0 + L_1\|\nabla f(x)\|_*$. Integrating the resulting bound on gradient growth along a segment of length $r$ gives, for the steepest-descent step $d$ with $\|d\|=r$ and current dual gradient norm $G$:

$$f(x+d) \le f(x) - rG + \frac{L_0 + L_1 G}{L_1^2}\left(e^{L_1 r} - 1 - L_1 r\right).$$

The argument is a Grönwall-type inequality: the dual gradient norm can grow at most exponentially at rate $L_1$ along the path. One-dimensional functions whose curvature meets the condition with equality show the bound cannot be improved. Minimizing the right-hand side over $r$ gives a closed-form step length:

$$r^\star(G) = \frac{1}{L_1}\ln\!\left(1 + \frac{L_1 G}{L_0 + L_1 G}\right).$$

For small gradients this behaves like an ordinary step of size about $G/L_0$. For large gradients it saturates at $\ln 2 / L_1$. That saturation value is the clipping radius, with an exact leading constant.

The proposal turns on three points. First, this sharp form holds in any norm, including the layer-wise product norms used by Clipped Scion. Second, $L_0$ and $L_1$ can be estimated cheaply during training. Third, the resulting radius, used without tuning, matches or beats a tuned radius. The first point is a short derivation; the substantive risk lies in the second and third.

Relation to Prior Work

Three papers supply the setting and motivate the gap.

Analysis of Gradient Clipping and Adaptive Scaling with a Relaxed Smoothness Condition introduced $(L_0,L_1)$-smoothness in Euclidean geometry. It proved that clipped gradient descent needs $\mathcal{O}(L_0/\epsilon^2 + L_1^2/L_0)$ iterations, against a cost that grows with the largest gradient norm for plain gradient descent, but it gave only order-of-magnitude constants. Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness lifted clipping to arbitrary norms as a trust-region subproblem, yielding Clipped Sign, Clipped Spectral and Clipped Scion. It proved an $\mathcal{O}(n^{-1/4})$ stochastic rate and reported speedups on NanoGPT and DeiT, but it leaves the radius hand-tuned and does not estimate curvature parameters. Mirror Descent Under Generalized Smoothness showed empirically, on GPT-2 trained on FineWeb, CIFAR-10 CNNs and a WMT'16 Transformer, that curvature follows an affine link to the gradient's dual norm under $\ell_1$-type geometry. It uses that observation only to justify an assumption; its step sizes still depend on constants it describes as unknown in practice.

We reuse the non-Euclidean step of the second paper and the measurement methodology of the third. What we add is the sharp constant and the use of measurements to set the step, which neither paper does.

Two papers shape the design as constraints. Robustness to Unbounded Smoothness of Generalized SignSGD shows that Adam's coordinate-wise normalization already acts as implicit clipping, with Adam training curves "virtually identical" with and without explicit clipping on GPT-2/Wikitext-103. This is a real threat: explicit radii may not matter for Adam-like updates. Our experiments therefore target the steepest-descent family, where the radius is the operative control, and include Adam as a competitor rather than a host. Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysis documents a theory-practice gap: its proofs need momentum $\beta<0.5$ while practice uses $0.9$. This warns that sharp one-step constants may still be wrong under momentum. We address that by testing the formula with momentum-averaged gradients and reporting the gap honestly.

Adaptive backtracking line search is the closest precedent for optimizing step-size constants online without extra cost. It reports, for example, a "57.6% reduction in elapsed time" for gradient descent on logistic regression across 15 datasets. That method is Euclidean, assumes ordinary smoothness, and needs exact function values, so it cannot run on mini-batches. Our method is its non-Euclidean, stochastic counterpart. Instead of repairing a fixed shrink factor, it computes the step from a measured curvature model.

The motivation for treating constants as the object of study comes from the constant-sharpening literature, such as Learning with Square Loss: Localization through Offset Rademacher Complexity. It also comes from the classic warning in A Local Clustering Algorithm for Massive Graphs and Its Application to Nearly Linear Time Graph Partitioning, whose provably safe parameters were too conservative to be adopted.

Research Program

The theoretical piece has three parts. We prove the descent inequality above for general norms. We also prove that $r^\star$ is worst-case optimal among fixed-radius steps given $(L_0,L_1,G)$, by exhibiting functions on which the bound holds with equality. Finally, we extend the result to product norms, where each layer has its own $(L_0^{(l)},L_1^{(l)})$ and the coupled one-step problem separates into per-layer radii. We then redo the stochastic analysis of the GGNC paper with momentum gradient estimates in place of $G$. The goal is to state the convergence bound with explicit constants and to quantify how much slack the original $\mathcal{O}(n^{-1/4})$ analysis carried.

The estimation piece works as follows. Every $k$ steps, we evaluate two gradients on the same mini-batch at consecutive iterates. This removes sampling noise from the difference and costs one extra backward pass per $k$ steps, which we charge to our compute budget. From each pair we form a secant curvature in the layer's operator norm, $\|g_{t+1}-g_t\|_*/\|x_{t+1}-x_t\|$. We fit $L_0^{(l)}$ and $L_1^{(l)}$ by an upper-quantile linear regression of secant curvature against $\|g_t\|_*$ over a sliding window. The upper quantile is used because the theory needs an upper envelope, not a mean fit. The quantile level is the method's only free parameter. We fix it once on a small proxy task and never retune it, so the method is honestly tuning-free at the target scale.

The empirical piece is the decisive one. For each task we grid-search the clipping radius of Clipped Scion and compare the best radius with the radius our formula predicts from measured constants.

Baselines

The naive reference point is the same steepest-descent optimizer (Scion with its layer norms) with no clipping, at its best tuned learning rate. The GGNC paper reports Clipped Scion as ">10% speedup over the unclipped baseline" on 1B NanoGPT/FineWeb and an "11% speedup" on DeiT-base/ImageNet-1k, so this baseline tells us whether any clipping gain is present at our scale. We also include Euclidean clipped SGD with momentum as in the original $(L_0,L_1)$ paper, which reports 95.2% top-1 on CIFAR-10 with ResNet20, and AdamW at default momentum settings, because Generalized SignSGD argues Adam already clips implicitly.

The strongest competitor is Clipped Scion with its radius and learning rate tuned by grid search. It is the strongest because it uses the same non-Euclidean step we use, and tuning absorbs any constant the theory gets wrong. For the 124M NanoGPT scale we use, no reported validation-loss number was found on Lacuna for tuned Clipped Scion, so we reproduce it ourselves. For fairness, every method gets the same data order, token budget and hardware. Each baseline gets a grid of at least eight radius values and eight learning rates. Our method gets one run per seed, with the extra gradient passes counted in wall-clock time. Beating it means matching its final validation loss within seed variance at equal wall-clock time while using no radius search. A stronger win, which we do not assume, is lower loss at equal time because per-layer, time-varying radii adapt in a way no single tuned constant can.

Three ablations isolate the key ingredient. The first keeps the measured constants but replaces $\ln 2/L_1$ with a typical loose analysis constant. This tests whether sharpness matters. The second measures constants in the Euclidean norm and plugs them into the non-Euclidean step. This tests whether native-norm measurement matters, in the spirit of the $\sqrt{n}$ penalty that the mirror-descent paper attributes to Euclidean analysis. The third uses one global $(L_0,L_1)$ pair instead of per-layer pairs.

Why not just…?

Why not just grid-search the radius, as GGNC does? At small scale this is a fair answer, and if a single radius transfers across model sizes it is a cheap one. The threat is real. Our proposal earns its place only if the tuned radius drifts with scale, across layers, or over training, so that one-shot tuning at 124M is suboptimal at larger sizes or later phases. We test this directly. We record the tuned optimum at two model sizes and check whether our predicted radius tracks its movement. If the tuned radius is constant and our method merely ties, we report that the contribution is a validated explanation of the tuned value, not a performance gain.

Why not just use Adam, which already clips implicitly? Generalized SignSGD shows explicit clipping adds nothing to Adam on GPT-2/Wikitext-103. But GGNC reports that non-Euclidean product-norm steps beat its baselines at 1B scale and on ImageNet-1k. The relevant question is therefore whether radius-setting matters within the steepest-descent family, and AdamW is included as a head-to-head baseline so readers can see whether that family is worth using at all.

Why not just estimate Euclidean $(L_0,L_1)$ and use ordinary clipping? This is cheaper and needs no operator-norm computations. The second ablation tests exactly this. The mirror-descent paper's finding that curvature aligns with non-Euclidean norms predicts that Euclidean estimates will be loose for matrix layers. If they are not, the non-Euclidean machinery is unnecessary for step-setting and we will say so.

Why not just run a stochastic Armijo line search? A mini-batch backtracking search also adapts the step without tuning. However, adaptive backtracking line search, the strongest line-search method in the file, is restricted to exact function values and Euclidean geometry, and a mini-batch Armijo test requires extra forward passes at every step. We include a same-batch Armijo variant applied to the Scion direction as a baseline at matched wall-clock cost. Its own constants stay at textbook defaults.

Proposed Experiments

The tasks follow the file's benchmarks: AWD-LSTM on Penn Treebank, where the $(L_0,L_1)$ paper reports clipping permits "learning rate as high as 30"; ResNet and CNN on CIFAR-10; NanoGPT at 124M on FineWeb, with a 350M run if compute allows; and MovieLens 100K matrix factorization and synthetic phase retrieval, where curvature is known to grow with the gradient. Metrics are final validation loss or perplexity or accuracy, wall-clock time to a target loss, and the ratio of predicted to tuned-optimal radius.

The first decisive check is that the ratio of predicted to tuned radius lies within a factor of two on every task. Missing that would falsify the claim that sharp theory predicts practice. The second check is the sharp-versus-loose-constant ablation. If loose constants cost more than seed variance while sharp ones do not, the constant itself matters. The third check is calibration of the measured envelope. On held-out step pairs, the fraction of secant curvatures exceeding the fitted $L_0 + L_1 G$ should match the chosen quantile. If it does not, the affine curvature model, not the constant, is what fails.