Breaking Precision Walls in First-, Second-, and Zeroth-Order Optimizers: One Constant, Set in Advance

Lngel KyoarXiv 2026 · arXiv

In the world of optimization, we often judge success by the average: how quickly does the validation loss drop? But this average can be deceptive. In many real-world datasets, signal strength follows a power law—a few "strong" directions dominate, while many "weak" directions contain rare but vital information.

This paper reveals that standard optimizers often "silently miss" these weak directions, and it introduces a way to fix this by tuning a single, often-ignored hyperparameter.

TL;DR

The paper introduces the precision wall—a metric for how well an optimizer learns weak signals in power-law data. It demonstrates that a single denominator constant (like $\epsilon$ in Adam) determines this wall and can be calculated in advance to significantly improve an optimizer's ability to capture rare data patterns.

What Problem Does This Paper Address?

When data follows a power law (common in everything from word frequencies to feature spectra), an optimizer might fit the strong directions perfectly while completely failing to learn the weak ones. Because the weak directions contribute little to the total loss, this failure is hidden by a low average loss.

The authors define the precision wall as the steepest signal decay rate at which an optimizer can still recover the weakest direction to within 5% given a fixed budget. They tackle the challenge of identifying what sets this wall and how to "break" it to allow for more precise learning.

Methods

The researchers analyzed how the "small constants" added to the denominators of adaptive optimizers—like $\epsilon$ in Adam, damping in Shampoo, or grafting constants—control the learning rate for weak signals.

  • First-Order Analysis: They found that for methods like Adam and grafted Shampoo, once the signal becomes weak enough, the optimizer effectively behaves like momentum gradient descent with a step size determined by the ratio of the learning rate to this constant ($\eta/c$).
  • Zeroth-Order Analysis: For optimizers that estimate gradients from loss values alone (like MeZO), they developed a mean-square model to predict a "noise floor" and a critical constant value ($\epsilon^*$) below which the optimizer fails.
  • The "One Constant" Rule: They proposed setting these constants based on the problem's curvature or spectrum before training, rather than using arbitrary defaults.

Datasets

The study evaluated these ideas across several synthetic and real-world-inspired benchmarks:

  • Low-rank matrix sensing: A standard task for studying signal recovery.
  • Quadratics: Simple diagonal and complex Hadamard-rotated versions.
  • Population Logistic Regression: A theoretical model of classification.
  • Handwritten Digits (MNIST): Logistic regression on digits, projected onto principal modes to precisely control signal decay.

Key Insights and Ideas

  • Average loss is a lie: An optimizer can achieve a very low average loss while completely failing to learn the "tail" of the data distribution.
  • The "Stability-Precision" Trade-off: The constant $\epsilon$ isn't just for avoiding division by zero; it sets the "resolution" of the optimizer. If it's too large, weak signals are ignored; if it's too small, the optimizer can become unstable.
  • Predictable Walls: The failure point of an optimizer is mathematically predictable. As shown in the paper's results, theoretical forecasts of where the wall would land were remarkably accurate.
Figure 3: The 68 scored walls against their forecasts. The differences, right, lean below zero.
  • Grafting restores Shampoo: By matching the grafting constant in Shampoo to the square root of its damping, the authors were able to restore its ability to learn weak signals, succeeding where previous hypotheses had failed.

Key Results

By setting the denominator constant according to their derived rules, the authors successfully raised the precision wall, allowing optimizers to learn much weaker signals than default settings allowed:

  • Adam: Raised the wall by 1.96 and 1.15 units on matrix sensing tasks.
  • Grafted Shampoo: Saw improvements of 1.59 and 1.74 units.
  • Zeroth-order Adam: Improved the wall by 1.12 and 0.73 units on rotated quadratics.
  • Forecast Accuracy: Across 68 scored walls, 56 landed below the forecast, with a root-mean-square difference of only 0.1276.

Limitations and Future Work

  • Scale: The studies were mostly full-batch on relatively small problems (up to 288 parameters). Testing these rules on massive models like LLMs is a necessary next step.
  • Curvature Knowledge: The most effective version of the rule requires knowing the problem's curvature at the solution. While the authors explored estimating this from data, more robust estimation methods are needed for complex, non-quadratic landscapes.
  • Minibatch Noise: The current zeroth-order model assumes low-noise settings; extending this to handle the high gradient noise of typical minibatch training is a key area for future research.

Summary updated

Abstract

Under power-law signal decay across directions, an optimizer can fit strong directions and miss weak ones while average loss hides it. A precision wall measures the steepest decay where the weakest direction recovers within 5% in a fixed budget. On low-rank matrix sensing and quadratics one constant sets the wall: once it dominates the denominator, Adam or grafted Shampoo is momentum descent at step $η/c$, setting the wall; in Shampoo, matching grafting's constant to damping restores the wall after three hypotheses failed. In zeroth-order Adam, $\varepsilon$ acts against $\varepsilon^\ast$ from a mean-square spectral model; below $\varepsilon^\ast$ error hits a noise floor. Set from true curvature or spectrum on new shapes, the rule raised the wall above the same-step default by a forecast amount: on two sensing shapes by $1.96, 1.15$ (Adam) and $1.59, 1.74$ (grafted Shampoo), and on two Hadamard-rotated quadratics by $1.12, 0.73$ (zeroth-order Adam, longer budget), rising on all seeds; fixed-step momentum at that step reaches those walls. A cosine-scheduled default (small constant giving weak modes large steps) is higher on one sensing shape, matched on new seeds of the other, and lower with the rule's constant. A default step retuned per budget catches up on rotated quadratics and population logistic regression, fails at 1 query/step on two diagonal quadratics (catching up at 16), and exceeds the rule on digits logistic regression (full-batch and minibatches, where gradient noise sets the $η= 0.01, 0.1$ default walls). On logistic regression the rule did not raise the first-order wall: same-step default Adam was higher (its wall on 4 new shapes at the forecast stability limit), while in zeroth order on the population loss setting $\varepsilon$ raised the wall by $0.98, 0.88$.
BibTeX
@misc{kyo2026breaking,
  author = {Lngel Kyo},
  title = {{Breaking Precision Walls in First-, Second-, and Zeroth-Order Optimizers: One Constant, Set in Advance}},
  year = {2026},
  eprint = {2611.10730},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10730},
}
Figure