Scale Invariant and Parameter Free Optimization

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 193 papers121 conceptsfrozen 2026-10-04

The performance of Stochastic Gradient Descent (SGD) is famously sensitive to the choice of learning rate. If the learning rate is too high, the model diverges; if it is too low, training stalls. This sensitivity is often a byproduct of scale variance: if you multiply your loss function by a factor of 10, you must typically divide your learning rate by 10 to maintain the same update trajectory.

A growing body of research seeks to eliminate this "pesky" hyperparameter tuning by designing optimizers that are scale-invariant. These algorithms ensure that the sequence of model updates remains identical regardless of how the loss function, gradients, or input features are scaled.

The Intuition of Scale Invariance

In standard SGD, the update $\Delta w = -\eta \nabla L(w)$ is directly proportional to the magnitude of the gradient. If the loss $L$ is scaled by a constant $\alpha$, the gradient $\nabla L$ also scales by $\alpha$, forcing the user to manually adjust the learning rate $\eta$ to compensate.

Scale-invariant optimizers solve this by incorporating the "units" of the problem into the denominator of the update rule. As noted by Tom Schaul, Sixin Zhang, and Yann LeCun in No More Pesky Learning Rates, incorporating inverse curvature and gradient variance can create a scale-invariant process. When the optimizer accounts for the variance of the gradients, a global scaling of the loss affects both the numerator and the denominator, effectively canceling out.

Achieving Invariance through Normalization

One of the most direct ways to achieve scale invariance is to modify the denominator of adaptive optimizers. Standard methods like AdaGrad use a square root of the sum of squared gradients. However, Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad introduces KATE, which removes the square root. This simple change ensures that the update step scales linearly with the input data, making the optimizer robust to poorly scaled features in generalized linear models.

Similarly, Xiaoxia Wu, Leon Bottou, and Rachel Ward proposed WNGrad: Learn the Learning Rate in Gradient Descent. WNGrad achieves scale invariance relative to the loss function by adjusting the learning rate based on observed gradients, ensuring that multiplying the loss by a constant does not alter the optimization trajectory.

In the context of deep networks, Zhenxun Zhuang argues in Understanding AdamW through Proximal Methods and Scale-Freeness that the success of AdamW stems from its "scale-free" nature. By being invariant to the component-wise rescaling of gradients, AdamW provides a form of automatic preconditioning that handles the wildly varying gradient scales common in deep architectures.

Optimization as a Betting Game

A more radical departure from traditional calculus is found in the work of Francesco Orabona and Tatiana Tommasi. In Backprop without Learning Rates Through Coin Betting, they introduce COCOB, which reframes optimization as a game of betting on coin flips.

Instead of a learning rate, the algorithm decides how much of its current "wealth" to bet on the direction of the next gradient. Because the betting strategy is based on relative gains rather than absolute magnitudes, the algorithm is naturally scale-invariant. This allows COCOB to match the performance of expertly tuned Adam or SGD without requiring any manual hyperparameter setting.

Meta-Learning Scale Invariance

If manually designing scale-invariant rules is difficult, can we train an optimizer to be scale-invariant? Ashok Cutkosky, Aaron Defazio, and Harsh Mehta developed Mechanic: A Learning Rate Tuner, a wrapper that automatically tunes the global learning rate. Mechanic is designed to be scale-free, ensuring that model updates remain consistent even if input gradients are multiplied by a constant.

This idea is extended in Celo: Training Versatile Learned Optimizers on a Compute Diet by Abhinav Moudgil. By using task augmentation—randomly re-scaling the parameters and losses of the problems the optimizer is training on—the meta-optimizer is forced to learn scale-invariant features. This prevents the optimizer from overfitting to specific loss magnitudes and allows it to generalize across vastly different architectures.

Stability in Complex Domains

The benefits of scale invariance extend beyond simple classification. In physics-based simulations, where gradients can be highly non-linear and ill-conditioned, Nils Wandel and colleagues show in Metamizer: a versatile neural optimizer for fast and accurate physics simulations that scale-invariant normalization allows a single neural optimizer to solve diverse physical systems (like fluids and cloth) without problem-specific tuning.

Similarly, in the realm of online learning and bandits, Mingyu Chen and Xuezhou Zhang demonstrate in Scale-free Adversarial Reinforcement Learning that "strong scale-freeness" can be achieved. By initializing clipping thresholds at zero, they ensure that an agent's actions remain identical regardless of how rewards are rescaled, providing robust performance even when the range of possible losses is entirely unknown.

Summary

Scale-invariant optimization represents a shift from manual tuning to structural robustness. By designing update rules that are mathematically indifferent to the magnitude of the objective, researchers are creating a new generation of "plug-and-play" optimizers. Whether through adaptive denominators, betting frameworks, or meta-learned invariance, the goal remains the same: an optimization process that is defined by the geometry of the problem, not the arbitrary units of its description.

Papers in this version

Showing 60 of 93 papers.

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scale_invariant_and_parameter_free_optim_7105d47b,
  title = {Scale Invariant and Parameter Free Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-invariant-and-parameter-free-optimization/rdr_7105d47b24d24d4688c22bbdcfa7227f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_scale_invariant_and_parameter_free_optim_7105d47b_v1,
  title = {Scale Invariant and Parameter Free Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scale-invariant-and-parameter-free-optimization/rdr_7105d47b24d24d4688c22bbdcfa7227f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}