Do Optimizer Add-ons Stack? Declared Invariants as a Predictor of Composition

Over the last few years, many small "add-ons" for standard optimizers like AdamW have each been shown to help on their own. One centers gradients to zero mean. One raises gradients to a power before the optimizer sees them. One reduces gradient noise with a recursive momentum estimate. One blends each layer's update with its neighbors'. One keeps a slow copy of the weights that trails a fast copy. Practitioners naturally want to stack several of them, but no published work tests whether the gains add up. There is reason to doubt they do: several add-ons quietly assume properties of their input that another add-on destroys. This proposal asks a simple empirical question and offers a cheap tool to answer it in advance. When two optimizer add-ons are stacked, does the combined gain equal the sum of the parts? And can we predict which stacks will fail by having each add-on declare what it assumes and what it guarantees about the signal it passes on?

The idea borrows a move from software engineering. Evaluation systems such as OmniEvaluator became reliable by forcing each stage to emit a shared intermediate format, so that any stage could feed any other. We apply the same discipline to the optimizer step. The step is split into a few fixed stages. Each add-on declares the invariants it needs, such as a zero-mean gradient slice, per-layer independence, or an unchanged update scale, and the invariants it establishes. A checker then flags stacks where a later stage breaks an earlier stage's guarantee. The scientific bet is that these flags predict, before any training run, which pairs interfere. If the bet holds, practitioners get a principled way to compose add-ons. If it fails, we learn that add-on interactions are not explained by these simple algebraic properties, which is also worth knowing.

Core Claim

Optimizer add-ons, which the literature calls "wrappers" because they sit around an unchanged base optimizer, are now evaluated one at a time against a single base optimizer. Some are additionally tested for compatibility with a second base optimizer such as Muon. Our claim has two parts. First, pairwise stacks of published add-ons show substantial non-additive interactions, both positive and negative, once learning rate is retuned for each stack. Second, the sign of each interaction is predicted well above chance by whether the stack violates a declared invariant. A corollary is directly testable: re-establishing a violated invariant after the offending stage should recover most of the lost gain at negligible cost. Re-establishing means re-centering after a power transform, or rescaling after cross-layer blending to restore the base optimizer's update norm.

The motivating tensions are concrete. Gradient Centralization projects each weight slice's gradient onto a zero-mean hyperplane and reports CIFAR-100 gains of 0.5% to 2%. GradPower applies $|g|^p\,\mathrm{sign}(g)$ with $p=1.2$ before AdamW. Because this map is nonlinear, applying it after centering breaks the zero mean, and applying centering after it changes the power-law relation the method relies on. Gradient Smoothing mixes each layer's update with its neighbors' and explicitly renormalizes to keep the base optimizer's scale. Any later add-on that rescales updates per layer could undo that. MARS moved clipping inside its variance-reduced estimator because clipping at the update level behaved differently. That is direct evidence that where an operation sits in the step matters. Lookahead operates on whole weight trajectories rather than single steps, so its interaction with per-step transforms is a separate open question.

Research Program

We first build a thin staged interface for the optimizer step, with five stages. The gradient-transform stage covers centering and power transforms. The estimator stage covers MARS-style variance reduction and momentum. The base-step stage runs AdamW or Muon unchanged. The update-transform stage covers depth-wise smoothing and norm rescaling. The weight-track stage covers Lookahead's slow and fast weights. Each add-on is implemented once against this interface rather than as a subclass per base optimizer, as the MARS codebase currently does with MARS-AdamW, MARS-Lion and MARS-Shampoo (AGI-Arena/MARS).

Each add-on also ships a short machine-readable contract drawn from a small vocabulary of properties:

  • zero mean per weight slice;
  • sign preservation;
  • per-parameter independence, meaning no cross-layer coupling;
  • invariance to uniform scaling of the gradient;
  • preservation of the base optimizer's per-layer update norm;
  • whether the transformed gradient remains an unbiased estimate of the true gradient.

The contracts are written from each paper's stated mechanism, before any composition experiment is run. This matters because it makes the checker's predictions genuinely out-of-sample.

The interface itself is not the contribution. Composable chains of gradient transformations already exist in common libraries. What is new is the contract layer and the composition study it enables.

The second part of the work is the measurement. For each add-on $A$, we define its gain $\Delta_A$ as the reduction in log tokens-to-target-loss relative to the tuned base optimizer. For a stack $A{\to}B$, where order matters, the interaction is

$$I_{A\to B} = \Delta_{A\to B} - \Delta_A - \Delta_B,$$

where $\Delta_{A\to B}$ is the gain of the stack. We use the log because speedups multiply. Zero interaction therefore means the two speedups compound independently. The checker's prediction is binary: violation or no violation. We test whether violated stacks have significantly lower $I$ than compatible ones. As a third arm, we test whether automatic invariant re-enforcement raises $I$ for the violated stacks.

Baselines

The naive reference is the base optimizer alone: AdamW, and Muon as a second base. Its learning rate, weight decay and warmup-stable-decay schedule are tuned on a fixed grid. Every add-on and every stack receives the same grid, the same data order and the same token budget, over three seeds. The seed count is needed because full determinism is not guaranteed on GPU, a limitation DeepDIVA itself reports. Retuning the learning rate per stack is essential. Several add-ons change the effective step size, so an untuned stack could look like interference when it is merely miscalibration.

The strongest single competitor is MARS on AdamW, run from its official code. MARS reports reaching a GPT-2 Large validation loss of 2.58 in 28 billion tokens, compared with 50 billion for AdamW, with final losses of 2.51 versus 2.58 at 50 billion tokens (MARS). Our main runs use GPT-2 small and medium on OpenWebText. I found no reported result on Lacuna for MARS at those scales, so we reproduce it ourselves. On vision, the strongest single add-on reference is Gradient Smoothing's reported ViT-B/4 CIFAR-100 improvement from 74.56% to 75.62% (Gradient Smoothing).

Beating these baselines means that the best contract-compatible stack, or the best stack after re-enforcement, reaches the target loss in significantly fewer tokens than the best single add-on, with the margin exceeding seed variance. Merely beating plain AdamW does not count.

The key ablation is each violated stack with and without invariant re-enforcement, holding everything else fixed. This isolates whether the declared invariant is causally responsible for the interaction or only correlated with it. A second ablation swaps stage order, for example centering-then-power versus power-then-centering. This tests the specific order-dependence the contracts predict.

Why not just…?

Why not just tune AdamW harder? Several add-ons report gains over tuned AdamW. GradPower beats a well-tuned AdamW under identical schedules from 66M to 2B parameters, and Lookahead improves Penn Treebank perplexity to 57.72 from Adam's 59.33. Our protocol retunes the base for every comparison, so any surviving gain is by construction not a tuning artifact. If extra tuning erases all single gains at our scale, the study reports that, and it is a meaningful finding.

Why not just use the best single add-on? This is the real threat. If interactions are mostly negative, the best single add-on wins, and the practical value of stacking disappears. The experiment settles the question directly: the full pairwise grid shows whether any stack beats the best single add-on. The contract question remains valuable even then, because it explains why stacking fails instead of leaving practitioners to rediscover it.

Why not just search over stacks with black-box hyperparameter optimization? Search can find a good stack without any theory. However, the number of ordered stacks grows quickly, and each evaluation is a full training run. Our claim is that contracts prune the space before training. We measure this by comparing how many runs a random search needs to find a stack within seed noise of the best, with and without contract-based pruning.

Why not a learned optimizer that discovers the combination itself? Optimizer Amalgamation distills several base optimizers into a recurrent learned optimizer. It needed weight perturbations to stabilize meta-training and was evaluated only on small CNNs and ResNets. That is far from the transformer pre-training regime we target, and it offers no interpretable account of why combinations work.

Proposed Experiments

The primary testbed is GPT-2 small (124M) on OpenWebText, using the MARS codebase pipeline. The add-on pool is centering, GradPower at $p=1.2$, MARS-approx, Gradient Smoothing, and Lookahead with $k=5$ and $\alpha=0.5$. These settings are the published static values; scheduling them over training is out of scope. Every single add-on and every ordered pair is run on the AdamW base. The pairs that pass compatibility on AdamW are then rerun on the Muon base. The predicted-best and predicted-worst stacks are confirmed at 355M.

Vision replication uses ViT-B/4 and ResNets on CIFAR-100, so that the reported Gradient Smoothing and centering numbers can be checked first. One stress test uses streaming video on WalkingTour (WTvenice). There, orthogonal-gradient projection lifts kNN ImageNet accuracy from AdamW's 1.8% to 51.8% (Learning from Streaming Video with Orthogonal Gradients). Projection assumes a particular gradient geometry that power transforms would alter, which makes it a sharp contract case.

Metrics are tokens-to-target validation loss, final validation loss, HellaSwag accuracy for the language models, wall-clock overhead per step, and the interaction $I$ for each stack.

The decisive check is pre-registered. All contract verdicts are committed before the composition grid runs. The study succeeds only if three things hold:

  • Violated stacks show significantly lower $I$ than compatible stacks across seeds.
  • Re-enforcement recovers at least half of the lost interaction on average.
  • The predicted order-dependence appears in the order-swap ablation.

If the verdicts predict nothing, we report that add-on interactions are not explained by these invariants.