Target-Free Curvature for Faster Masked Autoencoder Pretraining

Masked autoencoders learn visual features by hiding most of an image's patches and training a network to repaint the missing pixels. The recipe works well, but it is slow: competitive models are pretrained for hundreds to over a thousand passes over ImageNet using AdamW, a first-order optimizer that sees only gradients. Second-order optimizers instead scale each parameter's step by how sharply the loss curves along that parameter. In language models, one such optimizer roughly halved pretraining cost. Nobody has checked whether this carries over to masked image pretraining, and the vision setting has a structural property that changes how such an optimizer should be built.

The property is simple. When the reconstruction loss is squared error on pixels, the Gauss–Newton curvature does not depend on the target pixels at all. The Gauss–Newton matrix is the standard positive approximation to the Hessian, the matrix of second derivatives of the loss. It depends only on how the network's outputs at the masked positions respond to each parameter. Its diagonal can therefore be estimated without bias, using one extra backward pass on random noise injected at the masked outputs. No special derivation through attention layers is needed, and no labels are sampled. The same property holds for feature-distillation variants whose regression targets drift during training, so the curvature estimate stays valid when the teacher changes. This note proposes a Sophia-style optimizer built on this target-free diagonal for pixel and feature masked image modeling, together with the diagnostic that decides whether it should beat a well-tuned AdamW at all.

Core Claim

For masked image modeling with a squared-error loss, a periodically refreshed, target-free Gauss–Newton diagonal can do two things. It can reach the pretraining loss and downstream accuracy of a tuned AdamW recipe in materially fewer epochs at matched wall-clock cost. It can also be applied unchanged across bootstrapped-teacher stages. The claim is falsifiable by a cheap measurement made before any long run.

If, within each transformer block, the Gauss–Newton diagonal is nearly proportional to the square root of AdamW's running squared gradient, then AdamW is already a rescaled version of the same preconditioner. In that case a per-block learning-rate change should capture any gain. If the two disagree coordinate by coordinate inside a block, the dynamic per-parameter diagonal has room to help.

Mechanism

Let $f_p(\theta)$ be the decoder's prediction for masked patch $p$ in the masked set $M$, with $\theta$ the parameters and $y_p$ the normalized pixel target. The loss is

$$L(\theta) = \frac{1}{|M|}\sum_{p \in M} \lVert f_p(\theta) - y_p \rVert^2 .$$

Its Hessian with respect to the outputs is the constant $(2/|M|)I$. The Gauss–Newton matrix is therefore $G = (2/|M|)\sum_{p\in M} J_p^\top J_p$, where $J_p$ is the Jacobian of $f_p$ with respect to $\theta$.

To estimate the diagonal of $G$, draw Gaussian noise $u_p \sim \mathcal N(0, I)$ at each masked output. Backpropagate the scalar $\sum_p u_p^\top f_p$ to obtain $g = \sum_p J_p^\top u_p$. Then

$$\mathbb E_u[g \odot g] = \operatorname{diag}\Big(\sum_{p \in M} J_p^\top J_p\Big),$$

where $\odot$ denotes elementwise product. The target $y$ never appears in this expression.

The optimizer follows the update structure of Sophia. It keeps a gradient moving average, refreshes the curvature estimate every $k$ steps with this one-pass estimator, and divides the gradient by the curvature with per-coordinate clipping. Clipping is still needed because the diagonal can be tiny in flat directions.

Two design questions are specific to masked image modeling. The first is whether the curvature pass should reuse the training step's mask or draw a fresh mask. A fresh mask decorrelates the preconditioner from the gradient noise, at the cost of an extra encoder forward pass on 25% of the tokens. The second is whether curvature carries across teacher changes. For bootstrapped feature distillation of the kind used in dBOT, the loss is again squared error, so the curvature estimate carries across stage boundaries without reset even though the targets jump.

Relation to Prior Work

The optimizer literature supplies the method and motivates the transfer. Sophia pairs periodic diagonal curvature (Hutchinson or Gauss–Newton–Bartlett) with clipping and reports a 2x speedup over AdamW on GPT-style models. It explicitly leaves masked autoencoders and bidirectional encoders untested. Its Bartlett estimator samples labels from a softmax, a step that has no counterpart and no need in pixel regression; we replace it with the exact Gaussian-output estimator above.

AdaHessian estimates the true Hessian diagonal at every step and was tested on convolutional networks. That per-step cost is reported at roughly 2.4–3x first-order cost and is the main barrier we avoid.

HesScale and its follow-up show that starting from exact loss-layer curvature and propagating diagonally beats squared-gradient proxies in accuracy. The authors note that extending their recursion to attention is open. Our estimator also starts from the exact loss-layer curvature, which is trivial for squared error, but it obtains the parameter diagonal from an ordinary autodiff pass. It therefore needs no attention-specific rule, at the price of Monte Carlo variance. That variance is exactly what the estimator ablation below measures.

AGD reads curvature off successive gradient differences at about 7% overhead. It is the cheapest competitor and is tested here directly.

On the masked modeling side, MAE establishes the asymmetric encoder–decoder recipe whose encoder sees only visible patches. That design makes any extra backward pass cheaper than in dense transformers. The survey by Hondru et al. identifies pretraining cost and the missing account of the loss geometry as open problems, which motivates this work.

Self-Guided MAE and AdaMAE speed learning by changing which patches are hidden. We change how parameters move instead, so these methods are complementary rather than competing.

Hessian-aware ViT pruning found a "less–more–less" sensitivity profile across supervised DeiT layers and found value projections more sensitive than query and key projections. This constrains our design. If a similar profile exists in masked autoencoders and is mostly block-level, a block-wise learning-rate schedule could capture the gain, which is why that schedule is a central baseline.

Research Program

The work proceeds in three stages, each gating the next.

The first stage is a geometry study of standard MAE pretraining with AdamW. We use ViT-S and ViT-B on ImageNet-1K and take checkpoints across training. At each checkpoint we compute three quantities: the target-free Gauss–Newton diagonal, Hutchinson estimates of the true Hessian diagonal, and AdamW's second-moment buffer. These are broken down by encoder block, decoder block, and query, key, value and MLP projections. On ViT-Tiny we additionally measure what fraction of the true Hessian diagonal mass is negative, to check whether the Gauss–Newton approximation discards important negative curvature.

The headline statistic is the within-block rank correlation and scale dispersion between the Gauss–Newton diagonal and the square root of Adam's second moment. Low correlation, or high dispersion that changes over training, is the precondition for a per-coordinate optimizer to beat AdamW.

The second stage builds and tunes the optimizer. The tuning choices are the refresh interval $k$, the clipping threshold, and same versus fresh curvature masks. All are tested on 100-epoch ViT-B runs, with wall-clock time logged alongside steps.

The third stage tests the claim at longer budgets on pixel targets and on dBOT-style feature distillation. For distillation we check whether carrying curvature across stage boundaries helps compared with resetting it.

Baselines

The naive reference point is the published MAE AdamW recipe run unchanged at each epoch budget. The Self-Guided MAE paper reports this recipe for ViT-B at 400 epochs at 82.5% ImageNet fine-tuning top-1 and 61.4% linear-probe top-1. Our 400-epoch reproduction must land near these numbers before any comparison counts.

The strongest competitor is tuned AdamW. Most published optimizer gains shrink once the incumbent is tuned fairly, so this is the comparison that matters. We give AdamW and the proposed optimizer the same number of trials over peak learning rate, weight decay, and second-moment decay rate, with the same data, augmentation, masking ratio, batch size, and hardware.

Beating tuned AdamW means one of two outcomes. The first is reaching tuned AdamW's 400-epoch pretraining loss and fine-tuning accuracy in at most 60% of its wall-clock time. The second is exceeding its accuracy by at least 0.3 points fine-tuning at equal wall-clock, averaged over three seeds.

The key ablation isolates per-coordinate curvature from block-level scaling. We use the stage-one measurements to set a fixed per-block learning-rate multiplier proportional to the inverse mean Gauss–Newton diagonal of each block, then train with AdamW. If this static schedule closes most of the gap, the claim reduces to "masked autoencoders need block-wise learning rates" and we will report it as such.

A second ablation swaps our estimator for Hutchinson's true-Hessian diagonal inside the same Sophia update. This isolates Gauss–Newton versus true Hessian curvature. AGD is run with its published defaults plus the same tuning budget, as the cheapest curvature-aware alternative.

For context we also report Self-Guided MAE, run with AdamW and with our optimizer, to test whether the two gains add. For feature distillation the reference is dBOT, which reports 84.5% ImageNet fine-tuning top-1 for ViT-B under its full schedule. We compare against dBOT at matched reduced budgets rather than expecting to exceed this figure.

Why not just…?

Why not just tune AdamW harder? This is the real threat. Adam's second moment is a diagonal preconditioner too. The HesScale measurements showing that squared gradients approximate the Hessian diagonal poorly come from small convolutional networks, not vision transformers. Sophia's 2x speedup was measured on language models with a different loss. We do not argue this away. The equal-budget tuned-AdamW baseline and the stage-one correlation statistic settle it. If the Gauss–Newton diagonal is nearly proportional to Adam's preconditioner within blocks and tuned AdamW matches us, the conclusion is that MAE pretraining is not curvature-limited, and that is a publishable negative result about the geometry the survey calls undocumented.

Why not just use block-wise learning rates? The pruning study's layer-wise sensitivity profile suggests much of the curvature variation may be between blocks rather than within them. Block-wise rates would be far simpler than a new optimizer. The static per-block ablation tests exactly this, using our own curvature measurements to set the multipliers so the baseline is as strong as possible.

Why not just use gradient differences, as AGD does? AGD costs almost nothing. However, it sees curvature only along the direction the parameters just moved, and in masked image modeling every step uses a new random mask and therefore a different loss. That makes successive gradient differences mix curvature with mask-to-mask variation. This is a hypothesis, not established. The direct comparison, and a measurement of how well AGD's implied diagonal correlates with ours on the same checkpoints, will show whether the extra backward pass buys anything.

Why not just improve the masking instead? Adaptive masking already yields gains: Self-Guided MAE moves ViT-B fine-tuning from 82.5% to 83.2%. But masking changes what is learned, while preconditioning changes how fast it is learned, so the gains need not compete. We run the combination to check additivity. If the gains do not add, that suggests both methods address the same slow directions.

Why not avoid second-order cost altogether? The overhead objection is fair for per-step estimators. Our estimator adds one backward pass every $k$ steps over an encoder that processes only a quarter of the tokens. We report per-step overhead and judge every result on wall-clock time, not step counts.

Proposed Experiments

Pretraining uses ImageNet-1K with ViT-S and ViT-B at budgets of 100, 200 and 400 epochs, with three seeds per configuration. We report ImageNet fine-tuning and linear-probe top-1, plus pretraining loss against wall-clock time. Transfer is measured on ADE20K semantic segmentation and COCO detection. As reference points, Self-Guided MAE reports 45.2 mIoU on ADE20K and 43.3 box AP on COCO; CUB-200 is used as a fine-grained check, where Self-Guided MAE reports 82.8%. MAE's own ADE20K and COCO numbers are given in the file only qualitatively: no reported result found on Lacuna for MAE ViT-B mIoU or box AP.

The decisive checks fall in an order. The first is the stage-one statistic: if within-block correlation between the Gauss–Newton diagonal and Adam's preconditioner exceeds about 0.9 throughout training, we expect no gain and stop after the 100-epoch confirmation. The second is whether the optimizer beats tuned AdamW by the wall-clock criterion in the Baselines section. The third is whether that gain survives the static per-block ablation. The fourth is whether it persists from 100 to 400 epochs rather than vanishing as AdamW catches up. The fifth is whether carrying curvature across dBOT teacher stages beats resetting it, which is the direct test of the target-free property.