Exact Hypergradients Through Large-Batch Inner Loops

Many learning problems are nested: an outer learner tunes something (per-sample loss weights, learning-rate schedules, a small synthetic training set) by watching how an inner learner's training responds to it. The standard way to get the outer gradient, called the hypergradient, is to run the inner training for $T$ steps and backpropagate through all of them. That is expensive in memory along three axes at once. It must store the inner weights at every step, the layer activations of the network at every step, and those activations grow with the inner mini-batch size. In practice the third axis is the one that bites. Reweighting and meta-learning pipelines run with inner batches of 1 to 16 examples and one or a few unrolled steps. So the hypergradient is computed for a far noisier and shorter inner training run than the one the tuned quantity will later be used in.

This proposal builds a memory schedule for reverse-mode unrolled hypergradients whose peak memory does not grow with the unroll length $T$, the network depth, or the inner batch size. It combines three known pieces. The first runs the inner optimizer backwards exactly, rather than storing its trajectory. The second streams each inner batch through the network in small slices, the way gradient accumulation does, but extended to the second-order backward pass. The third rebuilds activations inside each step, either with reversible layers or with checkpointing. With memory no longer the constraint, the proposal then asks an empirical question no prior work has answered: when inner batch size and unroll length are freed from memory limits, how do hypergradient quality and outer-task results change? And under a fixed compute budget, where should the recomputation budget go?

Core Claim

The claim has two parts. The first is that the inner batch size is a first-class, largely unstudied variable in unrolled bilevel optimization. Small inner batches make the inner trajectory chaotic. We expect this to inflate hypergradient variance beyond what averaging over outer steps can repair, a hypothesis to be tested rather than assumed. The second is that the memory obstacle to testing this can be removed without paying a compounding numerical cost.

The key design observation is about where reconstruction error can accumulate. If the inner weights $\theta_t$ at each step are recovered exactly in reverse, using fixed-point reversal of SGD with momentum plus a buffer that holds only the discarded bits, then the activations at step $t$ can always be recomputed afresh. They come from the exact weights $\theta_t$ and a mini-batch regenerated from its random seed. Activation reconstruction error therefore lives within a single step and cannot compound over the $T$ steps. This defuses the worry, raised in the research file's gap analysis, that low-precision reversible reconstruction would drift catastrophically when nested inside an unrolled loop. We make it a testable prediction: hypergradient error from 16-bit activation reconstruction should stay flat in $T$ under this design, and grow with $T$ under a naive design that reconstructs activations from the next step's activations.

How the pieces fit, and what each borrows

The hypergradient for an outer variable $x$ after $T$ inner steps $\theta_{t+1} = \theta_t - \eta \nabla_\theta g(x, \theta_t; b_t)$ on batches $b_t$ is accumulated backwards. An adjoint vector $a_t$ starts as the validation-loss gradient at $\theta_T$ and is propagated by

$$a_t = a_{t+1} - \eta \nabla^2_{\theta\theta} g(x,\theta_t; b_t)\, a_{t+1}, \qquad \nabla_x F \mathrel{+}= -\eta \nabla^2_{x\theta} g(x,\theta_t; b_t)\, a_{t+1}.$$

Momentum adds a velocity adjoint but changes nothing essential. Every quantity needed at step $t$ is a Hessian-vector or mixed-derivative-vector product on batch $b_t$. Each is a sum over examples in the batch. So it can be computed slice by slice, holding only one slice's activations at a time. Peak memory is then roughly

$$\text{Mem} \approx c\,M + \text{Buf}(T) + \text{Act}(B_{\text{slice}}, L),$$

where $M$ is the parameter count, $c$ a small constant for weights, velocity, adjoints and optimizer state, $\text{Buf}(T)$ the discarded-bit buffer, and $\text{Act}$ the activation memory of one slice of size $B_{\text{slice}}$ through a depth-$L$ network. The buffer is the only term that grows with $T$, and Gradient-based Hyperparameter Optimization through Reversible Learning reports that it is up to roughly 1000 times smaller than storing the trajectory. The inner batch size $B_{\text{in}}$ appears only in compute, as the number of slices.

The roles of the cited work are distinct. Maclaurin et al. supply the weight-space reversal and the bit buffer, but they leave per-step activations at standard depth-linear cost and never vary the inner batch size. We reuse their technique and add the other two axes.

Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization introduced (FG)²U, which removes trajectory memory by using forward-mode directional derivatives along random vectors. It scales to 1.5B-parameter models. The cost is estimator variance that needs many directions, plus forward-mode overhead it reports at 2–3×, and it also leaves per-step activations and inner batch size unaddressed. We keep reverse mode, which is exact with no direction-sampling noise, and compete with it on matched compute.

For the within-step activation term we compare two options. One is reversible coupling blocks in the style of Reversible Residual Network, which rebuild activations with constant memory in depth at a reported 33–50% compute cost. Here the needed Hessian-vector products can be taken as forward-mode derivatives of the reversible backward pass, which also needs no stored activations. The other is ordinary checkpointing. Memory-Efficient Differentiable Transformer Architecture Search is the closest prior use of reversible backbones inside a bilevel loop. That work is limited to architecture search with a fixed batch size and does not reverse the optimizer trajectory, so it neither decouples memory from $T$ nor studies $B_{\text{in}}$.

Two further lines of work constrain the design rather than supplying it. Retrofitting pretrained models into reversible form, as in Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning and Dr2Net, works. However, both papers report that low-precision reconstruction needs 32-bit arithmetic, layer freezing, or careful scaling coefficients to avoid divergence, and MEFT reports roughly 2× slower iterations. This is why we keep reconstruction local to a step and treat reversible blocks as one option to be compared, not as the thesis.

On the theory side, single-loop methods are the alternative for small batches: A Single-Timescale Method for Stochastic Bilevel Optimization, BiAdam, and Randomized Stochastic Variance-Reduced Methods for Stochastic Bilevel Optimization. They show that constant-batch methods can reach good sample complexity. But they assume a strongly convex inner problem, and STABLE and SVRB require per-iteration matrix inversion or SVD whose cost is cubic in the inner dimension. They motivate the question of whether small batches are "enough," and our experiments test that claim in deep, nonconvex inner problems.

Research Program

The build is a PyTorch library with three components. The first is a fixed-point SGD-with-momentum inner optimizer with an exact reverse step and a bit buffer, following Maclaurin et al. The second is a seed-indexed batch sampler, so batch $b_t$ can be regenerated during the reverse pass rather than stored. The third is a per-step second-order backward that streams slices and accepts either reversible coupling blocks or checkpointed standard blocks as its activation strategy.

The analysis has three threads. First, we measure peak memory and wall-clock time as functions of $T$, $B_{\text{in}}$ and depth, confirming the memory expression above. Second, on a model small enough to differentiate exactly in 64-bit arithmetic with full storage, we measure hypergradient fidelity: cosine similarity and relative error against the exact hypergradient. This covers 16-bit versus 32-bit reconstruction and step-local versus naive cross-step reconstruction, as $T$ grows. Third, on a small model with many trajectory samples, we measure hypergradient variance across inner batch sequences as a function of $B_{\text{in}}$ and $T$. The aim is to see whether variance falls like $1/B_{\text{in}}$, which would mean it is ordinary sampling noise, or faster, which would mean small batches also destabilize the chain of Jacobian products in the adjoint update.

Baselines

The naive reference point is each benchmark's released pipeline at its default small inner batch and one-step lookahead. For reweighting this is Meta-Weight-Net from Meta-Weight-Net, run from its official code with its WRN-28-10 and ResNet-32 backbones. That paper reports 68.91% on long-tailed CIFAR-10 at imbalance factor 200, 58.75% on CIFAR-100 with 60% uniform label noise, and 73.72% on Clothing1M. Beating this means a statistically clear gain over three seeds on at least two of these settings, at no more than 2× its wall-clock budget.

For data condensation, the strongest existing method on this question is (FG)²U, which reports 29.37% on CIFAR-10 at one image per class, against 20.78% for truncated reverse-gradient unrolling. We run (FG)²U with its published settings. We then sweep its number of random directions so its total FLOPs per outer step match ours, and give both methods the same outer-step budget and the same learning-rate grid. Beating it means higher accuracy at matched FLOPs and matched peak memory. A tie at matched FLOPs with lower hypergradient variance would be a partial result, reported as such.

Truncated reverse unrolling, under the same memory cap as our method, serves as the cheap biased baseline. Gradient accumulation with one-step lookahead is the large-batch baseline that needs no new machinery. We describe it under the alternatives below; it is identical to our method at $T=1$, so it isolates whether $T>1$ matters at large batch.

Two ablations isolate the key ingredients. Replacing reversible blocks with checkpointing, at the same $T$ and $B_{\text{in}}$, isolates the activation strategy. Replacing exact optimizer reversal with storing every $k$-th weight vector and recomputing in between isolates the bit buffer. Finally, a VR-BiAdam-style variance-reduced outer update at the small default batch tests whether algorithmic variance reduction alone recovers the large-batch gains.

Why not just…?

Why not just use gradient accumulation to get a large inner batch? For one-step lookahead this works, and it is exactly what our method reduces to at $T=1$. One pass over slices without a graph forms the lookahead weights, and a second pass computes the mixed-derivative product. So the large-batch, one-step result is a baseline, not a contribution. Accumulation breaks down for $T>1$, because reverse mode needs each step's weights and activations again. Doing that without storage is precisely what optimizer reversal and per-step recomputation provide. If one-step accumulation matches multi-step large-batch unrolling on the benchmarks, the honest conclusion is that inner batch size, not horizon, was the bottleneck, and the paper reports that.

Why not just use gradient checkpointing instead of reversible layers? This is a real threat, so checkpointing is a full arm of the design rather than a strawman. Checkpointing has no reconstruction drift and needs no architectural change. Its activation memory is about $\sqrt{L}$ rather than constant, which matters only for deep or high-resolution inner networks. We expect checkpointing to match reversible blocks on WRN-28-10 at CIFAR resolution, and reversible blocks to win only in deeper or wider settings, where they allow larger slices and therefore better hardware utilization. The memory/time sweep will locate the crossover. The core claim does not depend on reversibility winning.

Why not just use implicit differentiation and skip unrolling? Implicit differentiation assumes the inner problem has reached a stationary point and requires an inverse-Hessian product. Shen et al. note that the stationarity assumption rarely holds within practical budgets and biases the hypergradient. The reweighting and condensation tasks studied here are inherently short-horizon, so the inner solution is far from converged. We include a conjugate-gradient implicit baseline at matched compute on condensation, where non-convergence should hurt it most.

Why not just keep small batches and use a variance-reduced single-loop method? STABLE, SVRB and VR-BiAdam give good sample complexity at constant batch size under strong convexity. But variance reduction smooths the outer estimate; it does not change which noisy inner trajectory is being differentiated. STABLE and SVRB also need cubic-cost inversions that rule them out at WRN scale. The VR-BiAdam-style baseline measures directly whether this is enough. If it closes the gap, that is a useful negative result for our thesis.

Proposed Experiments

Experiments run on CIFAR-10 and CIFAR-100 (long-tailed and noisy variants), Clothing1M, and CIFAR-10 condensation at one and ten images per class, all on a single 24 GB accelerator so memory is a binding constraint. The central experiment is a grid over inner batch size (from the default up to 512) and unroll length (1 to 50) at fixed total FLOPs per outer step, with final outer-task accuracy as the metric. This tells practitioners where to spend recomputation.

Three checks are decisive. If hypergradient error under 16-bit, step-local reconstruction grows with $T$, the central design argument fails. If hypergradient variance falls no faster than $1/B_{\text{in}}$ and outer accuracy is flat in $B_{\text{in}}$, then small inner batches were never the problem, and the field's default is vindicated. If accuracy rises with $B_{\text{in}}$ at fixed compute, and that gain survives the variance-reduced small-batch baseline, the claim that memory limits on the inner batch have been silently degrading bilevel results is supported. The checkpointing comparison then decides whether reversible layers earn their 33–50% compute cost.