Measuring Step Quality on a Fresh Augmented View: A Trust-Region Ratio for Training with Data Augmentation

Some optimizers decide how far to step by checking their own predictions. After each step they compare the drop in loss they actually got against the drop their local quadratic model promised. When the two agree, the optimizer takes bigger steps. When they disagree, it takes smaller ones. This check, the "actual-versus-predicted reduction ratio," is what lets second-order methods such as AdamQLR train without hand-tuned learning rates. Nearly every image model, however, is trained with data augmentation: random crops, flips, and color changes that give the network a different version of each image every time it sees it. With augmentation, the objective being minimized is an average over all possible augmented versions of the data. The optimizer's check is computed on one particular version. That mismatch goes unexamined.

We propose a simple change and a careful study of it. After proposing a step on one augmented view of a mini-batch, the optimizer measures the actual loss reduction on a second, independently augmented view of the same images. The loss of that second view has the same expected value as the first, so the cross-view ratio measures how much of the step's progress carries over to the augmented objective, rather than to the specific random crop the step was fitted on. The added cost is essentially nothing, because the optimizer already runs a forward pass at the candidate point. The gap between the same-view and cross-view ratios is a second product: a per-step, per-transformation measurement of how much the model is fitting view-specific detail instead of learning the intended invariance.

Core Claim

Trust-region control of step size, whether through an explicit radius or through Levenberg–Marquardt damping (adding a multiple of the identity to the curvature matrix so steps shrink), should be driven by a ratio whose numerator estimates reduction of the augmented objective. The standard same-view numerator is biased upward. The step was chosen to reduce loss on exactly those pixels, so the check rewards overfitting to that sample, much as training error flatters a model. A numerator computed on fresh images is unbiased but noisy. A numerator computed on a fresh augmentation of the same images keeps most of the variance reduction that comes from reusing images, and it removes the bias that comes from reusing the augmentation. Write $F(x)=\mathbb{E}_{i,a}\,\ell(x; a(z_i))$ for the augmented objective, where $z_i$ is an image and $a$ a random augmentation. Write $\ell_B^{a}$ for the mini-batch loss on images $B$ under augmentation draw $a$. The quadratic model $m_k$ is built on draw $a$. The proposed ratio is

$$\rho_k^{\times} = \frac{\ell_B^{a'}(x_k) - \ell_B^{a'}(x_k+p_k)}{m_k(0) - m_k(p_k)},$$

where $a'$ is an independent draw and $p_k$ is the proposed step. We compare it against the same-view ratio $\rho_k^{=}$, which uses $a'=a$, and the fresh-batch ratio $\rho_k^{\circ}$, which uses new images $B'$. The claim is empirical and falsifiable: driving damping with $\rho^{\times}$ gives better test accuracy and more stable damping trajectories than either alternative, and the advantage grows with augmentation strength.

Grounding in Prior Work

The method comes from Adam through a Second-Order Lens. That paper's AdamQLR keeps Adam's update direction but chooses step length and damping from a quadratic model built on Fisher-vector products (curvature-times-vector products computed without forming the full curvature matrix), adjusted by the reduction ratio. The file reports that untuned AdamQLR matches tuned first-order baselines on Fashion-MNIST, SVHN and CIFAR-10. It treats the data loader as static, and we change only the ratio's numerator. The same ratio-driven damping goes back to Deep learning via Hessian-free optimization. A consistently adaptive trust-region method modifies the ratio's denominator with a gradient-norm term. Our change is to the numerator, so the two are compatible, and we test that combination in one ablation.

Two stochastic trust-region papers constrain the design. Sub-sampled Cubic Regularization for Non-convex Optimization and TRSVR both show that sharing or adaptively sizing samples between the model and the check is workable, but their guarantees assume i.i.d. draws from a fixed finite dataset. Augmentation turns that into a continuous distribution over views. We therefore analyze the variance of $\rho^{\times}$ rather than claim their rates carry over.

Two papers motivate the proposal. Implicit Rugosity Regularization via Data Augmentation shows that augmentation suppresses the network's curvature along the data manifold (its "rugosity"). Rugosity drops from about 0.43 to about 0.10 on CIFAR-10 ResNets, and CIFAR-100 accuracy rises from 49.3% to 64.5%. That paper also finds that an explicit rugosity penalty does not reproduce augmentation's benefit. That negative result steers us away from replacing augmentation with a curvature term and toward making the optimizer respect the augmented objective. LA3 shows that augmentation effects are class-dependent: Posterize costs up to 5% on CIFAR-10 cats while helping dogs. This motivates our secondary use of the same-view/cross-view gap as a per-class, per-transformation diagnostic.

Two lines of work sit nearby without answering our question. When to Learn What (MADAug) adapts augmentation with a Tanh schedule plus a bi-level policy network. It adapts the data to the model, never the optimizer to the data. V2X-DGW uses "trust region" to mean a physically reliable spatial zone in a LiDAR scene. It shares the term only.

Research Program

We will build the three ratio variants into a single AdamQLR-style implementation. The only thing that differs between them is which view or batch the post-step forward pass uses.

The first strand is analysis. We decompose the expected same-view ratio into the cross-view ratio plus an "augmentation-overfitting" term. That term is the covariance between the step and view-specific loss noise, and it is zero for $\rho^{\times}$ by independence of $a$ and $a'$. We also bound the variance of $\rho^{\times}$ relative to $\rho^{\circ}$ in terms of the fraction of loss variance explained by image identity versus augmentation draw. This yields a checkable prediction: $\rho^{\times}$ should beat $\rho^{\circ}$ when image-to-image variance dominates, and the two should converge when augmentation variance dominates.

The second strand is empirical. It covers the training comparison and a set of diagnostics. We log damping trajectories, step-rejection rates and the same/cross gap across training. We break the gap down by transformation type and by class, and we test whether the gap predicts which transformations LA3-style search would find harmful for a given class.

The third strand is an optional extension that we will run only if the core result holds. It lowers the sampling probability of transformations whose gap stays large, and it is judged against the curricula in the Baselines section.

Baselines

The naive reference point is AdamQLR with the same-view ratio, run under each augmentation policy exactly as specified in Adam through a Second-Order Lens. It is the direct competitor because it is what a practitioner gets today by turning on augmentation with a ratio-driven optimizer. The file contains no reported CIFAR-10 accuracy for AdamQLR, so we will reproduce it ourselves. We count a win as higher mean test accuracy across five seeds, with non-overlapping standard-error intervals, at matched wall-clock time.

The key ablation is the fresh-batch ratio $\rho^{\circ}$, together with a clean-view ratio that evaluates the unaugmented images. These isolate whether the benefit comes specifically from an independent augmentation of the same images. If $\rho^{\circ}$ matches $\rho^{\times}$, the contribution reduces to "don't reuse the batch." We would report it as that.

The strongest practical competitor is a well-tuned first-order pipeline: SGD with momentum and a cosine schedule, with RandAugment and with a MADAug-style Tanh curriculum. Each receives the same tuning budget that we spend on AdamQLR's few hyperparameters, and we also give them a larger budget to be conservative. For reference, When to Learn What reports CIFAR-100 error with PyramidNet of 10.5% for MADAug, 10.7% for AutoAugment and 11.9% for RandAugment. LA3 reports 98.55% CIFAR-10 accuracy with PyramidNet and 79.97% ImageNet top-1 with ResNet-50. Second-order passes on PyramidNet exceed our budget, so we will not claim to beat those absolute numbers. We will rerun RandAugment and a Tanh curriculum on our ResNet backbones under identical epochs and data. Beating them there, or matching them without schedule tuning, is the bar.

Why not just…?

Why not just evaluate the ratio on the same augmented batch, as implementations already do? This is the naive baseline itself. The argument above says it rewards steps that fit the specific crop. The decisive test is whether the same/cross gap grows with augmentation magnitude and whether same-view damping settles too low under strong augmentation, taking overconfident steps that hurt test accuracy. If neither effect appears, the same-view ratio is fine and our proposal fails.

Why not just use a fresh batch, or smooth the ratio with a running average? A fresh batch removes bias but adds image-sampling noise. The file flags that noise as a known source of false rejections in stochastic trust regions, and it is what SCR and TRSVR handle with larger or shared samples. Averaging the ratio over time reduces noise but leaves the bias in place. Both are run as ablations: $\rho^{\circ}$, and an exponentially averaged $\rho^{=}$. This alternative is a real threat when augmentation variance dominates, and the variance analysis predicts exactly where it should tie.

Why not just use SGD with a tuned schedule and a good augmentation policy? Evidence in the file favors this for peak accuracy. LA3's offline policy reaches strong results cheaply, and AdamQLR carries roughly 4x SGD's memory. We do not claim to beat tuned SGD at unlimited tuning budget. Our claim is that ratio-driven training stays untuned-competitive under augmentation only if the ratio is fixed, and that the fix costs no extra passes. The matched-budget comparison settles whether that holds.

Why not just regularize flatness in weight space instead? Noise Stability Optimization finds its Hessian-trace penalty complementary to augmentation, not a substitute. Flatness regularization changes where training ends up. Our change governs how large each step is, so the two are orthogonal, and we include one run of NSO combined with our ratio as a compatibility check.

Proposed Experiments

We train ResNet-18 on CIFAR-10 and CIFAR-100, and smaller networks on SVHN and Fashion-MNIST to match AdamQLR's settings. Augmentation strength sweeps RandAugment magnitude from none to heavy, with five seeds per cell. Metrics are test accuracy, damping and step-size trajectories, step-rejection rate, and wall-clock time. As a curvature analysis, we measure end-of-training rugosity using the protocol of Implicit Rugosity Regularization via Data Augmentation. This checks whether cross-view control keeps augmentation's rugosity reduction, which we expect, or undoes it.

Four checks are decisive. First, the accuracy advantage of $\rho^{\times}$ over $\rho^{=}$ must increase with augmentation magnitude and vanish with no augmentation. At zero augmentation the two ratios are identical by construction, so this doubles as a sanity check. Second, $\rho^{\times}$ must beat $\rho^{\circ}$ at small batch sizes, where image noise is largest, and tie at large ones, as the variance analysis predicts. Third, the per-class, per-transformation gap should rank-correlate with measured per-class harm from each transformation, which is a direct test against LA3's Posterize finding on CIFAR-10. Fourth, at matched wall-clock time, untuned AdamQLR with $\rho^{\times}$ should come within seed noise of tuned SGD with RandAugment on the ResNet benchmarks. If the first check fails, we drop the method claim and report the negative result on how ratio-based optimizers interact with augmentation.