What Is Left of Curriculum Augmentation After Equal Tuning?

Data augmentation means training on randomly altered copies of the inputs: crops, color shifts, noise, mixed images. A popular refinement changes the strength of these alterations over the course of training. Usually it starts gentle and ramps up. Some methods learn the ramp with an auxiliary network. Others adjust it per class or per sample. These "curriculum augmentation" methods report gains of one to four points over fixed augmentation.

Two problems sit behind those gains. First, the baselines are usually a fixed policy at default strength, not one tuned with comparable effort. The benchmarking literature has shown repeatedly that this kind of asymmetry can create the entire gain. Second, the field disagrees about the shape a schedule should take. Ramp-up methods end training at maximum strength. The "slingshot" finding instead ends training with augmentation switched off.

This note proposes a matched-budget study that settles both questions together. We describe every schedule, from fixed to ramp to slingshot, with the same four numbers. We tune that small family with exactly the same trial and compute budget that each learned method receives. We then ask how much of each published gain survives. Where gains survive, we test which part of the trajectory explains them.

Core Claim

We hypothesize that, under equal tuning and equal wall-clock time, three things hold. A four-number hand-set schedule matches learned-policy curricula on balanced classification to within seed noise. The only reliable residual advantage of adaptive, per-class or per-sample methods appears under class imbalance and label noise. Across all schedules, final accuracy is predicted better by augmentation strength in the last portion of training than by the total amount of augmentation applied.

Each part is falsifiable. If learned policies keep a margin larger than two standard errors on CIFAR-100 after equal tuning, the first part fails. If ramp-to-maximum and slingshot schedules with the same final-phase strength produce different accuracies, the third part fails.

Relation to Prior Work

The benchmarking side motivates the study and fixes its constraints. Three results show that unequal tuning can manufacture progress:

  • Melis et al. found that a plain LSTM given the same hyperparameter search budget beat newer recurrent cells.
  • Krichene found that well-regularized matrix factorization beat neural collaborative filtering.
  • The Illusion of Progress from Under-Optimized Baselines reported, across 176 tabular datasets, that tuning effort matters more than model family.

Two results show that augmentation is often the hidden variable. Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline (SimpleView) showed that unified augmentation and loss choices lift PointNet++ from 90.0% to 93.3% on ModelNet40. It also exposed test-set checkpoint picking. Data Augmentation is Dead, Long Live Data Augmentation showed that classical text augmentation does not beat simply duplicating examples once the baseline trains longer with label smoothing. Both studies examine only fixed augmentation. We extend their logic to schedules.

A counterweight comes from Granero Moya et al. and Jiang et al.. They show that forcing identical hyperparameters on all methods is itself biased. For that reason we equalize the tuning budget, not the hyperparameter values.

The curriculum side supplies the methods we audit.

  • MADAug combines a tanh-shaped global ramp with a policy network trained to reduce validation loss. It reports 10.5% error on CIFAR-100 against 11.9% for RandAugment used as a fixed policy. It never separates the ramp from the policy network.
  • CUDA sets strength per class from a running "level of learning" score, at about 1.3× the compute. It is never compared with a global ramp.
  • EfficientTrain and EfficientTrain++ use a searched linear RandAugment ramp together with a low-resolution start. This is evidence that cheap fixed schedules can compete.
  • RangeAugment learns magnitude ranges online at roughly 2% overhead.
  • AdaPC learns point-cloud augmentation, reporting 81.75% on ScanObjectNN. It was never run against SimpleView under one protocol.

The shape debate comes from Tradeoffs in Data Augmentation: An Empirical Study. It measures augmentation along two axes. "Affinity" is how close augmented data stays to clean data. "Diversity" is how much new variation it adds. The same paper documents the slingshot effect of switching augmentation off late in training.

The closest existing benchmark is CurBench. It standardizes 15 curriculum methods but covers only curricula that select or reweight samples. It includes no augmentation schedules, and it does not tune the baseline with an equal budget. We reuse its three data settings (clean, noisy labels, long-tailed) and its practice of reporting wall-clock time.

Research Program

The central object is a shared schedule family. Let $t \in [0,1]$ be training progress and $s(t) \in [0,1]$ the augmentation strength, meaning RandAugment magnitude for images and jitter or scale range for point clouds. The family is piecewise linear:

$$ s(t) = \begin{cases} s_0 + (s_1 - s_0)\, t/\tau & t < \tau \\ s_2 & t \ge \tau \end{cases} $$

The four numbers are defined as follows:

  • $s_0$ is the starting strength.
  • $s_1$ is the strength reached just before the switch point $\tau$.
  • $\tau$ is the fraction of training at which the schedule switches.
  • $s_2$ is the strength held for the remainder of training.

Special cases cover every regime in the debate. A fixed policy is $s_0=s_1=s_2$. A ramp-up is $s_0<s_1=s_2$ with $\tau$ near 1. A slingshot is high $s_0=s_1$ with $s_2=0$ at $\tau\approx 2/3$.

The protocol applies to every method: fixed RandAugment, the schedule family, MADAug, CUDA, RangeAugment and AdaPC. Each method receives the same number of tuning trials from one shared random-search tuner. The search covers the method's own knobs plus shared knobs: learning rate, weight decay, label smoothing and epoch count. Each method also receives the same total GPU-hours. Methods with overhead therefore train for fewer epochs or run fewer trials, and cheap methods may train longer.

Checkpoints are selected on a validation split carved from the training data, never on the test set. Every configuration runs with five seeds. We report the mean, the standard error, and wall-clock time, following CurBench's accounting. Where the official code exists we build on it, notably CVNets for RangeAugment.

To test the trajectory hypothesis, we sweep a grid of the schedule family. For each run we log two proxies from the Tradeoffs study: affinity, and an early-training estimate of diversity. We then regress final accuracy on two competing summaries, strength in the final 20% of training and the area under $s(t)$.

Baselines

The naive reference is fixed RandAugment at its library-default magnitude, trained with the recipe each paper used. It reproduces the comparison the literature actually made. The gap between it and the tuned fixed baseline measures how much of each published gain was a tuning artifact.

The first strong baseline is fixed RandAugment tuned with the equal budget. This is the competitor the benchmarking literature says is usually missing. We will count a curriculum method as better only if it wins by more than two pooled standard errors across seeds.

The strongest existing method on balanced data is MADAug. On CIFAR-100 it reports 10.5% error against AutoAugment's 10.7% and RandAugment's 11.9% with PyramidNet. We rerun it with its code under our budget. We add two ablations that isolate its ingredients: its tanh ramp alone with no policy network, and its policy network under a flat schedule.

On imbalanced data the strongest competitor is CUDA. It reports 42.7% on CIFAR-100-LT against 38.7% for plain cross-entropy. Its ablation replaces per-class strength with a single global schedule from our family that tracks CUDA's class-averaged strength over time.

On point clouds the strongest reference numbers are AdaPC's 81.75% and SimpleView's 79.5% on ScanObjectNN. Both are run under one protocol, and AdaPC-style schedules are applied to SimpleView as well as PointNet++.

Finally, the tuned schedule family is itself a baseline. Our method is the protocol, not a new augmenter. The schedule family "wins" if it falls within two standard errors of every learned method at no added compute.

Why not just…?

Why not just tune fixed augmentation strength harder? This is the most serious threat, and it is one of our experimental arms. The schedule family contains the fixed policy, and both receive the same trial count. If tuned fixed strength matches everything, that is the headline result. It would agree with what Data Augmentation is Dead found for text, and we would report it as such. The Tradeoffs study's slingshot result gives a reason to expect schedule shape to matter at least sometimes, but the study will decide.

Why not just train the baseline longer, or duplicate data? Epoch count sits inside the shared search space, and the compute budget lets cheap methods train longer. Any gain that is really "more optimization" is therefore absorbed. For vision, the Tradeoffs study reports that applying a transform once before training does worse than clean training. Duplicating fixed augmented copies is therefore unlikely to substitute for fresh random views. We still include a "Copy" arm on the low-data text sets to check this.

Why not just use a linear ramp, as EfficientTrain does? A linear ramp is a special case of our family. EfficientTrain's gain of 0.6 points on ResNet-50 on ImageNet also mixes the ramp with its low-resolution start. We treat the linear ramp as a sub-arm, so the result shows whether the extra switch point ($\tau$ and $s_2$) is needed.

Why not just extend CurBench? CurBench's code handles sample selection and loss reweighting, not schedules of input transforms. It also does not equalize the baseline tuning budget. We borrow its data settings and cost reporting, but the matched-budget tuner and the schedule family are the parts that answer this question.

Proposed Experiments

On CIFAR-100 and CIFAR-10 we compare all image methods with WideResNet and PyramidNet backbones. We report error on the clean test set and on the shifted sets CIFAR-10.1 and CIFAR-10-C.

On CIFAR-10 with 40% flipped labels, CurBench's sample-level curriculum reports 86.92% against an 82.75% baseline. We test whether augmentation schedules add to, or replace, that gain.

On CIFAR-100-LT we run CUDA against the global-schedule ablation. On ImageNet-1K we run a smaller budget with ResNet-50, comparing the tuned fixed policy, the schedule family, EfficientTrain and RangeAugment. On ModelNet40 and ScanObjectNN we run PointNet++, DGCNN and SimpleView, each with and without learned augmentation. The question there is whether learned augmentation changes the ranking of the architectures.

Three results are decisive. The first is whether any learned policy beats the tuned schedule family on balanced CIFAR-100 by more than two standard errors at matched compute. The second is whether the advantage of adaptive methods over global schedules concentrates in the long-tailed and noisy-label settings, as predicted. The third is whether final-phase strength explains more variance in accuracy than area under the schedule across the grid. A decisive test of the third is to pair ramp-up and slingshot runs that share the same final-phase strength and check that their accuracies coincide.