Automated data augmentation searches for a recipe of image transformations (rotate this much, shift colors that much) that makes a classifier generalize better. Because the search is expensive (AutoAugment used 5,000 GPU hours on CIFAR-10 and 15,000 on ImageNet), practitioners usually reuse a recipe found on one dataset for another. That works on natural-image targets. It fails, sometimes badly, on targets whose labels depend on exactly what the recipe scrambles: coat patterns on individual animals, color in infrared trap-camera images, the orientation that separates a "6" from a "9". This proposal makes that failure predictable and cheap to fix. Before any target training, we measure, for every transformation and strength in the borrowed recipe, how much label information still survives on the target data. Then we cut or cap the transformations that destroy it.
The measurement is a linear probe, meaning a single linear classifier fit on top of a frozen pretrained image encoder. For each transformation and strength, we train the probe on transformed target images and test it on held-out images transformed the same way. If labels stay recoverable, the transformation only changes how images look. If they do not, it erases information the task needs. Hundreds of such probes cost minutes. The result is a per-transformation curve of how much the target can tolerate, which turns two folklore problems into one measurable quantity: "which augmentations violate this domain" and "where is the narrow good range for each strength".
Core Claim
A transferred augmentation policy causes negative transfer mainly when it puts probability mass on transformation–strength pairs that destroy target label information. This can be detected without target training. Define, for transformation $o$ at strength $m$, the retention
$$R(o,m) = \frac{\mathrm{acc}_{o,m\to o,m}}{\mathrm{acc}_{\mathrm{clean}\to\mathrm{clean}}},$$
where $\mathrm{acc}_{a\to b}$ is the held-out accuracy of a linear probe on frozen features, trained on target images processed by $a$ and tested on images processed by $b$. Training and testing under the same transformation is the important design choice. A clean-trained probe tested on transformed images, the usual "accuracy drop" diagnostic, penalizes any shift in appearance: solarizing a flower changes how it looks without removing what makes it a flower. Retention penalizes only lost information: flipping a "6" makes it indistinguishable from an unflipped "9", so no probe can recover the label. The editing rule keeps each operation only up to the largest strength with $R(o,m)\ge\tau$ and removes operations that fall below $\tau$ even at their weakest setting. The policy-level score $S=\sum_{o,m}\pi(o,m)R(o,m)$, where $\pi$ is the transferred policy's sampling probability over pairs, should predict whether unedited transfer will beat a no-augmentation baseline.
What is new is not the idea of evaluating augmentations, but a training-free, information-based editor for transferred policies. It sits between using a borrowed policy blindly and re-searching on the target. It is evaluated on whether it predicts and prevents the documented negative-transfer cases.
Relation to Prior Work
The motivating evidence comes from two sources. AutoAugment AutoAugment defined the policy format we edit: sub-policies of paired PIL operations, each with a probability and a strength. It reported that ImageNet policies transfer to Flowers, Cars, Aircraft and Pets. The feral-cat study Exploring Automated Data Augmentation Approaches… Feral Cat Classification found the opposite on a constrained domain. RandAugment fell below the 26.03% no-augmentation baseline, and AutoDO reached 24.88%, while target-searched Faster AA reached 54.22%. Our claim reconciles the two: the editor should leave natural-image transfers nearly untouched and make large cuts on the cat data. If it edits Flowers heavily and still helps, that is a sign the retention measure is miscalibrated.
Four lines of work set the bar for adaptive methods. AdaAug AdaAug trains a policy network that outputs per-image probabilities and strengths, and it rediscovered flip suppression for 6/9 on SVHN. MetaAugment MetaAugment learns to down-weight harmful augmented samples at 3x to 4.5x training cost. Sample-Aware RandAugment SRA lowers strength on hard samples. All three work during target training with bilevel or reweighting machinery. Our editor runs once, before training, and needs no outer optimization loop. That makes it complementary: its output can initialize or constrain any of them.
Domain constraints are currently written by hand. AADG AADG restricted its retinal search to color operations and banned shearing to protect microvasculature. Re-ID pretraining Unsupervised Pre-Training for Person Re-Identification switched off color jitter. Our method tests whether such rules can be recovered from target data. Two prior results constrain the design. ST-SSAD Self-Tuning Self-Supervised Anomaly Detection tunes augmentation hyperparameters with an unlabeled validation loss but depends on handcrafted differentiable surrogates. We need a small labeled target set instead, and we state that requirement openly. RandConv Random Convolutions shows that texture-stripping helps shape tasks and hurts texture tasks. That is exactly the dependence on the target domain that retention is meant to expose. Finally, the observation that larger models want stronger magnitudes Cubuk et al. means our caps should be read as ceilings, not targets.
Research Program
We first build the probe suite: the 14-odd AutoAugment operations, each at ten strength levels, applied to a labeled target subset. Features come from frozen encoders. The main risk is that the encoder itself ignores the property being destroyed; a color-invariant self-supervised encoder cannot register that color was erased. We therefore compare a supervised ImageNet ResNet-50, a self-supervised encoder, and features concatenated from several depths, motivated by the finding that invariance differs by layer Rethinking the Augmentation Module in Contrastive Learning. Second, we apply the editing rule to transferred policies (AutoAugment ImageNet and CIFAR policies, and the PBA schedule Population Based Augmentation) and renormalize the remaining probabilities. Third, we compute retention per class pair from probe confusion matrices to produce class-conditional caps. This tests whether a training-free probe recovers AdaAug's finding that flips are safe for 0/1/8 but not 6/9. Fourth, we study sensitivity to the threshold $\tau$ itself. A method meant to tame hyperparameter fragility fails on its own terms if its threshold is fragile.
Baselines
The naive reference is standard crop-and-flip with no learned policy, plus the 26.03% no-augmentation figure reported for the feral-cat data. The direct competitor for our question is the unedited transferred policy: the same AutoAugment or PBA policy applied as-is. Any gain we claim is measured against it on identical data, architecture, schedule and seeds.
The strongest cheap competitor is leave-one-operation-out pruning. It runs one short target training run per operation with that operation removed and drops those whose removal helps. We give it the same wall-clock budget as our probes plus one extra full run. It is strong because it measures actual training benefit rather than a proxy. We beat it only if we match its final accuracy at a fraction of its cost, or exceed it when the budget limits it to very short runs.
A second competitor is RandAugment with its two knobs (number of operations and strength) grid-searched on target validation data under the same budget. This tests whether simply lowering global strength fixes negative transfer without per-operation editing.
The strongest existing method is full target search. On feral cats, the reported number is Faster AA at 54.22% top-1 after 39.5 hours Feral Cat study, with DADA at 50.90% in 29.1 hours. On Flowers, the reported number is AdaAug's transferred-policy error of 3.63% against RandAugment's 4.28% AdaAug. Success means landing within two points of Faster AA on cats with under an hour of editing compute, and at least matching AdaAug's 3.63% on Flowers in the same Wide-ResNet setting. For Aircraft, Cars and Pets, no reported result found on Lacuna for AutoAugment transfer, so we rerun the unedited transfer ourselves.
Three ablations isolate the key ingredient. The first replaces retention with the clean-trained accuracy-drop score; if it works as well, the "recoverable information" argument is wrong. The second removes the same number of operations at random. The third separates the effect of capping strength from the effect of removing operations.
Why not just…?
Why not just tune RandAugment on the target? It has two knobs and costs a small grid. However, the feral-cat result shows RandAugment below the no-augmentation baseline, and lowering global strength also weakens the harmless operations that the 54.22% searched policy relies on. Our matched-budget grid search will settle this directly. If a tuned RandAugment matches the edited policy, we will report that per-operation editing is unnecessary.
Why not just ablate each operation with short training runs? This is the real threat, because it measures what we only approximate. Its cost grows as operations times runs, and short runs may misjudge operations whose benefit appears late, as PBA's ramping schedules suggest. We run it as a baseline under equal compute. Agreement between its rankings and retention rankings is itself a validation result. If it clearly wins at equal cost, the probe adds nothing.
Why not just run DADA or AdaAug on the target? DADA reports 1.3 GPU hours on ImageNet, but it took 29.1 hours on the cat data and still trailed Faster AA. AdaAug carries memory overhead from evaluating every augmentation path. The editor is not a rival to these methods. If edited policies make good warm starts for them, that is a positive outcome, and we test it by initializing AdaAug from the edited policy.
Why not just have experts switch off color or rotation? Binary rules miss graded optima. The visible-infrared Re-ID work Gray Augmentation Exploration shows grayscale is needed but only at a calibrated rate. We compare against expert rules taken from AADG and Re-ID practice and check whether the caps we find reproduce them.
Proposed Experiments
We use Wide-ResNet-28-10 at 32×32 for SVHN, CIFAR-100 sources and the feral-cat data, if its release permits. If it does not, we rely only on its reported numbers. ResNet-50 is used for Flowers, Aircraft, Cars, Pets and Caltech-101, with three seeds each.
The first decisive check is prediction. Across all source→target pairs, the Spearman correlation between the policy score $S$ and the measured transfer gain over no augmentation should be strongly positive. Specifically, $S$ should flag the cat transfer as negative before any target training.
The second is mechanism. On SVHN, class-pair retention should suppress flips for 6/9 and not for 0/1/8, matching AdaAug. On infrared cat images, retention should flag solarization, matching the feral-cat study's analysis.
The third is no harm. On Flowers and the other natural fine-grained targets, the editor should change little and should not reduce accuracy.
The fourth is threshold stability. Accuracy should vary little for $\tau$ between 0.8 and 0.95. A sharp peak would mean we have only moved the sensitivity problem to a new knob.
A stretch test applies the probe to AADG's retinal setting, measured by Dice score, to see whether it rejects shear without being told to.
References
- AutoAugment: Learning Augmentation Strategies From Data.
- Exploring Automated Data Augmentation Approaches for Deep Learning: A Case Study of Individual Feral Cat Classification
- AdaAug: Learning Class- and Instance-adaptive Data Augmentation Policies
- MetaAugment: Sample-Aware Data Augmentation Policy Learning
- Sample-Aware RandAugment: Search-Free Automatic Data Augmentation for Effective Image Recognition
- AADG: Automatic Augmentation for Domain Generalization on Retinal Image Segmentation
- Unsupervised Pre-Training for Person Re-Identification
- Self-Tuning Self-Supervised Anomaly Detection
- Robust and Generalizable Visual Representation Learning via Random Convolutions
- G-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR
- Rethinking the Augmentation Module in Contrastive Learning: Learning Hierarchical Augmentation Invariance with Expanded Views
- Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
- Gray Augmentation Exploration with All-Modality Center-Triplet Loss for Visible-Infrared Person Re-Identification