Abstract
Data augmentation is a powerful technique to improve performance in applications such as image and text classification tasks. Yet, there is little rigorous understanding of why and how various augmentations work. In this work, we consider a family of linear transformations and study their effects on the ridge estimator in an over-parametrized linear regression setting. First, we show that transformations that preserve the labels of the data can improve estimation by enlarging the span of the training data. Second, we show that transformations that mix data can improve estimation by playing a regularization effect. Finally, we validate our theoretical insights on MNIST. Based on the insights, we propose an augmentation scheme that searches over the space of transformations by how uncertain the model is about the transformed data. We validate our proposed scheme on image and text datasets. For example, our method outperforms random sampling methods by 1.24% on CIFAR-100 using Wide-ResNet-28-10. Furthermore, we achieve comparable accuracy to the SoTA Adversarial AutoAugment on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.
Introduction
Data augmentation (DA) is a cornerstone of modern deep learning, yet our understanding of why specific augmentations work remains largely anecdotal. While "folklore wisdom" suggests that more data is always better, different types of augmentations—such as rotating an image versus mixing two images—have fundamentally different effects on a model's learning process. This paper addresses the gap between simple noise-based models and the complex, multi-step augmentation pipelines used in practice, seeking to quantify how different categories of transformations affect the bias and variance of a model.
Theoretical Framework
The authors analyze these effects using an over-parametrized linear regression model ($p > n$) and a ridge estimator. They categorize transformations into three primary groups:
- Label-invariant: Transformations that preserve the original label (e.g., rotation, horizontal flip).
- Label-mixing: Transformations that combine two data points and their labels (e.g., Mixup).
- Compositions: Sequences of transformations.
Uncertainty-Based Sampling
Building on their theoretical findings, the authors propose a new selection strategy. Instead of picking augmentations at random (like RandAugment), their algorithm generates several augmented versions of a single data point, computes the loss for each using the current model, and selects the versions with the highest loss—those the model is most "uncertain" about—for the actual training step.
Key Insights: Bias vs. Variance
The study reveals a fundamental distinction in how augmentations improve generalization:
- Label-Invariant Transforms Expand the Span: In over-parametrized settings, training data only covers a small subspace of the feature space. Label-invariant transforms effectively "rotate" data points into previously unseen dimensions, allowing the model to estimate parameters in those directions and reducing the intrinsic error (bias).
- Mixup as a Regularizer: Mixup does not add new information because mixed samples stay within the span of existing training data. Instead, it reweights the training data relative to the $\ell_2$ regularization term, reducing the instability (variance) of the model.
As shown in the table above, different augmentations impact the Error Score (bias) and Instability Score (variance) differently, with transformations like Rotation achieving high accuracy by significantly lowering both metrics compared to the baseline.
Key Results
The proposed uncertainty-based sampling scheme was validated across standard benchmarks including CIFAR-10, CIFAR-100, SVHN, and ImageNet.
Key performance highlights include:
- Superior Accuracy: The method outperforms RandAugment by 0.59% on CIFAR-10 and 1.24% on CIFAR-100.
- Efficiency: It achieves accuracy comparable to state-of-the-art Adversarial AutoAugment (Adv. AA) while being at least 5x cheaper to train, as it avoids the need for a separate adversarial controller network.
- Cross-Domain Success: The approach also improved performance on the IMDb sentiment task, demonstrating that the theory applies to text classification beyond image pixels.
Conclusion
By providing a rigorous framework for data augmentation, this work moves beyond heuristic selection toward a principled understanding of bias and variance reduction. While the current theory is grounded in linear models, the empirical success on deep neural networks suggests these insights are broadly applicable. Future work may extend these proofs to non-linear regimes and explore the automatic discovery of base transformations.
Summary updated