Abstract
Data augmentation has been widely applied as an effective methodology to improve generalization in particular when training deep neural networks. Recently, researchers proposed a few intensive data augmentation techniques, which indeed improved accuracy, yet we notice that these methods augment data have also caused a considerable gap between clean and augmented data. In this paper, we revisit this problem from an analytical perspective, for which we estimate the upper-bound of expected risk using two terms, namely, empirical risk and generalization error, respectively. We develop an understanding of data augmentation as regularization, which highlights the major features. As a result, data augmentation significantly reduces the generalization error, but meanwhile leads to a slightly higher empirical risk. On the assumption that data augmentation helps models converge to a better region, the model can benefit from a lower empirical risk achieved by a simple method, i.e., using less-augmented data to refine the model trained on fully-augmented data. Our approach achieves consistent accuracy gain on a few standard image classification benchmarks, and the gain transfers to object detection.
Introduction
Modern deep learning architectures rely heavily on intensive data augmentation techniques such as Mixup, Cutmix, and AutoAugment to prevent overfitting and enhance generalization. While these methods are effective, they introduce a significant "distribution gap." Because the training process optimizes for highly distorted images, the resulting model often exhibits a surprisingly high empirical risk (training loss) when evaluated on the clean data distribution seen during testing. This paper investigates whether closing this distribution gap at the end of training can further improve model performance.
The Refinement Strategy
To address the distribution gap, we propose Refined Data Augmentation, a two-stage training framework that balances the benefits of intensive augmentation with the precision required for clean data.
- Stage 1 (Exploration): The network is trained for the majority of epochs using intensive data augmentation. This phase encourages the model to navigate toward a robust, generalized region in the loss landscape, effectively regularizing "minor features" (e.g., background textures) and focusing on "major features" (e.g., object shapes).
- Stage 2 (Refinement): In the final epochs, intensive augmentation is disabled in favor of standard, lightweight transformations (e.g., horizontal flips and random crops). By maintaining a low learning rate, the model is allowed to settle into a local minimum optimized specifically for the clean data distribution.
As shown in the loss curves, the transition to the refinement phase (indicated by the dashed vertical line) results in a sharp decrease in the training loss on clean data, effectively bridging the distribution gap created by intensive augmentation.
Analytical Perspective
The efficacy of refinement can be understood through the lens of feature representation. Using Singular Value Decomposition (SVD) on the network's penultimate layer, we observe that intensive augmentation significantly distorts the distribution of singular values compared to standard training. This distortion can reduce the discriminative power of the final classification layer.
The refinement phase pulls these singular values back toward a more natural distribution. This "restoration" of the feature space allows the model to better distinguish between classes on clean data, explaining the observed accuracy gains.
Experimental Results
The proposed refinement strategy was validated across multiple benchmarks:
- CIFAR-100: Refinement yielded consistent improvements. For instance, a PreActResNet-18 using Mixup improved from 78.95% to 80.85% accuracy.
- ImageNet: A ResNet-50 trained with Mixup achieved a Top-1 accuracy of 77.69% after refinement, up from 77.47%.
- Object Detection: Intensive augmentation often degrades performance in downstream tasks like object detection due to spatial distortions. On PascalVOC 2007, a ResNet-50 backbone trained with Mixup dropped to 67.4 mAP from a 72.0 mAP baseline. However, applying refinement boosted performance to 73.3 mAP, demonstrating that the model can relearn crucial spatial features.
Conclusion
Refined Data Augmentation offers a simple, zero-overhead solution to the distribution gap problem. By separating the training process into a global exploration phase and a local refinement phase, models can achieve superior generalization without sacrificing precision on the target distribution. Future work may involve integrating this mechanism into automated machine learning (AutoML) pipelines or exploring specialized learning rate schedules for the refinement phase.
Summary updated