Abstract
This paper establishes a formal mathematical link between ablated data augmentation (ADA) and regularization penalties. By analyzing linear models and extending the theory to deep neural networks via Integrated Gradients, the authors introduce the Contribution Covariance Penalty (CCP). They demonstrate that different ablation strategies, such as Mean Ablated Data Augmentation (MADA) and Inverted Input Dropout (IID), impose distinct constraints on model behavior, specifically targeting feature anticorrelation and weight scaling respectively.
Introduction
In deep learning, "ablated data augmentation" (ADA) techniques like Input Dropout, Cutout, and Random Erasing are widely used to prevent overfitting. While these methods are effective, their underlying mathematical mechanisms have remained largely intuitive. This research provides a rigorous framework to understand the specific "penalties" these techniques impose on a model's internal logic.
Theoretical Framework
Linear Theory
The authors first derive closed-form solutions for linear regression to show that ADA is equivalent to optimizing a standard loss function plus a specific penalty term:
- Mean Ablated Data Augmentation (MADA): Equivalent to Ordinary Least Squares (OLS) plus the Contribution Covariance Penalty (CCP). This penalty discourages features from having "anticorrelated" contributions, preventing them from canceling each other out to fit noise.
- Inverted Input Dropout (IID): Equivalent to OLS plus a Modified L2 Penalty (ML2P), which scales based on the feature's variance and mean.
The Neural Network Bridge
To apply these findings to non-linear deep models, the researchers use Integrated Gradients (IG). By treating feature attributions as local linear coefficients, they can measure the emergent CCP and ML2P in neural networks. This allows for an empirical validation of the theoretical penalties in complex architectures.
Empirical Validation
The theory was evaluated on structured datasets using feed-forward neural networks (MLPs) with up to 10 hidden layers. The results demonstrate how the strength of data augmentation influences the magnitude of the emergent penalties.
Key Insights and Results
- Distinct Regularization Effects: The study found that MADA and IID are not interchangeable. As shown in the empirical results, increasing the augmentation strength ($\lambda$) for MADA consistently decreases the CCP while potentially increasing the ML2P. Conversely, IID primarily targets the ML2P.
- Anticorrelation Targeting: The CCP is unique because it specifically targets pairs of features that work against each other. This prevents the model from assigning large, opposing weights to correlated features to overfit on noise.
- Persistence in Depth: The mathematical properties derived for simple linear models persist in deep, non-linear networks when viewed through the lens of feature attribution.
Limitations and Future Work
While the theory is general, the current empirical focus is on structured data and MLPs. Extending this analysis to Convolutional Neural Networks (CNNs) for computer vision or Transformers for natural language processing is a key next step. Additionally, the choice of baseline for Integrated Gradients can influence the measured penalties, suggesting a need for further investigation into baseline selection sensitivity.
Summary updated