The Penalty Imposed by Ablated Data Augmentation

Frederick Liu, Amir Najmi, Mukund SundararajanarXiv 2020 · arXiv

Abstract

This paper establishes a formal mathematical link between ablated data augmentation (ADA) and regularization penalties. By analyzing linear models and extending the theory to deep neural networks via Integrated Gradients, the authors introduce the Contribution Covariance Penalty (CCP). They demonstrate that different ablation strategies, such as Mean Ablated Data Augmentation (MADA) and Inverted Input Dropout (IID), impose distinct constraints on model behavior, specifically targeting feature anticorrelation and weight scaling respectively.

Introduction

In deep learning, "ablated data augmentation" (ADA) techniques like Input Dropout, Cutout, and Random Erasing are widely used to prevent overfitting. While these methods are effective, their underlying mathematical mechanisms have remained largely intuitive. This research provides a rigorous framework to understand the specific "penalties" these techniques impose on a model's internal logic.

Theoretical Framework

Linear Theory

The authors first derive closed-form solutions for linear regression to show that ADA is equivalent to optimizing a standard loss function plus a specific penalty term:

  • Mean Ablated Data Augmentation (MADA): Equivalent to Ordinary Least Squares (OLS) plus the Contribution Covariance Penalty (CCP). This penalty discourages features from having "anticorrelated" contributions, preventing them from canceling each other out to fit noise.
  • Inverted Input Dropout (IID): Equivalent to OLS plus a Modified L2 Penalty (ML2P), which scales based on the feature's variance and mean.

The Neural Network Bridge

To apply these findings to non-linear deep models, the researchers use Integrated Gradients (IG). By treating feature attributions as local linear coefficients, they can measure the emergent CCP and ML2P in neural networks. This allows for an empirical validation of the theoretical penalties in complex architectures.

Empirical Validation

The theory was evaluated on structured datasets using feed-forward neural networks (MLPs) with up to 10 hidden layers. The results demonstrate how the strength of data augmentation influences the magnitude of the emergent penalties.

Figure 1: Evolution of ML2P and CCP as a function of augmentation strength (λ) across different network depths.

Key Insights and Results

  • Distinct Regularization Effects: The study found that MADA and IID are not interchangeable. As shown in the empirical results, increasing the augmentation strength ($\lambda$) for MADA consistently decreases the CCP while potentially increasing the ML2P. Conversely, IID primarily targets the ML2P.
  • Anticorrelation Targeting: The CCP is unique because it specifically targets pairs of features that work against each other. This prevents the model from assigning large, opposing weights to correlated features to overfit on noise.
  • Persistence in Depth: The mathematical properties derived for simple linear models persist in deep, non-linear networks when viewed through the lens of feature attribution.
Figure 2: Consistency of emergent penalty behavior across different datasets, demonstrating the generalizability of the CCP and ML2P framework.

Limitations and Future Work

While the theory is general, the current empirical focus is on structured data and MLPs. Extending this analysis to Convolutional Neural Networks (CNNs) for computer vision or Transformers for natural language processing is a key next step. Additionally, the choice of baseline for Integrated Gradients can influence the measured penalties, suggesting a need for further investigation into baseline selection sensitivity.

Summary updated

Abstract

There is a set of data augmentation techniques that ablate parts of the input at random. These include input dropout, cutout, and random erasing. We term these techniques ablated data augmentation. Though these techniques seems similar in spirit and have shown success in improving model performance in a variety of domains, we do not yet have a mathematical understanding of the differences between these techniques like we do for other regularization techniques like L1 or L2. First, we study a formal model of mean ablated data augmentation and inverted dropout for linear regression. We prove that ablated data augmentation is equivalent to optimizing the ordinary least squares objective along with a penalty that we call the Contribution Covariance Penalty and inverted dropout, a more common implementation than dropout in popular frameworks, is equivalent to optimizing the ordinary least squares objective along with Modified L2. For deep networks, we demonstrate an empirical version of the result if we replace contributions with attributions and coefficients with average gradients, i.e., the Contribution Covariance Penalty and Modified L2 Penalty drop with the increase of the corresponding ablated data augmentation across a variety of networks.
BibTeX
@misc{liu2020penalty,
  author = {Frederick Liu and Amir Najmi and Mukund Sundararajan},
  title = {{The Penalty Imposed by Ablated Data Augmentation}},
  year = {2020},
  eprint = {2006.04769},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.04769},
}
Figure