Learning Invariances in Neural Networks from Training Data

Gregory W. Benton, Marc Finzi, Pavel Izmailov, Andrew Gordon WilsonNeurIPS 2020 · OpenReview

Introduction

Augerino is a practical framework that allows neural networks to automatically "learn" which symmetries—such as rotation, scaling, or translation—exist in a dataset using only the training data. By optimizing a distribution of data augmentations alongside the model's weights, Augerino discovers the optimal level of invariance needed to generalize better, entirely removing the need for manual tuning or a separate validation set.

The Challenge of Hard-Coded Symmetries

In deep learning, we know that certain symmetries are incredibly powerful. For instance, the fact that a cat remains a cat even if shifted by a few pixels (translation equivariance) is hard-coded into the very architecture of Convolutional Neural Networks (CNNs). However, for many other symmetries like rotation, scale, or shear, we often do not know a priori if the model should be invariant to them, or to what extent.

Consider the digit '6'. It is rotationally invariant up to a slight angle, but if you rotate it 180 degrees, it becomes a '9'. Hard-coding total rotational invariance would be a catastrophic mistake for this dataset.

MNIST digits demonstrate the danger of hard-coded rotational invariance (e.g., a rotated '6' becomes a '9').

Traditionally, researchers solve this by manually picking data augmentations or using expensive search algorithms (like AutoAugment) that require a hold-out validation set. The authors of this paper ask a fundamental question: Can a model discover these symmetries automatically from the training data alone?

How Augerino Works

The authors introduce Augerino, a method that treats data augmentation not as a fixed preprocessing step, but as a learnable distribution.

The Augerino architecture: input images undergo parameterized affine transformations before being passed through the network and averaged.
  1. Augmentation Averaging: Instead of a standard model $f(x)$, Augerino uses an averaged model $\bar{f}(x) = \mathbb{E}_{g \sim \mu_\theta} f(gx)$, where $g$ is a transformation sampled from a distribution $\mu$ parameterized by $\theta$.
  2. The Reparameterization Trick: To make the augmentation parameters $\theta$ trainable via backpropagation, they use the reparameterization trick (similar to Variational Autoencoders). This allows gradients to flow from the loss function directly into the parameters that control the range of rotations, scales, and other transformations.
  3. Lie Groups for Affine Transforms: They parameterize a wide space of transformations (translation, rotation, scaling, and shear) using Lie Algebra generators and the exponential map. This provides a smooth, differentiable way to explore the space of affine transformations.
  4. Regularized Loss: Because training loss alone favors flexibility over constraints, the authors add a small regularization term $R(\theta)$. This term "pushes" the distribution to be as wide as possible. The model will expand its invariance until it hits a point where the training loss starts to increase (e.g., when a '6' starts looking like a '9'), at which point the gradient from the loss halts the expansion.

Key Insights

  • The "Flat" Loss Landscape: The most striking insight is that for augmentations that are "correct" for the data, the training loss is essentially flat. Augerino uses regularization to slide along this flat region to find the maximum possible invariance, stopping only when the loss spikes at the boundary of "harmful" transformations.
  • Invariance vs. Equivariance: Augerino isn't just for classification (invariance). By averaging inversely transformed outputs, it can learn equivariance for tasks like segmentation, where the output mask must rotate along with the input image.
  • Generalization via Flatness: Learning invariances leads the model toward "flatter" minima in the weight space. In deep learning theory, flatter minima are strongly correlated with better generalization on unseen data.
  • Simplicity: Unlike previous "AutoML" approaches for augmentation, Augerino doesn't need a reinforcement learning loop or a validation set; the discovery happens seamlessly during standard training.

Datasets and Results

The paper evaluates Augerino across a diverse set of tasks, proving its versatility:

  • CIFAR-10: Augerino achieved 93.81% accuracy, outperforming models with no augmentation (90.6%) and matching or exceeding sophisticated methods like Fast AutoAugment (92.65%), despite being much simpler to implement.
  • rotMNIST: It recovered nearly full rotational invariance, achieving 98.9% test accuracy using only training data to discover the symmetry.
  • Molecular Data (QM9): On the HOMO energy prediction task (a regression problem), Augerino significantly reduced Mean Absolute Error (MAE) from 52.7 to 38.3, proving it works well beyond just 2D images.
  • Interpretability: Because the parameters $\theta$ correspond to physical transformations, researchers can inspect the learned values to see exactly what symmetries the model discovered (e.g., confirming the model learned to be invariant to exactly $\pm 15$ degrees of rotation).

Limitations and Future Work

While powerful, Augerino does have limitations. It requires the transformations to be differentiable. While this covers affine transforms and color shifts, it is harder to apply to discrete augmentations like "cutout" or non-differentiable image filters. Additionally, while the authors note the choice of the regularization weight $\lambda$ is largely forgiving, setting it too high can cause training instability.

Future work could explore learning more complex, non-linear symmetries or applying the framework to different modalities like audio or natural language, further automating the discovery of inductive biases in deep learning.

Summary updated

Abstract

Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present in the data, or to what extent a model should be invariant to a given augmentation. We show how to learn invariances by parameterizing a distribution over augmentations and optimizing the training loss simultaneously with respect to the network parameters and augmentation parameters. With this simple procedure we can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations, on training data alone. We show our approach is competitive with methods that are specialized to each task with the appropriate hard-coded invariances, without providing any prior knowledge of which invariance is needed.
BibTeX
@inproceedings{benton2020learning,
  author = {Gregory W. Benton and Marc Finzi and Pavel Izmailov and Andrew Gordon Wilson},
  title = {{Learning Invariances in Neural Networks from Training Data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=2o1e86t77vm},
}
Figure