MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes

Henry Gouk, Bernhard Pfahringer, Eibe Frank, Michael J. Cree

TL;DR

MaxGain is a novel regularization technique that prevents neural networks from overfitting by limiting the "gain" (the ratio of output to input magnitudes) of each layer. By providing a computationally efficient, empirical alternative to Lipschitz continuity constraints, it improves generalization across various architectures and works effectively alongside standard methods like Dropout and Batch Normalization.

What Problem Does This Paper Address?

Deep neural networks are notorious for overfitting due to their massive number of parameters. One way to combat this is through Lipschitz continuity—ensuring that small changes in the input don't lead to wild swings in the output. While theoretically sound, calculating and enforcing the "best" Lipschitz constant for complex layers (like convolutions) is mathematically difficult and computationally expensive.

Furthermore, most Lipschitz methods assume the network must be "smooth" across the entire possible input space. However, the manifold hypothesis suggests that real-world data (like images) actually lives on a low-dimensional "manifold" within that high-dimensional space. The authors argue we don't need the network to be smooth everywhere—just on the parts of the space where data actually exists.

Methods

The authors introduce MaxGain, an empirical analogue to the Lipschitz constant.

  1. Defining Gain: For any layer with weight matrix $W$ and input $x$, the gain is defined as the ratio of the norms: $Gain(W, x) = \frac{||Wx||}{||x||}$.
  2. The Constraint: The goal is to ensure that for every layer, the maximum gain observed on the training data does not exceed a user-defined hyperparameter, $\gamma$.
  3. Stochastic Projection: Rather than calculating the gain for the entire dataset (which is impossible for large sets), the algorithm uses a "stochastic projection." After every gradient update, the weights are checked against the current minibatch. If the maximum gain in the batch exceeds $\gamma$, the weight matrix is rescaled: $$W_{new} = \frac{\gamma}{\max(1, \text{estimated\_gain})} W_{old}$$
  4. Batch Norm Integration: The authors provide a specific modification to handle the interaction between MaxGain and Batch Normalization, using running averages of statistics to ensure stable convergence.

Datasets

The paper evaluates MaxGain on several standard benchmarks:

  • CIFAR-10 & CIFAR-100: Standard image classification tasks.
  • SVHN (Street View House Numbers): A dataset where training and test distributions differ slightly.
  • SINS-10 (Scaled ImageNet Subset): A novel dataset introduced in this paper. It consists of 100,000 images (96x96 pixels) across 10 classes, divided into 10 folds. It is specifically designed to make statistical significance testing easier for researchers using convolutional networks.

Key Insights and Ideas

  • Stopping "Confidence Inflation": The authors observe an "exploding activation" effect in unregularized networks. Once a network learns to classify data correctly, the optimizer often tries to minimize loss further by simply making the weights larger to increase the "confidence" (logit magnitude) of the prediction. MaxGain prevents this artificial inflation, forcing the network to learn more robust features instead.
  • Empirical vs. Theoretical: By focusing on the observed gain on actual data rather than the theoretical maximum of the weight matrix (the spectral norm), MaxGain is much easier to implement for convolutional layers without needing complex power iteration methods.
  • The Manifold Advantage: Because it only constrains the network where data actually appears, it is potentially less restrictive than global Lipschitz constraints, allowing the model to maintain higher capacity where it matters.

Key Results

The experiments show that MaxGain is a "team player" that often improves performance when added to existing setups:

  • CIFAR-10: Using a VGG-19 network, MaxGain + Batchnorm achieved 91.76% accuracy, outperforming Batchnorm alone (90.80%) and Batchnorm + Dropout (90.90%).
  • CIFAR-100: On a Wide Residual Network, the combination of MaxGain, Batchnorm, and Dropout yielded the best result at 76.44%.
  • SVHN: MaxGain helped reach 97.98% accuracy when combined with other techniques.
  • Generalization: The authors confirmed that constraining gain on the training set successfully leads to lower, more stable gains on the test set, proving the method's effectiveness as a regularizer.

Limitations and Future Work

  • Norm Selection: While the theory supports any $p$-norm, this paper focuses exclusively on the $\ell_2$ norm. Exploring $\ell_1$ or $\ell_\infty$ norms remains for future study.
  • Hyperparameter Tuning: Like many regularizers, MaxGain introduces a new hyperparameter ($\gamma$). While the authors show the model is sensitive to $\gamma$ in a predictable way (low $\gamma$ leads to underfitting, high $\gamma$ to overfitting), it still requires tuning for optimal results.
  • Stochastic Approximation: Because the projection is based on minibatches, the constraint is only approximately satisfied over the whole training set. Future work could investigate if more precise estimation methods yield better results.

Summary updated

Abstract

Effective regularisation of neural networks is essential to combat overfitting due to the large number of parameters involved. We present an empirical analogue to the Lipschitz constant of a feed-forward neural network, which we refer to as the maximum gain. We hypothesise that constraining the gain of a network will have a regularising effect, similar to how constraining the Lipschitz constant of a network has been shown to improve generalisation. A simple algorithm is provided that involves rescaling the weight matrix of each layer after each parameter update. We conduct a series of studies on common benchmark datasets, and also a novel dataset that we introduce to enable easier significance testing for experiments using convolutional networks. Performance on these datasets compares favourably with other common regularisation techniques. Data related to this paper is available at: https://www.cs.waikato.ac.nz/~ml/sins10/ .
BibTeX
@inproceedings{gouk2018maxgain,
  author = {Henry Gouk and Bernhard Pfahringer and Eibe Frank and Michael J. Cree},
  title = {{MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes}},
  booktitle = {ECML/PKDD},
  year = {2018},
  url = {https://openreview.net/forum?id=E7V8XZmPb8C},
}
Figure