Regularising Deep Networks with DGMs

Matthew Willetts, Alexander Camuto, Stephen J. Roberts, Chris C. HolmesarXiv 2019 · OpenReview

TL;DR

This paper introduces "Pilot," a novel method for regularizing deep neural networks by learning a generative model of layer activations. By imputing these learned activations during training, Pilot acts as a form of data-aware dropout, leading to improved test accuracy and better calibrated uncertainty estimates compared to standard regularization techniques.

What Problem Does This Paper Address?

Deep neural networks are prone to overfitting, leading to poor generalization on unseen data. While various regularization techniques like dropout, batch normalization, and data augmentation exist, there's a continuous need for more effective methods. Specifically, this paper addresses the challenge of improving both the accuracy and the reliability of uncertainty estimates from deep networks. The authors propose that by understanding and modeling the internal workings (activations) of a network, we can better regularize it, akin to how data augmentation regularizes based on input data. They also highlight the importance of well-calibrated uncertainty for robust decision-making, a property often lacking in standard deep learning models.

Methods

The core of the proposed method, "Pilot," involves training a deep generative model (DGM) over the activations of a discriminative neural network. This DGM learns a probability distribution of these activations. During training, the discriminative network receives imputed activations drawn from this learned distribution, effectively acting as a form of "data-aware dropout" or a generalization of data augmentation to hidden layers.

Key aspects of the method include:

  • Deep Generative Model (DGM) over Activations: They leverage ideas from VAEAC (Variational Autoencoder with Arbitrary Conditioning) to learn a conditional distribution of a subset of activations given the rest. This is achieved by training a DGM that can impute missing activation values.
  • Joint Training Objective: The DGM and the discriminative classifier are trained simultaneously. The overall objective function combines the standard classification loss (e.g., cross-entropy) with a loss for training the DGM over activations.
  • Data Augmentation Generalization: The approach is framed as a generalization of data augmentation, extending it from the input layer to any hidden layer of the network.
  • Dropout Analogy: It's also viewed as a form of dropout where activations are imputed rather than simply set to zero.
  • Uncertainty Estimation: At test time, similar to MC Dropout, sampling from the DGM can provide uncertainty estimates for predictions.

Datasets

The paper uses the following standard datasets for training and evaluation:

  • CIFAR-10: A widely used dataset for image classification.
  • SVHN (Street View House Numbers): Another standard dataset for digit recognition.

The authors do not introduce new benchmarks but evaluate their method against existing regularization techniques on these established datasets.

Key Insights and Ideas

  • Activations as Data: The fundamental insight is treating the activations of a neural network's layers as data that can be modeled and augmented, similar to how input data is treated.
  • Data-Aware Regularization: By learning a distribution over activations, the regularization becomes "data-aware," meaning it's tailored to the specific patterns and distributions observed within the network's internal representations.
  • Uncertainty Calibration: The method demonstrably improves the calibration of uncertainty estimates, meaning the model's confidence in its predictions more accurately reflects its true accuracy. This is crucial for reliable decision-making in real-world applications.
  • Generalization of Augmentation/Dropout: Pilot unifies and generalizes existing regularization techniques like data augmentation and dropout by applying them to internal network states.

Key Results

The paper reports significant improvements across several metrics:

  • Increased Test Accuracy: Pilot, particularly the "Pilot a-aug" variant, consistently achieved higher test accuracy on both CIFAR-10 and SVHN compared to standard baselines, including dropout, L2 regularization, batch norm, and data augmentation. For CNNs on CIFAR-10, Pilot a-aug reached 70.1% accuracy, a substantial jump from the vanilla model's 63.0%.
  • Lower Test-Set Cross-Entropy (NLL): The method also resulted in lower negative log-likelihoods, indicating better predictive performance.
  • Improved Calibration: Pilot models showed significantly better calibration, as measured by Expected Calibration Error (ECE), outperforming baselines. For CNNs on CIFAR-10, Pilot a-aug achieved an ECE of 0.012%, a notable improvement.
  • Effective Uncertainty Estimation: When used for uncertainty estimation (PilotMc), the method provided comparable or better accuracy than MC Dropout, while also offering improved calibration in many cases.

Limitations and Future Work

  • Computational Cost: While the DGM is trained amortized, the overall training process for Pilot is more computationally intensive than standard methods due to the added DGM training.
  • Calibration of MC Variants: While Pilot generally improves calibration, the MC variants (PilotMc) sometimes showed a slight degradation in calibration (ECE) compared to their non-MC counterparts, which the authors attribute to the fixed variance of the decoder.
  • Future Work: The authors suggest further research into refining the calibration of MC models and exploring more sophisticated DGM architectures. They also propose that this approach could be viewed as a form of transfer learning or experience replay.

Summary updated

Abstract

We develop a new method for regularising neural networks. We learn a probability distribution over the activations of all layers of the model and then insert imputed values into the network during training. We obtain a posterior for an arbitrary subset of activations conditioned on the remainder. This is a generalisation of data augmentation to the hidden layers of a network, and a form of data-aware dropout. We demonstrate that our training method leads to higher test accuracy and lower test-set cross-entropy for neural networks trained on CIFAR-10 and SVHN compared to standard regularisation baselines: our approach leads to networks with better calibrated uncertainty over the class posteriors all the while delivering greater test-set accuracy.
BibTeX
@misc{willetts2019regularising,
  author = {Matthew Willetts and Alexander Camuto and Stephen J. Roberts and Chris C. Holmes},
  title = {{Regularising Deep Networks with DGMs}},
  year = {2019},
  eprint = {1909.11507},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.11507},
}
Figure