Information Dropout: Learning Optimal Representations Through Noisy Computation

Alessandro Achille, Stefano SoattoIEEE TPAMI 2018 · OpenReview

TL;DR

This paper introduces "Information Dropout," a novel regularization technique that enhances deep learning models by injecting adaptive multiplicative noise into network activations. This method, grounded in information theory, leads to more optimal and disentangled representations, outperforming traditional dropout, especially in capacity-limited models.

What Problem Does This Paper Address?

Deep learning models often struggle with learning "optimal representations" of data. While the standard cross-entropy loss encourages representations that are sufficient for a given task (i.e., contain enough information to perform the task), it doesn't explicitly enforce other crucial properties like minimality (discarding irrelevant information) and invariance (being robust to nuisance factors). This can lead to models that overfit to spurious correlations in the training data, resulting in poor generalization to unseen examples. Existing techniques like dropout offer some regularization but lack a strong theoretical foundation and adaptive capabilities. The authors aim to bridge this gap by developing a method that explicitly optimizes for these desirable representation properties.

Methods

The core contribution is Information Dropout, a generalization of dropout rooted in information theory and the Information Bottleneck (IB) principle. The IB principle seeks representations that are minimal sufficient for a task. The authors show that minimizing the IB Lagrangian can be achieved by adding a regularization term to the standard cross-entropy loss. This regularization term is equivalent to injecting multiplicative noise into the network's activations.

Key aspects of their method include:

  • Information Bottleneck Regularization: They reformulate the IB principle as a per-sample loss function and show that a regularization term, related to the KL divergence between the noisy activation distribution and a prior, can be added to the cross-entropy loss.
  • Adaptive Multiplicative Noise: Unlike standard dropout (which uses binary noise) or Variational Dropout (which uses Gaussian noise), Information Dropout allows the noise distribution to be adaptive to the input data and the network's structure. This is achieved by parameterizing the noise distribution (e.g., log-normal) with parameters that can vary per sample.
  • Disentanglement via Factorized Prior: They demonstrate that enforcing a factorized prior on the representation components naturally leads to disentangled representations, meaning that different underlying factors of variation in the data are captured by separate components of the representation.
  • Connection to VAEs: When the task is reconstruction (i.e., predicting the input itself), Information Dropout with a specific choice of parameters reduces to the loss function of a Variational Autoencoder (VAE). This establishes a theoretical link between information theory and VAEs.

Datasets

The paper uses the following datasets:

  • Cluttered MNIST: A custom dataset designed to test robustness to occlusion and nuisance factors, consisting of MNIST digits with distractors.
  • MNIST: A standard dataset for image classification.
  • CIFAR-10: A standard dataset for image classification.
  • Occluded CIFAR: A newly created dataset where CIFAR-10 images are occluded by MNIST digits to specifically test invariance to occlusions.

Key Insights and Ideas

  • Noisy Computation for Optimal Representations: Counter-intuitively, injecting multiplicative noise into network activations, rather than hindering computation, actually helps in learning more optimal representations by enforcing minimality and invariance.
  • Information Theory as a Unifying Framework: The paper provides a strong theoretical grounding for dropout-like methods by connecting them to the Information Bottleneck principle, offering a unified perspective on various regularization techniques.
  • Adaptive Noise is Crucial: The ability of Information Dropout to adapt the noise level to individual data samples and network capacity is key to its superior performance, especially in smaller models where aggressive noise reduction by standard dropout can be detrimental.
  • Disentanglement as a Natural Consequence: The paper shows that disentanglement of latent factors is not an ad-hoc addition but can be achieved naturally by enforcing a factorized prior in the Information Bottleneck framework.
  • VAE as a Special Case: The connection to VAEs highlights how Information Dropout can be seen as a more general framework that encompasses existing powerful generative models.

Key Results

  • Improved Generalization: Information Dropout consistently achieved comparable or better generalization performance than binary dropout on MNIST and CIFAR-10 datasets, particularly on smaller models.
  • Robustness to Nuisances: Experiments on Cluttered MNIST and Occluded CIFAR demonstrated that Information Dropout significantly improves invariance to nuisance factors (like occlusions), making the learned representations more robust.
  • Disentanglement: The paper shows that increasing the Information Bottleneck parameter $\beta$ leads to lower total correlation in the representations, indicating better disentanglement, and this correlates with improved test error.
  • Adaptive Noise Benefits: Information Dropout outperformed a baseline using constant log-normal noise, confirming the advantage of adaptive noise.
  • VAE Equivalence: When applied to reconstruction tasks, Information Dropout yielded results comparable to standard VAEs, validating the theoretical connection.

Limitations and Future Work

  • Approximation Guarantee: The paper acknowledges that Information Dropout provides an approximation to the optimal IB representation, not a guaranteed optimum, as it restricts the family of distributions considered. Iterative application of the method is suggested for better approximations.
  • Choice of Prior: The performance of Information Dropout depends on the choice of the prior distribution for the noise. While they propose log-normal and log-uniform priors for common activation functions, exploring more sophisticated priors could be a future direction.
  • Computational Cost: While Information Dropout can be more efficient for limited capacity networks, the overhead of adaptive noise computation might be a consideration for very large-scale, high-throughput applications.
  • Further Exploration of Links: The authors suggest that further work is needed to fully explore the deep connections between information theory, representation learning, variational inference, and nuisance invariance.

The paper lays a strong theoretical and empirical foundation for Information Dropout, presenting it as a principled and effective method for learning optimal, robust, and disentangled representations in deep neural networks.

Summary updated

Abstract

The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties. We show that this can be solved by adding a regularization term, which is in turn related to injecting multiplicative noise in the activations of a Deep Neural Network, a special case of which is the common practice of dropout. We show that our regularized loss function can be efficiently minimized using Information Dropout, a generalization of dropout rooted in information theoretic principles that automatically adapts to the data and can better exploit architectures of limited capacity. When the task is the reconstruction of the input, we show that our loss function yields a Variational Autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Finally, we prove that we can promote the creation of optimal disentangled representations simply by enforcing a factorized prior, a fact that has been observed empirically in recent work. Our experiments validate the theoretical intuitions behind our method, and we find that Information Dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.
BibTeX
@article{achille2018information,
  author = {Alessandro Achille and Stefano Soatto},
  title = {{Information Dropout: Learning Optimal Representations Through Noisy Computation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2018},
  url = {https://openreview.net/forum?id=LmXIK7VETK},
}
Figure