The Variational Deficiency Bottleneck

Pradeep Kr. Banerjee, Guido Montúfar

TL;DR

This paper introduces the Variational Deficiency Bottleneck (VDB), a novel method for learning data representations by minimizing "information deficiency" instead of the traditional "information sufficiency." VDB offers a more principled way to approximate complex data channels with simpler ones and, importantly, leads to more robust models that generalize better to unseen data, particularly under distribution shifts.

What Problem Does This Paper Address?

The Information Bottleneck (IB) is a powerful framework for learning compressed representations of data that are minimal with respect to the input but sufficient for a target task (e.g., classification). However, computing the mutual information, which is central to IB, is often challenging. Existing methods use variational approximations, but this paper identifies a limitation: they focus on maximizing sufficiency. The authors propose a new approach that instead minimizes "information deficiency," which they argue provides a more robust way to approximate complex data channels and leads to better generalization. This deficiency concept has roots in decision theory and offers an operational interpretation related to the "risk gap" of decision problems.

Methods

The core of the paper is the introduction of the Deficiency Bottleneck (DB) and its variational counterpart, the Variational Deficiency Bottleneck (VDB).

  • Deficiency Bottleneck (DB): This method minimizes a new objective function that includes a regularization term on the mutual information between the representation and the input (minimality), and a term that minimizes the "deficiency" of a learned decoder with respect to the true, unknown channel.
  • Variational Deficiency Bottleneck (VDB): To make the DB practical, the authors derive a variational upper bound on the deficiency. This leads to an objective function that can be efficiently optimized using stochastic gradient descent, similar to the Variational Information Bottleneck (VIB).
  • Key Novelty: The VDB objective differs from VIB by having the expectation over Monte Carlo samples inside the logarithm, which allows for a more accurate approximation of the deficiency. The paper also explores different sampling strategies (M > 1) for the encoder during training, which can further improve performance.

Datasets

  • MNIST: Used for evaluating the VDB's ability to learn compressed representations and for visualizing latent spaces.
  • MNIST-C and CIFAR-10-C: These are benchmark datasets designed to test model robustness against common corruptions (noise, blur, weather, digital) at various severity levels. They are used to evaluate the out-of-distribution generalization capabilities of VDB-trained models.

Key Insights and Ideas

  • Deficiency as a Principle: Framing representation learning around "deficiency" offers a new perspective, focusing on how well a learned model can approximate a complex underlying process, rather than just how much information it retains about a target.
  • Operational Interpretation: The deficiency has a clear interpretation as the "risk gap" in decision problems, connecting information theory to classical statistical decision theory. This provides a theoretical grounding for why VDB might lead to better generalization.
  • VDB vs. VIB: The VDB objective is an upper bound to the VIB objective. When using a single sample (M=1) for the encoder, they coincide. However, using multiple samples (M>1) in VDB allows for a more accurate deficiency estimation and can lead to more compressed representations while maintaining sufficiency.
  • Robustness Advantage: The experiments strongly suggest that VDB training leads to models that are significantly more robust to distributional shifts (corruptions) compared to VIB, especially when using multiple encoder samples during training.

Key Results

  • Improved Minimality: VDB can achieve more compressed representations (lower I(Z;X)) than VIB for similar levels of sufficiency (I(Z;Y)). For example, with M=12, VDB required storing ~50 fewer bits of input information compared to M=1 while retaining the same output information.
  • Enhanced Robustness: VDB consistently outperforms VIB on the MNIST-C and CIFAR-10-C datasets in terms of mean corruption error (mCE) and relative mCE. For instance, on MNIST-C, VDB with M=6 and sequential training achieved significantly lower mCE and relative mCE than the baseline VIB (oneshot/M1).
  • Visualization: Latent space visualizations show that VDB can lead to better separation of classes, particularly when using the sequential training strategy.

Limitations and Future Work

  • Computational Cost: While VDB is a modification of VIB, using multiple samples (M>1) for the encoder during training increases computational cost.
  • Bias-Variance Trade-off: The paper speculates that a bias-variance trade-off might occur with VDB for high values of M, similar to Importance Weighted Autoencoders (IWAEs), and suggests this is an area for future investigation.
  • Unsupervised VDB: The paper mentions an unsupervised version of VDB that shares similarities with IWAE but notes that IWAE's objective cannot be decomposed for M>1, potentially limiting its ability to learn useful representations with bottleneck constraints. This suggests exploring unsupervised VDB further.

Summary updated

Abstract

We introduce a bottleneck method for learning data representations based on information deficiency, rather than the more traditional information sufficiency. A variational upper bound allows us to implement this method efficiently. The bound itself is bounded above by the variational information bottleneck objective, and the two methods coincide in the regime of single-shot Monte Carlo approximations. The notion of deficiency provides a principled way of approximating complicated channels by relatively simpler ones. We show that the deficiency of one channel with respect to another has an operational interpretation in terms of the optimal risk gap of decision problems, capturing classification as a special case. Experiments demonstrate that the deficiency bottleneck can provide advantages in terms of minimal sufficiency as measured by information bottleneck curves, while retaining robust test performance in classification tasks.
BibTeX
@inproceedings{banerjee2020variational,
  author = {Pradeep Kr. Banerjee and Guido Montúfar},
  title = {{The Variational Deficiency Bottleneck}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=BdZbYE5O8e5},
}
Figure