Understanding the Behaviour of the Empirical Cross-Entropy Beyond the Training Distribution

Matías Vera, Pablo Piantanida, Leonardo Rey VegaarXiv 2019 · OpenReview

Introduction

In standard machine learning theory, models are typically assumed to operate in an "i.i.d." (independent and identically distributed) environment, where training and testing data share the same underlying distribution. However, real-world applications frequently encounter distribution shifts—such as a model trained on daytime photos being tested on nighttime images. This shift creates a "testing gap": the difference between a model's empirical performance on a finite set of new samples and its true expected performance on that new distribution.

This paper introduces a novel information-theoretic bound for this testing gap, demonstrating that the mutual information between inputs and their internal representations is a primary predictor of how well a model generalizes under distribution shifts.

Theoretical Framework

The authors model a classifier as a two-part system: an encoder that maps input data $X$ to a representation $U$, and a decoder that maps $U$ to a predicted label $\hat{Y}$. The core contribution is a rigorous mathematical bound on the testing gap that moves beyond traditional VC-dimension or Rademacher complexity. This bound is constructed from three key components:

  1. Mutual Information $I(X; U)$: Quantifies how much information the internal representation retains about the input.
  2. Hellinger Distance: Measures "decoder efficiency" by comparing the chosen decoder to an ideal decoder for the given representation.
  3. Discretization: Employs robust optimization techniques to partition the input space, making information-theoretic tools applicable to continuous data.

To validate this theory, the researchers utilize variational bounds to estimate mutual information in practical neural networks, allowing for a direct comparison between theoretical metrics and observed error gaps.

Experimental Methodology

The theory was evaluated using two primary benchmarks: MNIST and CIFAR-10. To simulate distribution shift, the researchers created a "disturbed" version of MNIST using random translations (up to 5 pixels) and rotations (up to $\pi/4$ radians). Small training sets of 5,000 samples were intentionally used to ensure models were susceptible to generalization issues, making the testing gap more observable.

Original MNIST digits (top) vs. digits with random rotation and translation (bottom) used to simulate distribution shift.

Key Insights and Empirical Validation

Compression as a Shield

The results strongly support the "Information Bottleneck" hypothesis. By limiting the mutual information $I(X; U)$—essentially forcing the model to compress the input—the testing gap shrinks. This suggests that "forgetting" irrelevant details about the input makes the model more robust to environmental changes. Furthermore, the bound explicitly accounts for decoder efficiency; a decoder poorly matched to the encoder's representations will increase the generalization gap even if the encoder is well-optimized.

Qualitative Tracking

While the mathematical bound may not be perfectly "tight" (it often overestimates the absolute error), experiments show it tracks the behavior of the gap with high fidelity. Across various stochastic architectures—including Variational Auto-Encoders (VAE), Information Dropout, and Restricted Boltzmann Machines (RBM)—the observed error gap moves in lockstep with the mutual information term as regularization strength ($\lambda$) is adjusted.

The 0.95-quantile of the observed testing gap (solid blue) tracks the mutual information term (dashed green) as the regularization strength (Lagrange multiplier) varies.

Conclusion

The study demonstrates that the testing gap scales at a rate of approximately $\frac{\log(n)}{\sqrt{n}}$, where $n$ is the number of test samples. Mutual information remains a reliable predictor of the error gap even when the model encounters data types it was not exposed to during training, such as rotated or translated digits.

Future work may address the current limitations of the bound, such as its focus on cross-entropy loss and geometric shifts. Extending these information-theoretic insights to other loss functions (like Mean Squared Error) and more complex semantic shifts (like background changes) remains a promising avenue for improving model robustness.

Summary updated

Abstract

Machine learning theory has mostly focused on generalization to samples from the same distribution as the training data. Whereas a better understanding of generalization beyond the training distribution where the observed distribution changes is also fundamentally important to achieve a more powerful form of generalization. In this paper, we attempt to study through the lens of information measures how a particular architecture behaves when the true probability law of the samples is potentially different at training and testing times. Our main result is that the testing gap between the empirical cross-entropy and its statistical expectation (measured with respect to the testing probability law) can be bounded with high probability by the mutual information between the input testing samples and the corresponding representations, generated by the encoder obtained at training time. These results of theoretical nature are supported by numerical simulations showing that the mentioned mutual information is representative of the testing gap, capturing qualitatively the dynamic in terms of the hyperparameters of the network.
BibTeX
@misc{vera2019understanding,
  author = {Matías Vera and Pablo Piantanida and Leonardo Rey Vega},
  title = {{Understanding the Behaviour of the Empirical Cross-Entropy Beyond the Training Distribution}},
  year = {2019},
  eprint = {1905.11972},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.11972},
}
Figure