How Tempering Fixes Data Augmentation in Bayesian Neural Networks

Gregor Bachmann, Lorenzo Noci, Thomas Hofmann

TL;DR: The "Cold Posterior Effect" (CPE)—the requirement to artificially sharpen Bayesian Neural Network (BNN) posteriors to achieve good performance—is largely a consequence of data augmentation. By treating correlated augmented samples as independent, standard BNNs overcount information. Tempering the posterior acts as a necessary correction to account for the true "effective sample size" of the data.

The Problem: Breaking the i.i.d. Assumption

In theory, Bayesian inference is optimal: given a good prior and model, the untempered posterior should yield the best possible predictions. In practice, however, BNNs are often outperformed by simple Stochastic Gradient Descent (SGD) unless researchers "cheat" by using a temperature $T < 1$ to sharpen the posterior. This phenomenon is known as the Cold Posterior Effect.

Data augmentation creates highly correlated samples, breaking the assumption of independent data.

The authors investigate why this happens, focusing heavily on Data Augmentation (DA). While DA is essential for state-of-the-art performance in computer vision, it breaks a fundamental assumption of Bayesian inference: that data points are independent and identically distributed (i.i.d.). If you take one image of an otter and create 100 rotated versions of it, you haven't actually gained 100 images' worth of new, independent information. Standard BNNs ignore this correlation, leading to a "misspecified" model that is overly confident in its redundant data.

Visualizing the "i.i.d. Lie"

Data augmentation produces samples that are highly correlated. When evaluating the residuals (errors) of these samples during training, a stark contrast emerges between augmented and unaugmented data.

Residuals of augmented samples (left) show strong correlation and clustering compared to the independent scatter of unaugmented residuals (right).

As shown above, the residuals of augmented samples form distinct clusters, whereas unaugmented samples show no such trend. This correlation means the "effective sample size" of the dataset is much smaller than the total number of augmented images processed by the network.

Methods: Invariance and the Tempering Fix

To prove that tempering corrects this overcounting, the authors combine theoretical proofs with controlled experiments using Group Equivariant Convolutions (G-Convs) and SG-MCMC sampling on datasets like CIFAR-10.

  • Invariance Drives Correlation: If a model is perfectly invariant to a transformation (e.g., it processes a rotated image exactly the same as an upright one), then augmenting with that transformation provides zero new information. The more invariant the model, the more redundant the augmentations become, and the more tempering is needed to fix the overcounting.
  • Tempering as a Rescaler: The authors theoretically demonstrate that the optimal temperature $T^*$ is directly related to the number of augmentations $B$. In their framework, applying a temperature $T$ effectively rescales the augmented dataset size to match its true information content.
  • Empirical Confirmation: When testing standard ResNets against G-ResNets (which are mathematically designed to be more invariant to rotations), standard ResNets required a significant shift in temperature to maintain performance when rotations were added. The G-ResNets showed almost no shift, confirming that model invariance is a primary driver of the CPE. In models designed to be nearly perfectly invariant, the optimal temperature $T$ closely matched the number of augmentations $B$.

Limitations and Future Work

While tempering significantly reduces model misspecification, it is only a partial fix. It perfectly matches the true posterior only in simplified or extreme cases of perfect correlation; for intermediate correlation values, tempering is a strong approximation rather than a perfect mathematical equivalent.

Furthermore, data augmentation isn't the sole cause of the Cold Posterior Effect. Factors like curated datasets (which aren't truly random samples) and poorly chosen priors also contribute. Future research may focus on developing automated estimators for the ideal temperature $T$ based on a model's measured invariance during early training, potentially eliminating the need for expensive grid searches.

Summary updated

Abstract

While Bayesian neural networks (BNNs) provide a sound and principled alternative to standard neural networks, an artificial sharpening of the posterior usually needs to be applied to reach comparab...
BibTeX
@inproceedings{bachmann2022how,
  author = {Gregor Bachmann and Lorenzo Noci and Thomas Hofmann},
  title = {{How Tempering Fixes Data Augmentation in Bayesian Neural Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=Gsgm2XMTUgB},
}
Figure