Abstract
While Bayesian neural networks (BNNs) provide a sound and principled alternative to standard neural networks, an artificial sharpening of the posterior usually needs to be applied to reach comparable performance. This is in stark contrast to theory, dictating that given an adequate prior and a well-specified model, the untempered Bayesian posterior should achieve optimal performance. Despite the community's extensive efforts, the observed gains in performance still remain disputed with several plausible causes pointing at its origin. While data augmentation has been empirically recognized as one of the main drivers of this effect, a theoretical account of its role, on the other hand, is largely missing. In this work we identify two interlaced factors concurrently influencing the strength of the cold posterior effect, namely the correlated nature of augmentations and the degree of invariance of the employed model to such transformations. By theoretically analyzing simplified settings, we prove that tempering implicitly reduces the misspecification arising from modeling augmentations as i.i.d. data. The temperature mimics the role of the effective sample size, reflecting the gain in information provided by the augmentations. We corroborate our theoretical findings with extensive empirical evaluations, scaling to realistic BNNs. By relying on the framework of group convolutions, we experiment with models of varying inherent degree of invariance, confirming its hypothesized relationship with the optimal temperature.
Introduction
In the realm of Bayesian Deep Learning, a persistent paradox has puzzled researchers: the Cold Posterior Effect (CPE). According to standard Bayesian theory, a well-specified model should perform optimally when the posterior is sampled at a temperature of $T=1$. However, in practice, Bayesian Neural Networks (BNNs) often require an artificial sharpening of the posterior ($T < 1$) to achieve state-of-the-art performance.
This paper investigates the root cause of this discrepancy, focusing on the role of Data Augmentation (DA). While DA is a staple of modern deep learning, it fundamentally violates the independent and identically distributed (i.i.d.) assumption central to Bayesian inference. The authors demonstrate that tempering is not merely a heuristic trick, but a mathematical correction for the way we handle correlated augmented data.
The Problem: The i.i.d. Lie
Standard Bayesian inference assumes that each data point provides a unique, independent piece of information. When we apply data augmentation—for instance, creating ten different rotations of a single cat image—we do not actually have ten independent "cats." Instead, we have ten highly correlated observations of the same underlying data point.
When a BNN treats these $B$ augmentations as i.i.d. samples, it overcounts the evidence, leading to an overconfident posterior. This model misspecification forces practitioners to manually adjust the "thermostat" of the model to regain performance. The authors argue that the cold posterior effect is largely a byproduct of this information overcounting.
Theoretical Framework: Effective Sample Size
To provide a rigorous account of this effect, the authors analyze simplified settings, including Gaussian mean estimation and linear regression. They prove that a tempered posterior under an i.i.d. assumption can exactly match the true posterior that accounts for data correlations.
In this framework, the temperature $T$ acts as a proxy for the effective sample size. If we have $N$ original images and $B$ augmentations per image, the total number of samples seen by the model is $N \times B$. However, the actual information content is somewhere between $N$ and $N \times B$. The optimal temperature $T^*$ corrects the likelihood to reflect this true information gain, effectively "downweighting" the redundant information provided by correlated augmentations.
The Role of Model Invariance
A key insight of the paper is that the severity of the CPE depends on the model's architecture—specifically its invariance.
- Perfect Invariance: If a model is perfectly invariant to a transformation (e.g., it sees a flipped image exactly the same as the original), the second sample provides zero new information.
- Zero Invariance: If the model treats the transformation as entirely new data, the information gain is maximized.
By utilizing Group-equivariant (G-equivariant) convolutions, the authors were able to build models with varying degrees of inherent invariance. They hypothesized—and later confirmed—that models with higher architectural invariance require less tempering because they are less prone to the misspecification caused by overcounting augmented data.
Empirical Results
The researchers validated their theory using Stochastic Gradient MCMC (SG-MCMC) on standard benchmarks:
- CIFAR-10 Benchmarks: Comparing standard ResNet20 architectures against G-equivariant versions, they found that G-convolutions significantly reduced the need for extreme tempering. The "temperature landscape" for invariant models was much flatter, meaning they performed well across a broader range of $T$ values.
- Error Correlation: Using the Dogs vs. Cats dataset, the authors visualized the residuals of augmented images. They found that augmented samples formed distinct clusters in the error space, confirming their strong statistical dependence compared to unaugmented data.
- Optimal Temperature Scaling: The experiments confirmed Theorem 3.1, showing that the optimal temperature $T^*$ is an increasing function of the number of augmentations ($B$).
Conclusion and Future Directions
This work provides a long-sought theoretical justification for the cold posterior effect in the context of data augmentation. By framing tempering as a correction for effective sample size and model invariance, the authors move the field away from heuristic tuning toward a more principled understanding of BNNs.
Future work may involve automating the selection of $T$. By measuring a model's invariance during the initial "burn-in" phase of training, it may be possible to estimate the ideal temperature on the fly, eliminating the need for expensive grid searches and making Bayesian methods more accessible for large-scale applications.
Summary updated