Bayesian neural networks predict best when their posterior is artificially "cooled." Cooling means raising the likelihood to a power $1/T$ with temperature $T<1$, which makes the model act as if it had seen more data than it did. Much of this effect is tied to data augmentation, the practice of training on randomly cropped and flipped copies of each image. The copies are not new observations, so any likelihood built from them makes an arbitrary choice about how much evidence each copy carries. Standard practice picks one of two extremes without saying so: count each image once whatever the number of views, or count every view as an independent example. We propose to measure the right amount instead. For each training image, we estimate how many effectively independent observations its augmented views provide, using how strongly the model's errors on those views are correlated. That estimate fixes a per-image evidence weight that has an explicit statistical meaning and requires no temperature grid search.
The measurement depends on what the model is currently trying to explain. We therefore build the posterior by sequential error correction, the idea behind gradient boosting: a sequence of learners in which each one fits the errors left by the ones before it. Each stage gets its own measured evidence weight. This yields two results. First, a direct test of how much of the cold posterior effect is miscounted evidence and how much is something else, such as underfitting or a poor prior. Second, a posterior that runs at $T=1$ on a likelihood whose weights are measured rather than tuned.
Core Claim
Tempering at temperature $T$ is the same as giving every training image an evidence weight of $1/T$, relative to the convention of one view per image with the likelihood scaled by the dataset size $N$. Under the opposite convention, which treats all $B$ views as independent, the weight is $B$. The correct weight lies between these two, and it can be estimated. Suppose seed image $i$ has $B$ augmented views, and $\rho^{(m)}_{i,bb'}$ is the correlation between the stage-$m$ error vectors on views $b$ and $b'$. These error vectors are the gradient of the log-likelihood with respect to the output logits, which gradient boosting calls "pseudo-residuals." We define
$$ \mathrm{ESS}^{(m)}_i = \frac{B^2}{\sum_{b,b'} \rho^{(m)}_{i,bb'}}, $$
the usual effective-sample-size formula. It comes from the variance of an average of $B$ equally weighted correlated quantities. The value is 1 when the views are redundant and $B$ when they are independent. Stage $m$'s likelihood then weights view $b$ of image $i$ by $\mathrm{ESS}^{(m)}_i/B$ and is sampled at $T=1$.
We make two falsifiable claims. The first is that the implied temperature $T_{\text{implied}} = 1/\overline{\mathrm{ESS}}$, where $\overline{\mathrm{ESS}}$ is the average over images, predicts where a grid search places the best temperature on augmented benchmarks, up to a remainder we can quantify. The second is that per-stage weights differ enough that a single global temperature cannot reproduce the staged posterior's predictive quality. Because $\mathrm{ESS}$ is at most $B$, this approach cannot explain temperatures colder than $1/B$. Any cooling beyond that bound is, by construction, not a counting problem.
Relation to Prior Work
Two papers motivate the counting question and frame it differently from each other. How Tempering Fixes Data Augmentation in Bayesian Neural Networks shows that augmented views produce tightly clustered, correlated training errors. It argues that the best temperature scales with the number of augmentations, and it concedes two limits: the argument is exact only when views are perfectly correlated, and in practice $T$ is still found by expensive grid search. We keep its diagnostic, correlated errors across views, and turn it into a per-image estimator that also covers intermediate correlation. Data augmentation in Bayesian neural networks and the cold posterior effect replaces ad hoc augmentation with valid likelihoods that average predictions over the full set of transformed images, called the orbit. It finds the best temperature still at $T=0.01$ or $T=0.001$, even with exact orbit likelihoods. That result constrains us: fixing the likelihood's form alone does not remove the cold posterior effect. Our proposal differs because it does not average views into one term. It asks how much total evidence the views jointly carry, a separate quantity that orbit averaging leaves at one image's worth.
A second group supplies the competing explanations that our measurement must separate. Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect finds $T=1$ optimal on CIFAR-10 and SVHN without augmentation, but sees the effect return when data are subsampled, which points to prior misspecification. If there is no underfitting, there is no Cold Posterior Effect and The Cold Posterior Effect Indicates Underfitting, and Cold Posteriors Represent a Fully Bayesian Method to Mitigate It prove that cooling helps only when the $T=1$ model underfits, and show that going from ResNet-18 to ResNet-50 removes the effect. How Good is the Bayes Posterior in Deep Neural Networks Really? rules out the sampler as the cause. It also reports the effect growing with MLP depth, which sits awkwardly with the underfitting account. We use these papers to predict where our method should fail: on subsampled data, on text without spatial augmentation, and beyond the $1/B$ bound.
Sequential error correction supplies the method. Gradient Boosting Neural Networks: GrowNet trains shallow networks on pseudo-residuals and stacks their features. It adds a "corrective step" that jointly retrains all stages to avoid greedy traps, and it observes overfitting once weak learners exceed about two hidden layers. It is entirely deterministic. Variational Boosting boosts a mixture approximation to the posterior over one fixed model's parameters, not an additive model of the prediction function, and never considers augmentation. Our stages are additive in function space, as in GrowNet, but each carries a posterior sampled on a measured-weight likelihood. Boosting matters here because each stage's residual target defines what "redundant views" means for that stage. A single monolithic network has only one such target.
Research Program
The model is an additive logit ensemble $F_M(x) = F_0(x) + \sum_{m=1}^M h_m(x)$. $F_0$ is a ResNet-20, and each $h_m$ is a small convolutional network (one or two hidden layers, following GrowNet) that takes the image plus stacked features from earlier stages. Stage $m$ is fit with cyclical stochastic-gradient Hamiltonian Monte Carlo, as used by Wenzel et al., on the categorical likelihood with logits $F_{m-1}+h_m$. The earlier stages $F_{m-1}$ are held at their posterior-mean logits during this fit. Before sampling stage $m$, we draw $B$ views for a random subset of training images and compute pseudo-residuals under $F_{m-1}$. From these we estimate $\mathrm{ESS}^{(m)}_i$, shrinking noisy per-image estimates toward the per-class mean, and set the view weights. A final joint pass samples all stages together at their measured weights. This is the Bayesian analogue of GrowNet's corrective step. Measuring the weights costs $B$ extra forward passes on a subset once per stage, which is negligible next to sampling.
The analysis has three parts. The first records how $\overline{\mathrm{ESS}}$ changes across stages and augmentation types, because the gap in the literature is that nobody knows whether redundancy sits in the coarse early fit or persists in the fine residuals. The second compares $T_{\text{implied}}$ with the grid-searched temperature at every stage count $M$, splitting the log-temperature gap $\log_{10} T^*_{\text{grid}} - \log_{10} T_{\text{implied}}$ into a counting part and a remainder. The third tracks that remainder as stages are added. If Zhang et al. are right, the remainder should shrink as underfitting falls. If the prior explanation of Noci et al. dominates, it should persist.
One conflict in the literature has to be faced directly. Bachmann et al. describe augmentation as over-counting evidence. Under the one-view-per-step convention used in practice, however, a model that is not invariant receives partly new information from each view, so the pipeline may be under-counting. Cold temperatures would then be the correction. Which reading holds depends on the convention, and the measured $\mathrm{ESS}$ settles it empirically. A useful consistency check is that rotation-invariant group-equivariant ResNets should give $\mathrm{ESS}\approx 1$ under rotation augmentation. That would match Bachmann et al.'s finding that such models show almost no temperature shift.
Baselines
The naive reference point is the standard Bayesian neural network at $T=1$ with random augmentation and a Gaussian prior on ResNet-20 CIFAR-10. How Good is the Bayes Posterior in Deep Neural Networks Really? reports it at roughly 91% test accuracy, with SGD at roughly 93%. Can a Confident Prior Replace a Cold Posterior? reports a comparable baseline at about 89%. We re-run this baseline with our sampler so that differences in the reported figures do not contaminate the comparison.
The strongest existing tuned method is the grid-searched cold posterior, reported at approximately 94% accuracy at $T=0.01$ by Wenzel et al. At $T=1$, the strongest principled competitor is the DirClip confident prior, reported at about 94% on ResNet-20 CIFAR-10 by Marek et al., initialized from SGD as that paper requires. Neither paper reports test log-likelihood or calibration numbers, so we measure these ourselves. We also run the orbit-averaged probability likelihood of Nabarro et al. with $K=8$ samples, where that paper reports gains plateau; it gives no reported accuracy value. The capacity baseline is a monolithic width-scaled ResNet whose parameter count and sampling compute match the full staged ensemble, following Zhang et al.
Every method gets the same data, augmentations, number of gradient evaluations and posterior samples, and the same temperature grid of seven log-spaced values from 1 down to 0.001, wherever it tunes a temperature. Our method gets no grid at all. Two ablations isolate the key ingredient. The first uses the same staged architecture with one global grid-searched temperature in place of per-stage measured weights. The second is a single-stage model with measured weights, which tests whether staging matters. The single-stage model with $F_0$ fixed at SGD weights is essentially a Bayesian treatment of a correction layer on a point estimate, which also covers the "Bayesian head on SGD" alternative.
The method beats the baselines if it matches the grid-searched cold posterior's accuracy within noise, about 94%, with equal or better test log-likelihood and calibration error, while using no temperature search. It must also beat the global-temperature ablation on log-likelihood. If the global-temperature ablation performs as well, the per-stage structure is unnecessary, and we will report the contribution as the counting diagnostic alone.
Why not just…?
Why not just grid-search a global temperature? It works for accuracy, and that is exactly why it is our main baseline and the target of our ablation. A global $T$ has no stated meaning, costs a full sampling run per grid point, and by Bachmann et al.'s own analysis is only approximate when views are partly correlated. If per-stage weights come out nearly equal across stages, a global $T$ fitted to $T_{\text{implied}}$ will do just as well. The measurement then still replaces the grid search, but staging adds nothing. The ablation will show which case holds.
Why not just use a bigger network, since underfitting explains the effect? Zhang et al. show that ResNet-50 removes the effect ResNet-18 shows, so this is a real threat to any claim that boosting improves accuracy. We make no such claim. Stages serve as a controlled way to add capacity at which evidence can be measured, and the width-matched monolithic model is run at equal compute. What a bigger network cannot do is separate counting from underfitting, and Wenzel et al.'s finding that the effect grows with MLP depth shows that capacity alone is not a complete account.
Why not just use orbit-averaged likelihoods? Nabarro et al. already found the best temperature at 0.01 or below with exact orbit likelihoods. Averaging decides how views are combined but still credits each image with one observation's worth of evidence. Our measurement targets that total evidence directly. The two can be combined, and we test the combination.
Why not just use a confident prior like DirClip? It reaches about 94% at $T=1$, but it needs an SGD warm start to get past its early divergence phase, and it attributes all the cooling to the prior. If DirClip matches us on log-likelihood and calibration as well as accuracy, the counting story adds nothing practical. The subsampling and no-augmentation controls below show which explanation carries the effect on which data.
Decisive Checks
The central experiment runs on CIFAR-10 and CIFAR-100 with crop-and-flip augmentation. Padded crops with flips form a finite set of views, so we can also test in the exact finite-orbit setting of Nabarro et al. We sweep the number of stages from 0 to 4. At each stage we report $\overline{\mathrm{ESS}}$, $T_{\text{implied}}$, $T^*_{\text{grid}}$, accuracy, negative log-likelihood and expected calibration error.
Three negative controls must behave as predicted, or the account is wrong. On SVHN and CIFAR-10 without augmentation, $\mathrm{ESS}$ is 1 by construction and the effect should be absent, as Noci et al. report. On subsampled CIFAR-10, the effect should reappear with $\mathrm{ESS}$ unchanged, leaving a remainder our method cannot close. On IMDB, which has no spatial augmentation, Wenzel et al. find the best temperature between 0.01 and 0.2, and our correction should explain none of it. The equivariance check on Rotated MNIST, comparing ordinary and group-equivariant networks under rotation augmentation, should give $\mathrm{ESS}\approx 1$ for the equivariant model and $\mathrm{ESS}>1$ for the ordinary one.
The idea survives if $T_{\text{implied}}$ tracks $T^*_{\text{grid}}$ across augmentation strengths within a factor of about two, and if the controls behave as described. It fails if $T_{\text{implied}}$ does not change as augmentation is added. It also fails if the measured weights push performance away from the grid optimum, because that would mean error correlation is the wrong measure of redundant evidence.
References
- How Tempering Fixes Data Augmentation in Bayesian Neural Networks
- Data augmentation in Bayesian neural networks and the cold posterior effect
- Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
- If there is no underfitting, there is no Cold Posterior Effect
- The Cold Posterior Effect Indicates Underfitting, and Cold Posteriors Represent a Fully Bayesian Method to Mitigate It
- How Good is the Bayes Posterior in Deep Neural Networks Really?
- Gradient Boosting Neural Networks: GrowNet
- Variational Boosting: Iteratively Refining Posterior Approximations
- Can a Confident Prior Replace a Cold Posterior?