Counting Augmented Evidence by Its Residuals: Predicting the Cold Posterior Temperature

Bayesian neural networks average predictions over a distribution of plausible weights. They are routinely trained with data augmentation, meaning random crops, flips, or rotations of each training image. They also routinely need a "cold" posterior to work well: the likelihood is raised to a power $1/T$ with $T \ll 1$. That temperature is chosen by grid search and has no probabilistic justification. This note proposes to replace the tuned global temperature with a per-image evidence weight that is measured rather than tuned. The weight is read off how correlated the model's errors are across the augmented copies of each image. If the model makes the same error on every copy, the copies are redundant and together count as roughly one observation. If its errors on different copies look unrelated, the copies behave more like separate observations and should count for more.

The claim is that most of the cold posterior effect in augmented networks is this redundancy correction applied crudely and globally. It can therefore be predicted before any temperature search. Whatever temperature remains after the correction should be explained by underfitting, and a stage-wise residual correction run at $T=1$ should remove that remainder. If the claim holds, practitioners get a temperature they can compute and a clean split of the cold posterior effect into its causes. If it fails, we learn that the redundancy story, which is currently the most popular mechanistic account, is not what sets the temperature.

Core Claim

Two findings motivate the idea. How Tempering Fixes Data Augmentation in Bayesian Neural Networks shows three things. Residuals of augmented copies of one image cluster tightly. The optimal temperature scales with the number of augmentations $B$. Model invariance governs how much shift is needed: rotation-invariant group-equivariant ResNets need almost none, while standard ResNets need strong cooling. The same paper concedes that tempering is exact only in the limit of perfect correlation and is a heuristic everywhere in between. Data augmentation in Bayesian neural networks and the cold posterior effect builds likelihoods that average predictions over each image's set of transformations. These count each image exactly once. Even with these likelihoods the optimal temperature stays far below one.

We read these two results as consistent. The orbit-averaged likelihood gives each image weight one. When the model is not invariant, its copies carry partly independent information, so a weight of one is too small and cooling compensates. The quantity missing from both papers is the degree of independence, measured per image, for the current model.

We borrow the object that sequential error correction works with: the pseudo-residual. This is the negative gradient of the loss with respect to the model's output. Gradient boosting machines, a tutorial defines it, and it is what each boosting stage fits. For classification it is $r_{ib} = e_{y_i} - \mathrm{softmax}(f_\theta(t_b(x_i)))$ on view $b$ of image $i$, where $e_{y_i}$ is the one-hot label.

Let $\rho_i \in [0,1]$ be the average pairwise cosine similarity of these residual vectors across the $B$ views, averaged over posterior samples. The effective number of observations contributed by image $i$ is then

$$n_{\mathrm{eff},i} = \frac{B}{1 + (B-1)\rho_i},$$

which is the standard variance of a mean of $B$ equally correlated quantities, inverted into an effective count. Each image's averaged log-likelihood is multiplied by $n_{\mathrm{eff},i}$, and sampling proceeds at $T=1$. The implied temperature for image $i$ is $T_{\mathrm{red},i} = 1/n_{\mathrm{eff},i}$. It ranges from $1$, for a fully invariant model where the views are redundant, to $1/B$, for a model whose errors on the views are uncorrelated. These two ends match the behavior Bachmann et al. report for invariant and standard networks.

The residual pattern changes as the posterior moves, so the weights are re-estimated in stages, much as boosting re-weights examples. The update uses boosting's shrinkage step $\nu$ to damp oscillation. We state plainly that this is a weighted composite likelihood, not an exact likelihood. The synthetic views still enter as evidence. What changes is that their weight is set by measured redundancy instead of a global knob.

The falsifiable prediction is quantitative. The dataset-level $\bar T_{\mathrm{red}}$ should predict the grid-searched $T^*$ across augmentation policies and architectures. The gap $\log T^* - \log \bar T_{\mathrm{red}}$ should track training underfitting as defined by If there is no underfitting, there is no Cold Posterior Effect. That paper defines the cold posterior effect as test loss falling as inverse temperature rises past one, and shows this occurs only when the $T=1$ posterior underfits.

Research Program

The first component is measurement. We instrument a stochastic-gradient MCMC sampler, starting from the released code of The Cold Posterior Effect Indicates Underfitting, and Cold Posteriors Represent a Fully Bayesian Method to Mitigate It at https://github.com/pyijiezhang/cpe-underfit. The instrumentation records per-view pseudo-residuals on a held-in subset every few epochs. That lets us estimate $\rho_i$ and its drift over training. Images whose residuals are already near zero on every view add almost no gradient, so we assign them $\rho_i = 1$ rather than estimate an unstable cosine.

The second component is the redundancy-weighted sampler. Its weights are refreshed in stages with shrinkage, and it always runs at $T=1$.

The third component is attribution. For each configuration we separately measure $T^*$, the predicted $\bar T_{\mathrm{red}}$, and the underfitting derivative of Zhang et al. We then ask how much of the variation in $\log T^*$ each one explains.

The fourth component tests the remaining underfitting. We use a small additive correction network whose output is added to the base network's logits. It is sampled at $T=1$ by blocked updates in which the base network's output is held fixed as an offset, in the spirit of Approximate blocked Gibbs sampling for Bayesian neural networks. Because the sum of the two networks is a single model with a single joint posterior, this stays a coherent posterior. It is not the ad hoc reuse of data that ordinary boosting performs. The prior on the correction network is scaled down so that it favors small corrections. That choice follows the finding in Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect that cooling acts partly as a simplicity prior.

This component is secondary. Its job is to check the attribution, and we compare it against simply scaling capacity.

Several prior results constrain the design. A Bayesian Boosting Model shows that residual-driven reweighting overfits mislabeled points. Our weights therefore depend only on the correlation of residuals, not their size, and we audit the lowest-$\rho$ images for label errors. Can a Confident Prior Replace a Cold Posterior? shows that part of the effect comes from curated labels, which no redundancy correction touches. Bayes without Underfitting removes underfitting by restricting posterior spread to directions that leave training predictions unchanged around a single MAP (maximum a posteriori) point. It serves as a comparison for the attribution stage, not as a competitor for the redundancy claim.

Baselines

The naive reference is stochastic-gradient MCMC at $T=1$ with standard augmentation. How Good is the Bayes Posterior in Deep Neural Networks Really? reports about 91% CIFAR-10 test accuracy for a ResNet-20 in this setting. The second reference point is $T=1$ without augmentation, where Noci et al. report an optimal temperature of $T^*=1$. It tells us what accuracy the redundancy weighting must recover.

The strongest competitor is a globally tempered posterior with $T$ tuned by grid search on validation negative log-likelihood. Wenzel et al. report about 94% CIFAR-10 accuracy for ResNet-20 at $T=0.01$, against about 93% for SGD. It is strongest because it is what practitioners use and because it has an extra tuned degree of freedom that our method does not. We run it with the same sampler, schedule, number of posterior samples, and augmentation policy. Its grid is at least eight temperatures. Our method gets zero temperature tuning and the same number of total sampler epochs, with the residual-estimation forward passes counted against its budget. Beating it means one of two outcomes. The first is matching its accuracy within seed noise while improving negative log-likelihood or expected calibration error (ECE), the gap between stated confidence and actual accuracy, on CIFAR-10 and CIFAR-10-C. The second, which is enough to support the scientific claim even if accuracy ties, is predicting its tuned $T^*$ to within a factor of two across configurations.

The principled-likelihood baseline is the orbit-averaged likelihood of Nabarro et al., in both logit-averaging and probability-averaging forms. It is run at $T=1$ and at its own tuned $T$. It is the closest method that also refuses global tempering. We also include DirClip from Marek et al. at $T=1$, reported at about 94% on CIFAR-10 with ResNet-20, using their MAP warm start. Deep ensembles serve as an external reference. MixupMP from Posterior Uncertainty Quantification in Neural Networks using Data Augmentation reports 96.83% for deep ensembles and 97.77% for MixupMP on CIFAR-10. The file does not specify the architecture, so we rerun ensembles on our ResNet-20 rather than compare to those figures.

Four ablations isolate the key ingredient. The first uses a single global weight computed from the mean $\rho$, which tests whether per-image weighting matters. The second randomly permutes the per-image weights while keeping their distribution, which tests whether the weights attach to the right images. The third computes weights from input-space similarity of the views instead of residual similarity, which tests whether the model's own errors are the right signal. The fourth freezes the weights at initialization instead of refreshing them in stages, which tests the sequential update.

Why not just…?

Why not just tune a global temperature, since it is one line of code? Accuracy-wise it may tie, and we say so. The case against it has three parts. First, $T^*$ moves unpredictably across data, architecture, and augmentation policy: Wenzel et al. report $T \in [0.01, 0.2]$ on IMDB, and Bachmann et al. find a dependence on $B$ and on invariance. Each new setting therefore needs a costly grid search over full sampling runs. Second, a single scalar cannot distinguish an image whose views are redundant from one whose views are informative. Third, What Are Bayesian Neural Network Posteriors Really Like? finds that posteriors degrade faster than SGD under corruption. We test whether per-image weighting changes that on CIFAR-10-C. If the global-weight ablation matches the full method everywhere, the per-image claim is dead, and the contribution reduces to "the temperature is predictable from residual correlation." That would still be useful, and it is exactly what the prediction experiment tests.

Why not just use the orbit-averaged likelihood, which is already principled? Nabarro et al. show it still needs $T \ll 1$. Our account explains why: it counts each image once even when the model's errors across views are nearly independent. This is a real point of contact, not a dismissal. If orbit averaging with the count set to $n_{\mathrm{eff},i}$ instead of one removes most of the effect, the two papers are reconciled. If it does not, our mechanism is wrong.

Why not just use a bigger network or a confident prior to remove underfitting? Zhang et al. show that moving from ResNet-18 to ResNet-50 weakens the effect, and DirClip matches cold accuracy at $T=1$. Both address underfitting and curation, not redundancy. DirClip, as its authors note, does not address augmentation overcounting at all. Both may therefore shrink the leftover gap after our correction without substituting for it. The attribution experiment settles this. If DirClip or ResNet-50 alone drives $T^*$ to one under heavy non-invariant augmentation, redundancy is not the dominant factor and our claim fails.

Proposed Experiments

The main benchmark is CIFAR-10 with ResNet-20, using the sampler setup of Wenzel et al. and Zhang et al. We cross four augmentation settings (none, flips, crop plus flip, and rotations with $B \in \{4, 8, 16\}$) with three architectures (ResNet-20, ResNet-50, and a rotation-equivariant ResNet as in Bachmann et al.). We add training-set subsamples at several sizes $n$, following Noci et al. CIFAR-100 tests transfer to more classes. For it, Nabarro et al. evaluate lower bounds but no reported result was found on Lacuna for a tempered-posterior accuracy baseline, so we produce our own. IMDB with the CNN-LSTM of Wenzel et al., trained without augmentation, is a negative control. Our method predicts $n_{\mathrm{eff}}=1$ there, so any cold posterior effect on IMDB must come from other causes. CIFAR-10-C measures robustness under shift. The metrics are test accuracy, negative log-likelihood, ECE, grid-searched $T^*$, predicted $\bar T_{\mathrm{red}}$, and the underfitting derivative at $\lambda=1$.

Three decisive checks follow. First, across the roughly thirty configurations, $\log \bar T_{\mathrm{red}}$ should explain most of the variance of $\log T^*$ in the augmented settings. It should give $\bar T_{\mathrm{red}} \approx 1$ exactly where Bachmann et al. find no shift, namely for equivariant networks under rotation. Second, the leftover gap should correlate with the underfitting derivative, should shrink with ResNet-50, and should shrink with the additive correction network at $T=1$. Third, the per-image method should beat the permuted-weight and input-similarity ablations on negative log-likelihood. If the first check fails, we report that residual redundancy does not set the cold temperature, which would be a useful negative result for the overcounting explanation.