Swap-Aware Compression of Bayesian Neural Network Posteriors

A Bayesian neural network keeps a probability distribution over its weights instead of a single weight vector. The most faithful way to obtain that distribution is Hamiltonian Monte Carlo (HMC), a sampler that simulates a particle moving over the loss surface. HMC returns thousands of weight vectors, and they are hard to use. Hidden neurons have no fixed order, so two samples can compute nearly the same function while storing it in shuffled coordinates. Averaging them coordinate by coordinate produces nonsense. The current fix is to permute every sample so that its neurons line up with a chosen reference network, a step called rebasin, and then fit one Gaussian to the aligned samples. This raises agreement between the compressed model and the HMC predictions from 12% to 82% on MNIST (A Compact Representation for Bayesian Neural Networks By Removing Permutation Symmetry). We ask what happens in the remaining 18%, and we propose that much of it has one specific geometric cause that can be repaired.

Our claim is that rebasin fails locally, not globally. When two hidden units in a sample are nearly identical, which happens near what geometers call a collision point, the choice of which one matches which reference neuron becomes a coin flip. That flip varies from sample to sample. Each such pair then produces a two-humped marginal distribution in the aligned coordinates, and a single Gaussian smears it into a wide blob centered on a low-probability average. This is exactly the picture predicted by the proof that neural weight space, once symmetries are removed, has branch points shaped locally like $z \mapsto z^2$, where no single consistent way of labeling neurons exists (No Global Gauge in Neural Weight Space). We propose to detect these ambiguous pairs, and dead units, in aligned posterior samples. We then represent each detected pair in coordinates that do not change when the two units are swapped, and leave everything else as an ordinary Gaussian. The result is a compact posterior that is a product of small local mixtures rather than one global Gaussian.

Core Claim

After rebasin, the error left when HMC samples are compressed into a Gaussian is concentrated in a small, detectable set of neuron pairs whose alignment is ambiguous. Treating those pairs with a swap-invariant local representation should close most of the gap to raw HMC. It should do so at almost no parameter cost, and with far fewer parameters than a generic mixture model needs to match it.

The mechanism is simple to state. Let $a$ and $b$ be the incoming-plus-outgoing weight vectors of two units that the matching step finds hard to tell apart. Their sum $s = a + b$ is well defined regardless of labeling, and so is the difference $d = a - b$ up to sign. We model $s$ with a Gaussian and $d$ with a mirrored two-component mixture $\tfrac12 \mathcal{N}(d;\mu,\Sigma) + \tfrac12 \mathcal{N}(d;-\mu,\Sigma)$. This is the local chart that the branched geometry demands. It uses the same number of parameters as the naive Gaussian and has no label-switching variance. For tanh networks, sign-flip ambiguity gets the same mirrored treatment. For units that are effectively dead, whose outgoing weights the data does not constrain, the outgoing weights revert to the prior instead of receiving a misleading fitted variance.

Suppose there are $k$ ambiguous pairs that behave independently. A generic Gaussian mixture over the full weight vector would need on the order of $2^k$ components to express the same distribution. The factorized local charts need $k$ two-component factors. This combinatorial gap is the reason to use geometry instead of a larger mixture.

Relation to Prior Work

Three papers motivate the proposal by showing that symmetry corrupts posterior inference.

  • Xiao et al. (A Compact Representation for Bayesian Neural Networks By Removing Permutation Symmetry) introduce the rebasin-plus-Gaussian compression we build on. They describe the aligned posterior as approximately unimodal and leave both reference sensitivity and residual structure uncharacterized. We keep their pipeline and target what it misses.
  • Gelberg et al. (Variational Inference Failures Under Model Symmetries) prove that as width grows, symmetric modes move closer together. Standard variational inference then averages between them, which they call the proximity trap. Their remedy symmetrizes the variational distribution over the whole permutation group, using a mutual-information estimator whose noise grows with group size. We symmetrize only over the handful of pairs that are actually ambiguous, which is cheaper and does not require an estimator.
  • Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry (paper) shows that the predictive distribution can be computed on a symmetry-free region, and argues for many short chains. It does not build a compact posterior representation.

The geometry paper supplies the method. No Global Gauge in Neural Weight Space proves that any global canonicalization breaks near unit collisions, and that learning needs an atlas of local charts. It is purely theoretical and never touches posteriors. Our swap-invariant pair coordinates are the simplest concrete instance of such a chart, applied to Bayesian samples.

Two lines of work constrain the design. Lim et al. (The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof) remove symmetry architecturally by fixing untrainable weights. This rescues deep variational inference but changes the model and its prior, and they report no calibration results. Function-space samplers (Functional Stochastic Gradient MCMC for Bayesian Neural Networks) avoid weight space entirely, but they depend on the placement of measurement points and do not produce a compact weight distribution.

The proposal also offers a resolution of the "width paradox" in the file. Debugging ReBasin finds that wider networks align almost perfectly for merging, while Gelberg et al. find that wider networks break variational inference. Both can hold if width creates more near-duplicate units. Near-duplicates are functionally interchangeable, which is harmless when merging point estimates. They are distributionally ambiguous, which is harmful for Gaussian summaries. This is a testable prediction.

Research Program

We will run long HMC on tanh and ReLU multilayer perceptrons at widths from 16 to 512, following the setups of Xiao et al. and the symmetry-MCMC paper. We then align samples by rebasin and add a detection step. For every pair of units in every layer, we measure how often the matching assignment for that pair flips across samples, and how small the matching cost margin is. Pairs above a threshold are marked ambiguous, and units with near-zero incoming norm are marked dead. The compressed posterior is fit by maximum likelihood on the aligned samples in the mixed coordinates.

A second phase moves the idea from post-hoc compression into training. We will build a variational family with the same structure and compare it with symmetrized mixtures. In that family, ambiguous pairs are chosen periodically from the current variational samples.

The analysis will track three quantities against width, and relate them to Theorem 3.1 of Gelberg et al.:

  • the number of ambiguous pairs;
  • the share of residual predictive disagreement those pairs explain;
  • sensitivity to the choice of reference sample.

Baselines

The naive reference point is the unaligned coordinate-wise Gaussian on raw HMC samples. Xiao et al. report 12% agreement with HMC predictions for it on MNIST. It anchors the scale.

The strongest existing compressed representation is rebasin followed by a diagonal Gaussian, run exactly as Xiao et al. describe, for which they report 82% agreement on MNIST. It is the direct competitor. Our method is that pipeline plus one step, so any gain isolates the local charts. We will reuse the same HMC samples, reference sample and matching algorithm for both. We count beating it as a statistically reliable agreement gain across five reference choices, with matched parameter count, together with lower predictive negative log-likelihood.

The upper reference is the full HMC sample ensemble itself. It is not a compressed method; it measures how much gap remains.

We will also run standard mean-field variational inference and the symmetrized mixture of Gelberg et al.:

  • Gelberg et al. report an ELBO of -0.004 versus -0.022 for mean-field on their tractable model.
  • The symmetrized mixture reaches 95.55% accuracy at width 30 on MNIST.

These baselines matter for the training-time variant. We run them at the same widths with the same number of optimization steps and learning-rate sweeps. For deep networks, the strongest symmetry-aware baseline is the fixed-weight asymmetric network of Lim et al., which reaches 55.16% on CIFAR-10 with a 16-layer MLP under variational inference. We will reproduce it and also score its calibration, which that paper did not report.

Deep ensembles trained with the same total gradient budget serve as the practical baseline. For out-of-distribution detection we also report function-space SGHMC, which scores an AUC of 0.717 against 0.669 for SGLD on CIFAR-10 versus SVHN (Functional Stochastic Gradient MCMC). We add one comparison point for the Laplace side: FSP-Laplace reports 97.7% out-of-distribution detection accuracy on MNIST (FSP-Laplace).

Two ablations isolate the key ingredient:

  • Random pairs. Apply the swap-invariant treatment to randomly chosen pairs, in the same number as detected pairs. If this helps equally, the detection is doing nothing.
  • Mirrored mixture on single units. Apply the mirrored mixture per unit instead of per pair, to test whether the pairing structure matters.

Why not just…?

Why not just fit a Gaussian mixture model to the rebasin-aligned samples? This is the most serious threat, and we treat it as an experiment rather than an objection. At small width, with only a few ambiguous pairs, a mixture with a handful of components may match our representation. The prediction is that as ambiguous pairs accumulate with width, a generic mixture needs a number of components that grows exponentially with the count of independent pairs to keep up. Our factorized charts grow only linearly. We will fit mixtures with 2 to 64 components on the same samples and plot agreement against parameter count at each width. If generic mixtures match us at equal parameters at width 256 and above, the geometric story is wrong, and we will say so.

Why not just improve the alignment, for instance by choosing a better reference or iterating to a consensus network? Better references reduce flips caused by a poor reference, but they cannot remove flips between two units that are genuinely near-identical in the posterior. The geometry paper proves that no single global labeling is continuous there. The reference-sensitivity experiment settles this. If iterative consensus alignment drives the ambiguous-pair count to near zero, the chart machinery is unnecessary.

Why not just keep the raw samples or train a deep ensemble? Raw samples are the gold standard for prediction, but they cannot be summarized, pruned by uncertainty, or compared with variational inference. Xiao et al. show that aligned uncertainty estimates prune better than ensemble-only statistics, which is the use case compression serves. On predictive quality, the symmetry-MCMC paper reports that its multi-chain samples achieve higher log predictive density than deep ensembles on UCI, though the file gives no table values. We will measure the ensemble at matched compute rather than assume it loses.

Why not just remove the symmetry architecturally? Lim et al. show this rescues training, but it changes the model class and its implied prior over functions. Its uncertainty quality has not been measured. We evaluate it on the same calibration and out-of-distribution metrics. If it matches our method there, the simpler route wins for new models, while our method still applies to standard architectures that already exist.

Proposed Experiments

On MNIST we use MLPs of widths 16 to 512 with HMC reference chains, and measure agreement with HMC predictions, negative log-likelihood, expected calibration error, and accuracy after uncertainty-guided pruning at 70–90% sparsity. On the seven UCI regression sets, with tanh networks, we measure log pointwise predictive density and RMSE against HMC and ensembles. On CIFAR-10 we use MLPs and small convolutional networks with SGHMC samples in place of full HMC, and add out-of-distribution AUC against SVHN.

The decisive check is a single plot per dataset. Its horizontal axis is width, and it shows the number of ambiguous pairs together with the agreement gap between the rebasin Gaussian and HMC. The proposal predicts three things:

  • both curves rise with width;
  • local charts recover most of the gap;
  • the random-pair ablation recovers none of it.

A second decisive check is whether our representation's advantage over a size-matched generic mixture grows with width. That growth is what distinguishes geometric structure from extra capacity.