Gauge-Fixed Stochastic Perturbation Theory for Neural Network Posteriors

Many neural networks have exact continuous symmetries. You can rotate the hidden units of a low-rank factorization $BA$, or the shared inner dimension of attention's query and key matrices, without changing what the network computes. Under a Gaussian prior, the probability of the weights is unchanged as well. This proposal argues that these symmetries are the reason a natural calculation fails: computing corrections to a Bayesian network's predictions order by order in the noise level, beyond the usual Gaussian (Laplace) approximation around a trained solution. Around such a solution, each symmetry direction is perfectly flat. The Gaussian approximation is singular there, and any sampler that tracks corrections order by order drifts freely along those directions. That drift makes the noise grow rapidly with the order of the correction. Lattice gauge theory hit exactly this wall thirty years ago and found a fix: stochastic gauge fixing, a small corrective rotation applied at every simulation step that pulls the trajectory back toward a reference slice of the symmetry orbit. The correction leaves every symmetry-invariant quantity untouched.

We propose to port that machinery to neural networks. Numerical stochastic perturbation theory runs Langevin dynamics (gradient descent plus calibrated Gaussian noise, whose long-run samples follow the posterior) with every weight expanded as a power series in a small parameter, and propagates each coefficient separately. Here the small parameter is the square root of the posterior temperature. We add a Zwanziger-type gauge-fixing step built from the network's symmetry group. Two things become possible. First, higher-order posterior-predictive coefficients can be computed without enumerating the fast-growing set of derivative terms in a symbolic higher-order Laplace expansion. Second, those coefficients can be measured directly rather than extracted by extrapolating noisy samples taken at several temperatures. The result is a sampler that reports the second, third and fourth corrections to a network's predictive mean and variance, with error bars that stay bounded as the run lengthens.

Core Claim

The noise explosion in high-order stochastic perturbation theory comes from one structural cause: zero-curvature directions generated by a continuous symmetry. Whether that symmetry is a lattice gauge group or a neural network's hidden-unit rotation group is immaterial. The fix carries over when three conditions hold: the symmetry acts on the weights, it leaves the posterior exactly invariant, and the observable is invariant.

The mechanism is simple. Write the weights as $W(t)=W^\ast+\sum_{k\ge1}\epsilon^k W_k(t)$, where $W^\ast$ is a trained mode, $\epsilon=\sqrt{T}$ is the square root of the posterior temperature, and $W_k$ is the order-$k$ fluctuation. At first order, $W_1$ performs an Ornstein–Uhlenbeck process in the directions where the Hessian is curved. Along orbit tangents it performs a free random walk, so its spread grows like $\sqrt{t}$ in Langevin time $t$. Higher orders are driven by products of lower ones. If nothing stops the zero modes, the variance of the order-$k$ estimate therefore grows like a power of $t$ whose degree increases with $k$. This is the pattern Di Renzo et al. saw in Lattice perturbation theory on the computer and Weak coupling perturbation theory by Langevin dynamics: Fourth loop and beyond. In those papers the order-$g^4$ plaquette signal drowned without gauge fixing and settled onto a plateau with it.

The neural network version applies, at each step, an infinitesimal group element $\exp(-\alpha\,\omega[W])$. Here $\alpha$ is a damping step size and $\omega$ is the projection of the current fluctuation onto the Lie algebra of the symmetry group: the analog of the lattice divergence of the gauge field. The step is expanded order by order with the Baker–Campbell–Hausdorff formula, exactly as in the lattice case. The group action commutes with the Langevin dynamics and fixes every invariant observable, so the gauge-fixing step cannot bias predictions. Its only effect is to remove variance.

Relation to Prior Work

The lattice papers supply the method. They are Lattice perturbation theory on the computer, its fourth-loop sequel, and Developments and new applications of numerical stochastic perturbation theory. These works expand SU(N) link variables in the coupling, apply Zwanziger's divergence-based restoring transformation, and obtain plaquette coefficients through $g^4$. They never left lattice gauge theory. Our contribution is to identify when a learning problem has the same structure, construct the group-specific restoring term, and test it on posterior observables.

New issues for numerical stochastic perturbation theory takes a different route. It adds an explicit Faddeev–Popov gauge-fixing action and its determinant. We use it as an ablation that separates fixing the gauge through the target density from fixing it through the dynamics.

Two papers motivate the work. Symmetry in Neural Network Parameter Spaces documents that stochastic optimizers drift along symmetry orbits, but proposes no stochastic restoring force and does not connect the drift to perturbative variance. The diagrammatic programs Asymptotics of Wide Networks from Feynman Diagrams and Finite-Width Neural Tangent Kernels from Feynman Diagrams report that going past leading order makes the number of diagrams grow rapidly. That is the same kind of combinatorial wall NSPT was built to avoid. We do not claim to solve their width expansion. Our small parameter is temperature around a trained mode, and the width expansion is a separate question we mention only as a possible extension.

One paper constrains the design. Learning Physical Systems: Symplectification via Gauge Fixing in Dirac Structures reports that soft penalties do not stop constraint drift in learned dynamics, whereas explicit geometric gauge fixing does. This warns us against using weight decay or symmetry-breaking penalties as the fix. Its setting is deterministic and non-perturbative, so it supplies no stochastic method we can reuse.

Research Program

The work proceeds in three stages.

The first stage reimplements lattice NSPT on modern accelerators as a correctness anchor. It covers SU(3) on an $8^4$ lattice through order $g^4$, using the published Langevin step $\tau=0.02$ and gauge-fixing parameter $\alpha=0.02$. This stage confirms that our order-by-order engine reproduces the known coefficient and the known plateau behavior.

The second stage builds the neural network engine in JAX, which supports automatic differentiation of each order's drift. Each order's drift is obtained from Taylor-mode derivatives of the log posterior. For three model classes we derive the restoring term. In a low-rank factorized regression layer $BA$ with a Gaussian prior, the group is the orthogonal group $O(r)$ acting as $B\to BO$, $A\to O^\top A$. In a single attention head, the group acts jointly on the query–key pair and on the value–output pair. In a deep linear network, each hidden layer carries its own orthogonal group. In every case $\omega$ is the antisymmetric part of $B^\top \delta B - \delta A\, A^\top$ (and its analogs), evaluated order by order. These groups are exact symmetries of a Gaussian prior times any likelihood that depends only on the product. The invariance claim is therefore a theorem in these cases, not an approximation.

The third stage characterizes variance growth with and without gauge fixing and checks it against the power law predicted by the zero-mode random walk. It then measures how far in order the method reaches under a fixed compute budget.

Baselines

The naive reference point is the first-order Laplace approximation. Its singular Hessian is handled with a pseudo-inverse that discards the zero modes. Every method is required to match it at first order. Beating it means showing that the second-order and higher coefficients it omits are statistically nonzero and change predictive calibration on held-out data.

The direct ablation is order-by-order Langevin without gauge fixing. It uses the same models, step size, number of chains and wall-clock budget. Gauge fixing beats it if the order-$k$ coefficient's standard error stops growing with run length instead of growing as a power of $t$, and if at fixed compute the error bar at orders three and four shrinks by at least an order of magnitude. On the lattice the reported anchor is $c_4 = 1.211 \pm 0.003$ for the $1\times1$ Wilson loop on $8^4$, averaged over 2,048 iterations and costing 2.455 Gflops per Langevin iteration on the CM2 (Lattice perturbation theory on the computer). Our reproduction should match that value within its error. It should also show the no-gauge-fixing variant failing to do so at equal iteration count.

The strongest existing competitor for the neural network question is plain stochastic-gradient Langevin sampling at several small temperatures, followed by a polynomial fit in $T$ to the invariant observable. Because that observable is invariant, symmetry drift does not hurt this method at all. It gets the same total gradient evaluations and the same chains, and its temperatures and fit degree are tuned on a validation model. No reported result was found on Lacuna for higher-order temperature coefficients of neural network posteriors, so the comparison is against our own exact references (described below). Beating this baseline means lower mean squared error on the order-2 to order-4 coefficients at equal compute.

A second strong competitor is symbolic higher-order Laplace expansion via automatic differentiation of the log posterior. Its cost and memory are measured as order rises, and the order at which it stops fitting on one accelerator is recorded.

The final ablation pits our dynamical, Zwanziger-style fix against two alternatives under identical budgets. One is a Faddeev–Popov gauge-fixing action in the style of Burgio et al. The other is a hard projection onto a canonical slice at every step, for example by singular-value-decomposition balancing of $B$ and $A$.

Why not just…?

Why not just sample at a few small temperatures and extrapolate? This is the real threat, and the proposal is designed to settle it. Extrapolation needs no symmetry handling, because invariant observables never see the drift. Its weakness is statistical. Isolating the $T^k$ coefficient from samples at temperature $T$ amplifies sampling noise by roughly $T^{-k}$, so the bias–variance trade-off over the choice of temperatures worsens with order. NSPT estimates each coefficient as its own process and avoids that amplification, but only if the zero-mode variance is controlled. That control is precisely what gauge fixing provides. If extrapolation matches our method at equal compute through order four, we will report that the contribution reduces to a diagnosis of why ungauged NSPT fails, not a better estimator.

Why not just add weight decay or a penalty on the orbit coordinate? Extra weight decay changes the prior and therefore the posterior being expanded, so it biases the very coefficients we want. A symmetry-breaking penalty without its Faddeev–Popov determinant has the same problem at higher orders. Learning Physical Systems: Symplectification via Gauge Fixing in Dirac Structures also reports that soft penalties fail to stop drift in a related deterministic setting. We include a penalty variant and measure its bias against the exact references.

Why not just project onto a canonical gauge slice each step? This is legitimate. Gauge transformations commute with the dynamics, so any slice gives unbiased invariant observables. It may work as well as our method. Its drawback is that the projection is a nonlinear map, such as a singular value decomposition, that must itself be expanded order by order. The infinitesimal Zwanziger step expands cleanly with Baker–Campbell–Hausdorff. The ablation compares the two on variance and cost per order. If projection wins, we adopt it and say so.

Why not just stop at first order? On models with genuine curvature anharmonicity the higher coefficients may not be negligible. The ReLU case in Finite-Width Neural Tangent Kernels from Feynman Diagrams, where leading-order diagonal corrections vanish exactly, shows that first order can miss entire effects. We test this empirically by measuring whether second-order corrections change predictive calibration.

Proposed Experiments

The first check is the lattice reproduction described in the Baselines section: $c_4$ for the plaquette on $8^4$ through $g^4$, with and without gauge fixing.

The neural network tests use models small enough to have exact references. For a rank-one and rank-two factorized regression on synthetic Gaussian data, the posterior predictive can be computed by numerical quadrature over the orbit-reduced coordinates. That gives exact Taylor coefficients in $T$ through order four. The same task is then rerun with a single attention head trained on a synthetic copying task, and with a factorized output layer of a small multilayer perceptron trained on MNIST-1D, where an MLP reaches about 68% accuracy. On MNIST-1D the references come from very long, high-precision tempered runs. The metrics are the error of each order's coefficient against the reference, the growth exponent of each estimator's variance in Langevin time, and the change in expected calibration error when second-order corrections are added to the Laplace predictive.

The decisive checks are three. Without gauge fixing, the order-$k$ variance must grow with run length at the predicted rate, and with gauge fixing it must plateau. Gauge fixing must leave the invariant observable's mean unchanged within error, which tests the no-bias claim directly. And at equal compute, the gauge-fixed estimator must beat temperature extrapolation on the order-3 and order-4 coefficients. If the first check fails, the zero-mode explanation is wrong. If the third fails, the method is a diagnostic rather than a better tool, and we will report it that way.