The Geometry of Weight Space Symmetries
Imagine training a simple Multi-Layer Perceptron (MLP) with one hidden layer containing two neurons, $A$ and $B$. If you swap the positions of these two neurons—meaning you reorder the rows of the incoming weight matrix and the columns of the outgoing weight matrix accordingly—the mathematical function computed by the network remains completely unchanged. This is the essence of neuron permutation symmetry.
Because a hidden layer with $N$ neurons has $N!$ possible permutations, even a moderately sized network possesses an astronomical number of functionally identical weight configurations. As noted by Samuel Ainsworth et al. in Git Re-Basin: Merging Models modulo Permutation Symmetries, this means that a single function does not correspond to a unique point in weight space, but rather to a massive, discrete set of equivalent points.
Beyond discrete permutations, neural networks exhibit other structural symmetries. For instance, in networks with rectified linear units (ReLU), scaling up the incoming weights to a neuron while scaling down its outgoing weights by the same factor leaves the output unchanged. In Transformers, Binchi Zhang et al. show in Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion that continuous rotation symmetries exist within self-attention layers, where applying an orthogonal matrix transformation preserves the layer's functional behavior.
This structural redundancy creates what is known as parameter degeneracy or architectural degeneracy. As explored by Lucius Bushnaq et al. in Using Degeneracy in the Loss Landscape for Mechanistic Interpretability, this high level of degeneracy means that the specific parameter values found by stochastic gradient descent (SGD) represent only one arbitrary coordinate choice out of many possible implementations.
The Alignment Problem: Why Simple Averaging Fails
The presence of these symmetries introduces a major obstacle when we try to compare, average, or merge different neural networks. Suppose you train two identical architectures independently on the same dataset. Because they start from different random initializations, they will likely assign different internal indices to the features they learn. Neuron #42 in Model A might learn the exact same feature as Neuron #71 in Model B.
If you attempt to perform a simple element-wise weight average of these two models—a common objective in federated learning or model ensembling—you will end up averaging mathematically incompatible features. As Gideon Mann et al. point out in Efficient Large-Scale Distributed Training of Conditional Maximum Entropy Models, while simple weight averaging works well in convex optimization landscapes, it fails in the non-convex landscapes of deep neural networks due to this permutation problem. Averaging unaligned weights destroys the learned features, resulting in a merged model that performs no better than random guessing.
This misalignment creates a high loss barrier when interpolating linearly between two distinct solutions in weight space. However, Rahim Entezari et al. in The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks conjectured that these energy barriers are largely an illusion caused by ignoring permutation invariance. If we can properly align or "unscramble" the internal coordinate systems of the two models, they might actually reside within the same low-loss basin, enabling seamless model merging.
Unscrambling Symmetries via Parameter Matching
To overcome the alignment barrier, researchers have developed algorithms designed to find the optimal permutation matrices that map one model's weight space into another's.
A prominent approach is introduced in Git Re-Basin: Merging Models modulo Permutation Symmetries, where the authors frame model merging as an optimization problem over the discrete group of permutations. By matching neurons across layers using methods like coordinate descent or the Hungarian algorithm, they demonstrate that independently trained models can be "teleported" into a shared basin, allowing them to be linearly interpolated with zero loss in performance.
Similarly, in the context of federated learning, Mikhail Yurochkin et al. propose Probabilistic Federated Neural Matching (PFNM) in Bayesian Nonparametric Federated Learning of Neural Networks. PFNM uses Bayesian nonparametrics to match and aggregate neurons from local models that do not share a common initialization, allowing effective global model aggregation in as little as a single communication round. To handle cases where architectures differ entirely, Disha Makhija et al. introduce FedHeNN in Architecture Agnostic Federated Learning for Neural Networks, which aligns representations using a small, unlabeled dataset and Centered Kernel Alignment (CKA) rather than raw weight matching.
When moving from MLPs to Transformers, discrete permutations are no longer sufficient. Binchi Zhang et al. address this in Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion by developing an SVD-based parameter matching algorithm that accounts for continuous rotation symmetries in self-attention layers, significantly improving the fusion of transformer models.
Beyond merging, exploiting these symmetries can also optimize individual models. For example, Julieta Martinez et al. introduce PQF (Permute, Quantize, and Fine-tune) in Permute, Quantize, and Fine-Tune: Efficient Compression of Neural Networks. Instead of treating permutation invariance as a nuisance, they use it as a degree of freedom to reorder weights into compact, highly clusterable configurations, drastically reducing the error introduced during vector quantization.
Symmetry-Aware Weight Space Learning
A parallel line of research treats the weights of neural networks not as static parameters, but as data inputs for downstream models (often called metanetworks). If you want to train a model to predict a neural network's generalization gap or hyperparameter configuration from its weights, that model must be robust to the network's internal symmetries.
If a metanetwork uses standard self-attention or fully connected layers, it treats the weights as an rigid coordinate grid. As a result, it will fail to recognize that two differently permuted weight matrices represent the exact same function. To solve this, Allan Zhou et al. propose Neural Functional Transformers (NFTs) in Neural Functional Transformers. NFTs restrict attention mechanisms to respect only true neuron permutation symmetries—specifically by attending to weights that share a common neuron—preserving essential architectural structure while maintaining equivariance.
Similarly, Derek Lim demonstrates in Graph Metanetworks for Processing Diverse Neural Architectures that representing neural networks as graphs and processing them via Graph Neural Networks (GNNs) naturally respects permutation equivariance. This prevents the metanetwork from wasting capacity on learning redundant representations of functionally identical configurations.
To train these weight-space models effectively without massive datasets, researchers use symmetry-aware data augmentations. In Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction, Konstantin Schürholt et al. introduce permutation-invariant augmentations that reorder neurons during self-supervised training to build robust "hyper-representations" of weights. Furthermore, Aviv Shamsian et al. develop "Weight-Space Mixup" in Data Augmentations in Deep Weight Spaces, exploiting symmetries within activation functions to generate novel, valid weight configurations that prevent metanetworks from overfitting.
Finally, instead of relying on combinatorial solvers to align models at inference time, Odysseas Boufalis et al. propose a generative approach in Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization. They train symmetry-aware autoencoders to "canonicalize" neural networks, mapping functionally equivalent but misaligned models into a shared, standardized representation space where they can be merged via simple linear interpolation.