Aligning Neural Weights via Permutation Symmetries

277 concepts135 papers

When we train two neural networks with the same architecture on the same dataset, they often arrive at very different weight values. If we try to average these weights directly, the resulting model typically performs no better than random noise. This phenomenon, known as the interpolation barrier, suggests that neural networks reside in isolated "valleys" of the loss landscape.

However, recent research suggests these barriers are largely an illusion caused by permutation symmetry. Because the internal neurons of a layer can be reordered without changing the network's output, two models can be functionally identical yet numerically distant. By "unscrambling" these permutations, we can align independently trained models into a single, shared basin.

The Geometry of Permutation Invariance

A neural network is invariant to the ordering of its hidden units. If you swap two neurons in a hidden layer and rearrange their corresponding incoming and outgoing weights, the mathematical function $f(x)$ remains exactly the same. Entezari et al. argue that the high-loss barriers observed during linear interpolation are primarily artifacts of this symmetry. They conjecture that if we properly align the neurons of two models, they actually reside in the same low-loss basin.

This idea, popularized as the Git Re-Basin hypothesis by Ainsworth et al., suggests that the loss landscape of deep neural networks is much simpler than previously thought. By permuting the weights of one model to match the "basis" of another, we can achieve Linear Mode Connectivity (LMC)—a state where a straight line in weight space connects two different minima without ever crossing a region of high loss.

Mechanisms for Neuron Alignment

To align two models, we must find a permutation matrix $P$ for each layer that minimizes the distance between their weights. Researchers have developed several strategies to solve this combinatorial problem:

  • Weight-Based Matching: This approach treats alignment as a linear assignment problem. Ainsworth et al. use iterative coordinate descent to greedily optimize permutations layer-by-layer. Similarly, Zhan et al. show that sorting neurons by weight norms can serve as a powerful heuristic for reducing the interpolation barrier.
  • Activation-Based Matching: Instead of looking at weights, we can align neurons based on their behavior. Tatro et al. and Jordan propose matching neurons that exhibit similar firing patterns or correlations when presented with a shared dataset.
  • Optimal Transport (OT): To handle cases where a strict one-to-one mapping is too rigid, Imfeld et al. and Ferbach et al. formulate alignment as an optimal transport problem. This allows for "soft alignment," where a neuron in one model can be mapped to a weighted combination of neurons in another, providing the flexibility needed for complex architectures like Transformers.

Why Alignment Works

The intuition behind alignment is that over-parameterized models, despite different initializations, tend to learn a consistent set of features. Chijiwa demonstrates that even the trajectories of training runs become similar once aligned.

Theoretical work by Ferbach et al. and Zhan et al. suggests that as network width increases, the probability of finding a low-loss path between two permuted models approaches one. Essentially, in high-dimensional space, there is enough "room" to rotate and permute one model's weights until they align with the functional structure of another.

Applications Across the AI Ecosystem

Weight alignment is not just a theoretical curiosity; it enables a new suite of practical techniques:

  • Model Merging: We can combine specialized models (e.g., one trained on vision, another on text) into a single multi-task model without expensive retraining. Xu et al. and the authors of $C^2M^3$ use global alignment to merge multiple models while maintaining cycle consistency.
  • Federated Learning: In decentralized training, clients often have non-identical data, leading to "weight divergence." Yurochkin et al. and Su et al. use neuron matching to aggregate local models more effectively than simple averaging.
  • Weight Space Learning: If we want to train a "meta-model" that learns from the weights of other networks, we must first put those weights into a canonical form. Shamsian et al. and Schürholt show that alignment-aware representations are crucial for tasks like predicting model accuracy or generating new weights via diffusion.
  • Continual Learning: To prevent "catastrophic forgetting," Wasilewski et al. and Guerrero Peña use permutation-based interpolation to merge a model's current state with its previous state, preserving old knowledge while integrating new tasks.

By treating the discrete symmetries of neural networks as a bridge rather than a barrier, permutation alignment allows us to treat trained parameters as flexible, combinable components rather than isolated black boxes.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 163 papers
  2. Version 1 · 2026-10-04 · 159 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_permutation_based_weight_alignment_adca1996,
  title = {permutation-based weight alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/permutation-based-weight-alignment/rdr_adca1996521f48c08a9e2382b10b1268},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_permutation_based_weight_alignment_adca1996_v2,
  title = {permutation-based weight alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/permutation-based-weight-alignment/rdr_adca1996521f48c08a9e2382b10b1268/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake