Abstract
Understanding how neural networks learn and optimize remains a central point in machine learning, with implications for designing better models. While techniques like dropout and batch normalization are widely used, the underlying principles driving their success—such as symmetry breaking, a concept in physics—are underexplored. We propose the symmetry breaking hypothesis, showing that breaking symmetries during training (e.g., via input expansion) substantially improves performance across tasks. We develop a metric to quantify symmetry breaking in networks, revealing its role in common optimization methods and its connection to properties like equivariance. This metric offers a practical tool to evaluate architectures without exhaustive training or full datasets, enabling more efficient design choices. Our work positions symmetry breaking as a unifying principle behind optimization techniques, bridging theoretical gaps and providing actionable insights for improving model efficiency.
Introduction
Despite the unprecedented success of deep learning, many optimization techniques are often regarded as "black magic." While empirical evidence shows that methods like Batch Normalization or Dropout facilitate convergence, the underlying physical principles governing these successes remain largely qualitative. A significant hurdle in neural network training is the presence of symmetries and degenerate states—regions in the loss landscape where multiple parameter configurations yield identical outputs. These regions create flat plateaus and saddle points that impede gradient flow.
This research introduces the symmetry breaking hypothesis, proposing that effective optimization requires disrupting these degenerate states. Drawing a parallel to statistical physics, where systems like magnets must "choose" a direction to break energetic symmetry, we argue that neural networks must similarly break symmetry to achieve efficient learning.
Methodology
The study employs three primary methodological pillars to explore and validate the symmetry breaking hypothesis:
1. Input Dimension Expansion
The core experimental lever involves expanding input dimensions (e.g., pixels or text embeddings) by padding them with constant values, such as the data mean or a fixed value like 0.5. To ensure that performance gains are not merely a result of increased model capacity, the researchers often adjust the architecture to maintain a nearly identical total parameter count compared to baseline models.
2. Theoretical Mapping
The researchers establish a mathematical link between the Ising model—a fundamental model in statistical physics for magnetism—and Hopfield Networks, extending this connection to modern architectures like Transformers and CNNs. They demonstrate that adding a constant bias to the input is mathematically analogous to applying an external magnetic field to an Ising model, which compels the system to break its symmetry and settle into a specific state.
3. The Replica Distance (RD) Metric
Inspired by the "Parisi scheme" used in the study of spin glasses, the authors developed the Replica Distance (RD) metric to quantify symmetry breaking. By training multiple "replicas" of a model with different random seeds and projecting their high-dimensional weights into a lower-dimensional space using UMAP, the researchers calculate the Wasserstein distance between weight distributions. A higher RD indicates that the models have diverged into distinct, specialized configurations, signifying successful symmetry breaking.
Results and Discussion
Performance Across Domains
The hypothesis was tested across a diverse array of tasks, yielding significant results:
- Image Classification: On CIFAR-10, expanding input dimensions by a factor of 4 allowed a ResNet-18 to achieve 94.74% accuracy, a performance level typically associated with the much larger ResNet-101.
- Scientific Computing: In solving Partial Differential Equations (PDEs) via the PINNacle benchmark, 75% of tested equations showed lower loss with expanded inputs. In Quantum Chromodynamics (QCD) tasks, error rates in predicting pressure and energy density dropped significantly.
- Generative and NLP Tasks: Improvements were also observed in image coloring (CelebA) and sentiment analysis (IMDB).
Insights into Optimization
The study suggests that "symmetry is a trap." In a perfectly symmetric landscape, the model lacks direction. "Polluting" the input with constants creates a preferred direction in high-dimensional space, smoothing the loss landscape. Furthermore, the authors argue that standard techniques like Dropout and Batch Normalization are effectively symmetry-breaking tools—Dropout by randomly removing neurons to split degenerate states, and BatchNorm by shifting internal distributions to prevent redundant configurations.
Efficiency and Equivariance
A key finding is that embedding correct symmetries (equivariance) is more efficient than simple scaling. For instance, a Transformer model with doubled input embedding size but halved hidden layers achieved a 6.56% accuracy boost while utilizing fewer total parameters. However, the authors caution that embedding incorrect symmetries can hinder the model's ability to reach an optimal solution.
Conclusion
Symmetry breaking provides a unifying lens through which to view neural network optimization. While the method of input expansion shows diminishing returns beyond a certain point (typically 4–6 times the original spatial dimensions), it offers a powerful tool for enhancing model efficiency. The Replica Distance metric further provides a practical means to evaluate architectural choices without exhaustive training. Future work will focus on adaptive expansion protocols and refining the RD metric to further bridge the gap between theoretical physics and practical machine learning.
Summary updated