Heavy-tailed update distributions arise from information-driven self-organization in nonequilibrium learning

Xin-Ya Zhang, Chengchun TangProceedings of the National Academy of Sciences 2025 · DOI

TL;DR

This paper reveals that neural network training, far from equilibrium, exhibits a fundamental self-organization principle. It demonstrates that heavy-tailed update distributions, a signature of criticality, naturally emerge from a balance between maximizing exploration (maximum entropy) and ensuring task relevance (mutual information), offering a new perspective on AI learning dynamics.

What Problem Does This Paper Address?

While artificial neural networks (ANNs) have achieved remarkable success, the underlying dynamics of their training process remain somewhat mysterious. Much research focuses on initializations and final model states, but the iterative updates that drive learning are less understood. This paper tackles the question of whether there are fundamental principles governing these parameter updates during training, particularly in the context of ANNs operating in a "nonequilibrium" state due to continuous data injection and parameter reorganization. The authors aim to uncover universal statistical properties of these updates and explain their origin.

Methods

The researchers propose a theoretical framework based on information-driven self-organization. They posit that learning involves a trade-off between two principles:

  1. Maximum Entropy Principle: Encourages unbiased exploration of the parameter space.
  2. Mutual Information Constraint: Ensures that parameter updates are relevant to the task objective by relating them to neuron activations.

They derive that this trade-off naturally leads to heavy-tailed (power-law) distributions for parameter updates. To validate this, they:

  • Numerically analyze parameter updates: They track the absolute magnitudes of parameter updates across various neural network architectures (MLPs, CNNs, Transformers), datasets (MNIST, CIFAR-10), and training stages.
  • Perform perturbation analysis: To understand the intrinsic geometry of the loss landscape without the influence of mini-batch noise, they perturb parameters of well-trained models and observe the resulting loss changes.
  • Analyze temporal dynamics: They investigate the timing of large parameter updates, drawing parallels to phenomena like earthquakes.

Datasets

  • MNIST: A dataset of grayscale handwritten digits (0-9) used for training MLPs.
  • CIFAR-10: A dataset of color images across 10 object categories used for training CNNs.

Key Insights and Ideas

  • Learning as Nonequilibrium Self-Organization: The paper frames neural network training not as a static optimization problem but as a dynamic, nonequilibrium process governed by fundamental statistical principles.
  • The "Curiosity vs. Goal" Trade-off: The core idea is that learning balances the need to explore broadly (like curiosity) with the need to adapt to the specific task (goal-driven behavior). This is mathematically captured by the maximum entropy and mutual information principles.
  • Heavy Tails as a Universal Signature: The consistent emergence of heavy-tailed update distributions across different models and training conditions suggests this is a fundamental property of gradient-based learning, not an artifact of specific choices.
  • Multiscale Ruggedness of Loss Landscapes: The loss landscape isn't uniformly smooth or rough; it exhibits a complex geometry that is locally smooth but becomes rugged at larger scales, influencing how SGD navigates it.
  • Intermittent Learning: The timing of large updates isn't random but clustered, suggesting that learning progresses in bursts, similar to critical phenomena in physics.

Key Results

  • Heavy-Tailed Update Distributions: Empirical analysis consistently shows that the distribution of parameter update magnitudes follows a power-law, $p(|\Delta\omega|) \propto |\Delta\omega|^{-\alpha}$, with the exponent $\alpha$ remaining stable throughout training.
  • Stable Power-Law Exponent: The derived power-law exponent $\alpha$ is shown to be a function of mutual information and gradient directionality, and empirical proxies for these quantities suggest $\alpha$ remains relatively invariant during training.
  • Multiscale Loss Landscape: Perturbation analysis reveals that the loss landscape exhibits exponential decay in loss change for small parameter perturbations (indicating local smoothness) and transitions to power-law scaling for larger perturbations (indicating scale-invariant ruggedness).
  • Power-Law Inter-Update Intervals: The time intervals between large parameter updates also follow a power-law distribution, with exponents around 2.5-2.7 for MLPs and CNNs, respectively.

Limitations and Future Work

  • Adaptive Optimizers: The current theoretical framework is primarily developed for Stochastic Gradient Descent (SGD). The authors suggest extending the analysis to adaptive optimizers (like Adam) to understand how their per-parameter rescaling might alter the information-entropy trade-off and potentially explain why SGD might generalize better.
  • Time-Resolved Trajectory Modeling: While local update statistics are analyzed, incorporating time-resolved trajectory modeling could provide deeper insights into how these local dynamics lead to global learning behavior.
  • Generalization of Heavy Tails: While the paper shows heavy tails across various architectures and tasks, further exploration into whether other heavy-tailed forms beyond power-laws might also fit the data is acknowledged.
  • Interpretability: The findings offer a new lens for understanding AI interpretability, but further work is needed to translate these statistical insights into practical methods for building more interpretable AI systems.

Summary updated

Abstract

Like human decision-making under real-world constraints, artificial neural networks may balance free exploration in parameter space with task-relevant adaptation. In this study, we identify consistent signatures of criticality during neural network training and provide theoretical evidence that such scaling behavior arises naturally from information-driven self-organization: a dynamic balance between the maximum entropy principle that promotes unbiased exploration and mutual information constraint that relates updates with task objective. We numerically demonstrate that the power-law exponent of updates remains stable throughout training, supporting the presence of self-organized criticality. Furthermore, we show that the loss landscape exhibits exponential ruggedness under small perturbations, transitioning to power-law ruggedness at larger scales, in the absence of mini-batch noise, indicating an intrinsic geometric landscape. We also observe a power-law distribution in the intervals between large updates, indicating an intermittent learning process. Together, these findings suggest that neural network learning reflects a nonequilibrium process governed by the fundamental trade-off between randomness and relevance, highlighting its dynamic nature and offering insights into the interpretability of AI systems.
BibTeX
@inproceedings{zhang2025heavy,
  author = {Xin-Ya Zhang and Chengchun Tang},
  title = {{Heavy-tailed update distributions arise from information-driven self-organization in nonequilibrium learning}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2025},
  doi = {10.1073/pnas.2523012122},
}
Figure