Explosive neural networks via higher-order interactions in curved statistical manifolds

Miguel Aguilera, Pablo A. Morales, Fernando E. Rosas, Hideaki ShimazakiarXiv 2024 · DOI · arXiv

Introduction: The Challenge of Higher-Order Interactions

In both biological neural circuits and advanced artificial architectures like Transformers, information processing relies on complex, multi-neuron dependencies known as higher-order interactions (HOIs). In these systems, the state of a group of neurons influences others in ways that cannot be reduced to simple pairwise connections. However, modeling HOIs presents a significant mathematical challenge: the "combinatorial explosion." Accounting for every possible combination of $k$ neurons causes the number of parameters to grow exponentially, making analytical study and practical implementation nearly impossible.

Researchers have typically faced a trade-off: either simplify the models to the point of losing interesting physical behaviors or resort to complex simulations that lack theoretical clarity. This paper introduces "curved neural networks" as a parsimonious solution that captures HOIs of all orders using only a few parameters by leveraging the principles of Information Geometry.

Information Geometry and Curved Manifolds

The core innovation lies in extending the Maximum Entropy Principle (MEP). While standard physics and machine learning models typically use Shannon entropy—leading to the flat geometry of the Boltzmann distribution—this work employs Rényi or Tsallis entropy. This shift "deforms" the statistical manifold, the abstract space representing all possible probability distributions of the network.

Mapping higher-order interactions to curved statistical manifolds.

By introducing a curvature parameter, $\gamma$, the network becomes "curved." A Taylor expansion of the resulting probability distribution reveals that these curved manifolds implicitly incorporate interactions of all orders. When $\gamma = 0$, the model reverts to a standard "flat" Hopfield network. When $\gamma \neq 0$, the geometry itself encodes the complexity of HOIs, allowing the model to remain parsimonious while capturing sophisticated dynamics.

Explosive Retrieval Dynamics and Self-Regulated Annealing

A defining feature of these curved networks is "self-regulated annealing." In classical models, the effective temperature is usually a fixed external parameter. In curved networks, the effective temperature is dynamically coupled to the system's energy. As the network begins to recognize a stored pattern and its energy drops, the effective temperature also decreases (or the inverse temperature $\beta'$ increases).

This coupling creates a positive feedback loop that accelerates the retrieval process. Instead of the gradual transition from disorder to order seen in standard networks, curved networks exhibit "explosive" phase transitions. As shown in the retrieval dynamics, the overlap with a stored memory ($m$) undergoes an abrupt, first-order jump.

Explosive memory retrieval dynamics for different curvature parameters ($\gamma'$).

These transitions are characterized by hysteresis, meaning the network maintains its state even as external conditions fluctuate. This makes the retrieval process significantly more robust to noise compared to traditional associative memories.

Capacity, Robustness, and Modern AI

The researchers validated their theoretical framework using mean-field theory and the replica trick, a tool from the physics of disordered systems. They proved that negative curvature ($\gamma < 0$) significantly extends the memory-retrieval phase, allowing the network to store and retrieve a much higher number of patterns per neuron ($\alpha_c$) than classical Hopfield models.

To test practical relevance, the authors applied the model to binarized images from the CIFAR-100 dataset. The results demonstrated that curved networks excel at denoising and retrieving memories even when the initial cues are highly corrupted, outperforming standard models on real-world, correlated data.

Furthermore, the study establishes a formal link between these curved energy landscapes and the mechanisms underlying state-of-the-art AI, such as the Attention mechanism in Transformers and the denoising processes in Diffusion Models. This provides a grounding in theoretical physics for why these modern architectures are so effective at handling complex data dependencies.

Conclusion and Future Directions

Curved neural networks offer a powerful "middle way" for modeling higher-order interactions without the computational burden of parameter explosion. By baking complexity into the geometry of the statistical manifold, these models achieve explosive retrieval and enhanced storage capacity.

While the current work focuses on networks with fixed weights (Hebbian learning), future research will explore learning algorithms for the curvature parameter $\gamma$ itself. Additionally, investigating whether biological neural circuits implement similar "deformed" exponential dynamics could provide new insights into the efficiency of the human brain. This geometric approach paves the way for more robust and energy-efficient AI architectures grounded in the principles of non-extensive statistical mechanics.

Summary updated

Abstract

Higher-order interactions underlie complex phenomena in systems such as biological and artificial neural networks, but their study is challenging due to the scarcity of tractable models. By leveraging a generalisation of the maximum entropy principle, we introduce curved neural networks as a class of models with a limited number of parameters that are particularly well-suited for studying higher-order phenomena. Through exact mean-field descriptions, we show that these curved neural networks implement a self-regulating annealing process that can accelerate memory retrieval, leading to explosive order-disorder phase transitions with multi-stability and hysteresis effects. Moreover, by analytically exploring their memory-retrieval capacity using the replica trick, we demonstrate that these networks can enhance memory capacity and robustness of retrieval over classical associative-memory networks. Overall, the proposed framework provides parsimonious models amenable to analytical study, revealing higher-order phenomena in complex networks.
BibTeX
@misc{aguilera2024explosive,
  author = {Miguel Aguilera and Pablo A. Morales and Fernando E. Rosas and Hideaki Shimazaki},
  title = {{Explosive neural networks via higher-order interactions in curved statistical manifolds}},
  year = {2024},
  eprint = {2408.02326},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.02326},
}
Figure