Dense Hebbian Neural Networks: A Replica Symmetric Picture of Supervised Learning

Adriano Barra, Elena Agliari, Linda Albanese, Francesco Alemanno, Andrea Alessandrelli, Fosca Giannotti, Daniele Lotito, Dino PedreschiSSRN 2022

Abstract

This paper investigates dense, associative neural networks trained with supervision. Using the statistical mechanics of spin glasses and Monte Carlo simulations, we derive a phase diagram for network performance based on dataset quality, quantity, storage, and noise. We demonstrate that these networks can operate in "ultra-storage" or "ultra-detection" regimes, significantly outperforming shallow networks. Our analytical approach extends Guerra's interpolation to non-Gaussian distributions, while our computational framework utilizes Plefka's approximation to map dense interactions onto an equivalent Restricted Boltzmann Machine (RBM) structure.

Introduction

Standard associative memories, such as the classic Hopfield network, typically rely on pairwise interactions ($P=2$). While foundational, these models suffer from limited storage capacity, scaling linearly with the number of neurons (roughly $0.14N$). "Dense" neural networks, which utilize higher-order interactions ($P > 2$), offer a path toward significantly higher capacities. However, most research has focused on the retrieval of pre-stored information rather than the supervised learning of patterns from noisy data.

This paper provides a rigorous statistical mechanics framework for dense Hebbian networks operating in a supervised mode. The authors demonstrate that the introduction of a "teacher" (supervision) allows these networks to achieve "ultra-storage" and "ultra-tolerance" while drastically reducing the data requirements compared to unsupervised versions.

Theoretical Framework: Overcoming the Complexity Barrier

Analyzing dense networks presents significant challenges. Analytically, the internal fields (post-synaptic potentials) are non-Gaussian, making standard statistical mechanics tools like Guerra’s interpolation difficult to apply. Computationally, updating and evaluating $P$-order synaptic tensors is exponentially expensive.

To address these, the authors extended Guerra’s Interpolation to handle non-Gaussian distributions, proving they effectively behave as Gaussians in the large-network limit. Furthermore, they utilized Plefka’s Approximation to facilitate simulations. This approximation reveals a mathematical duality: a dense Hebbian network can be mapped to a Restricted Boltzmann Machine (RBM) where hidden neurons function as "grandmother cells" dedicated to specific archetypes.

Equivalence between a dense Hebbian neural network (left) and a Restricted Boltzmann Machine (right) via Plefka's approximation.

This mapping allows the network's dynamics to be simulated efficiently by avoiding the explicit calculation of massive $P$-order synaptic tensors, instead utilizing the bipartite structure of the RBM.

Key Insights: The Supervision Dividend

A primary finding of this work is the "Supervision Dividend." In unsupervised dense networks, the number of training examples $M$ required for successful learning scales as $r^{-2P}$ (where $r$ represents dataset quality). With supervision, this scaling improves to $r^{-2}$, becoming independent of the interaction order $P$. This makes high-order interactions practically "free" regarding data requirements.

The research identifies two distinct operational regimes:

  • Ultra-Storage Regime: In the high-load limit, the network can store a number of patterns scaling as $N^{P-1}$.
  • Ultra-Tolerance Regime: In the low-load limit, the network exhibits extreme robustness, recognizing patterns even when training data is heavily obscured by noise.

Empirical Validation

The theoretical phase diagrams, which delineate regions of successful retrieval, spin-glass confusion, and failure, were validated using both synthetic and real-world datasets.

  • Synthetic Data: Used to confirm analytical proofs regarding critical loads and stability.
  • MNist and Fashion MNist: Testing on these structured datasets confirmed that the benefits of higher-order interactions ($P > 2$) persist in real-world scenarios. Accuracy improved as $P$ increased, eventually reaching a performance plateau, demonstrating that the "random" theory provides a robust reference even for structured data.

Conclusion and Future Directions

While the current work assumes Replica Symmetry (RS), the authors note that certain features of the phase diagrams may be artifacts of this assumption. Future research will explore Replica Symmetry Breaking (RSB) for higher precision. Additionally, moving beyond "structureless" data to account for the intrinsic manifold of real-world datasets remains a key goal.

Finally, this framework contributes to the field of Sustainable AI. By understanding the phase diagrams of these networks, researchers can determine the minimal architecture (optimal $P$ and $N$) required for a task, optimizing energy and computational efficiency.

Summary updated

Abstract

We consider dense, associative neural-networks trained by a teacher (i.e., with supervision) and we investigate their computational capabilities analytically, via statistical-mechanics of spin glasses, and numerically, via Monte Carlo simulations. In particular, we obtain a phase diagram summarizing their performance as a function of the control parameters such as quality and quantity of the training dataset, network storage and noise, that is valid in the limit of large network size and structureless datasets: these networks may work in a ultra-storage regime (where they can handle a huge amount of patterns, if compared with shallow neural networks) or in a ultra-detection regime (where they can perform pattern recognition at prohibitive signal-to-noise ratios, if compared with shallow neural networks). Guided by the random theory as a reference framework, we also test numerically learning, storing and retrieval capabilities shown by these networks on structured datasets as MNist and Fashion MNist. As technical remarks, from the analytic side, we implement large deviations and stability analysis within Guerra's interpolation to tackle the not-Gaussian distributions involved in the post-synaptic potentials while, from the computational counterpart, we insert Plefka approximation in the Monte Carlo scheme, to speed up the evaluation of the synaptic tensors, overall obtaining a novel and broad approach to investigate supervised learning in neural networks, beyond the shallow limit, in general.
BibTeX
@misc{barra2022dense,
  author = {Adriano Barra and Elena Agliari and Linda Albanese and Francesco Alemanno and Andrea Alessandrelli and Fosca Giannotti and Daniele Lotito and Dino Pedreschi},
  title = {{Dense Hebbian Neural Networks: A Replica Symmetric Picture of Supervised Learning}},
  year = {2022},
  eprint = {2212.00606},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.00606},
  howpublished = {SSRN},
}
Figure