Dense Hebbian neural networks: a replica symmetric picture of unsupervised learning

Elena Agliari, Linda Albanese, Francesco Alemanno, Andrea Alessandrelli, Adriano Barra, Fosca Giannotti, Daniele Lotito, Dino Pedreschi

Abstract

We consider dense, associative neural-networks trained with no supervision and we investigate their computational capabilities analytically, via a statistical-mechanics approach, and numerically, via Monte Carlo simulations. In particular, we obtain a phase diagram summarizing their performance as a function of the control parameters such as the quality and quantity of the training dataset and the network storage, valid in the limit of large network size and structureless datasets. Moreover, we establish a bridge between macroscopic observables standardly used in statistical mechanics and loss functions typically used in the machine learning. As technical remarks, from the analytic side, we implement large deviations and stability analysis within Guerra's interpolation to tackle the not-Gaussian distributions involved in the post-synaptic potentials while, from the computational counterpart, we insert Plefka approximation in the Monte Carlo scheme, to speed up the evaluation of the synaptic tensors, overall obtaining a novel and broad approach to investigate neural networks in general.

Introduction

The classic Hopfield model, a cornerstone of associative memory, uses pairwise interactions to store and retrieve patterns. However, it faces significant limitations: a storage capacity restricted to approximately $0.14N$ patterns for $N$ neurons and poor performance under high noise. "Dense" neural networks, or $P$-spin models, utilize higher-order interactions to overcome these constraints. Despite their potential, these models are notoriously difficult to analyze, particularly in unsupervised learning scenarios where they must infer archetypes from noisy examples rather than having patterns directly encoded.

This research bridges the gap between statistical mechanics and machine learning by investigating how many examples a dense network requires to "learn" a concept and how network complexity ($P$) influences its ability to handle a large, noisy workload.

Methodology

The study employs a dual-pronged approach to characterize these networks:

  1. Analytical Framework: Utilizing Guerra’s interpolation technique under the Replica Symmetry (RS) approximation, the authors calculate the "quenched statistical pressure." A significant technical contribution is the extension of standard tools to handle the non-Gaussian local fields inherent in dense networks.
  2. Numerical Simulation: To validate the theory, Monte Carlo simulations were performed. To mitigate the $O(N^P)$ computational cost of updating synaptic tensors, the researchers implemented Plefka’s approximation. This effective dynamics approach tracks the mean values of observables, providing a substantial speedup without sacrificing accuracy.
  3. Theoretical Mapping: The authors established a mathematical bridge showing that dense Hebbian networks are dual to Restricted Boltzmann Machines (RBMs) with Gaussian hidden units, effectively linking Mattis magnetization to standard machine learning loss functions.

Learning Dynamics and the "Data Tax"

A central finding is the "data tax" associated with network complexity. While increasing the interaction order $P$ enhances storage and robustness, it exponentially increases the required training data. The number of examples $M$ needed to learn an archetype scales as $1/(Pr^{2P})$, where $r$ represents data quality.

The transition from "storing" to "learning" is marked by a shift in the energy landscape. Initially, the landscape contains many shallow minima corresponding to specific noisy examples. As $M$ increases, these merge into a single, deep, stable minimum representing the underlying archetype—an "aha moment" for the network.

Visual representation of the 'aha moment' in archetype reconstruction and the corresponding shift in the energy landscape $\mathcal{H}$ as more examples $M$ are processed.

Phase Transitions and Storage Capacity

The performance of these networks is summarized in detailed phase diagrams that delineate regions of Retrieval (successful inference), Spin-Glass (interference and confusion), and Ergodic (pure noise) behavior.

Phase diagrams showing the Ergodic (E), Spin-Glass (SG), and Retrieval (R) regions. The right panel illustrates how the retrieval region expands as the number of training examples $M$ increases.

Key results include:

  • Supra-linear Storage: In high-load regimes, dense networks can store $K \propto N^{P-1}$ archetypes, a massive improvement over the linear scaling of shallow networks.
  • Critical Load Expansion: As the number of training examples $M$ increases, the retrieval region expands, eventually saturating at the theoretical limit of a perfectly trained network.
  • Noise Robustness: Dense networks exhibit superior robustness to both thermal noise and synaptic flaws compared to pairwise models, provided sufficient training data is available.

Limitations and Future Work

While the Replica Symmetry (RS) approximation provides a powerful picture, signatures of Replica Symmetry Breaking (RSB) appear at low temperatures, suggesting a need for more complex future treatments. Additionally, the current theory assumes structureless, uncorrelated data; extending this framework to structured real-world datasets (like images or text) and exploring the behavior of odd-integer interaction orders $P$ remain open challenges.

Summary updated

Abstract

We consider dense, associative neural-networks trained with no supervision and we investigate their computational capabilities analytically, via a statistical-mechanics approach, and numerically, via Monte Carlo simulations. In particular, we obtain a phase diagram summarizing their performance as a function of the control parameters such as the quality and quantity of the training dataset and the network storage, valid in the limit of large network size and structureless datasets. Moreover, we establish a bridge between macroscopic observables standardly used in statistical mechanics and loss functions typically used in the machine learning. As technical remarks, from the analytic side, we implement large deviations and stability analysis within Guerra's interpolation to tackle the not-Gaussian distributions involved in the post-synaptic potentials while, from the computational counterpart, we insert Plefka approximation in the Monte Carlo scheme, to speed up the evaluation of the synaptic tensors, overall obtaining a novel and broad approach to investigate neural networks in general.
BibTeX
@misc{agliari2022dense,
  author = {Elena Agliari and Linda Albanese and Francesco Alemanno and Andrea Alessandrelli and Adriano Barra and Fosca Giannotti and Daniele Lotito and Dino Pedreschi},
  title = {{Dense Hebbian neural networks: a replica symmetric picture of unsupervised learning}},
  year = {2022},
  eprint = {2211.14067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.14067},
}
Figure