Abstract
The traditional Hopfield Model (HM) is a cornerstone of associative memory theory, but its reliance on uncorrelated random patterns limits its applicability to structured real-world data. This paper introduces the Random-Features Hopfield Model (RFHM), which generates patterns from a low-dimensional latent space to mimic the "manifold hypothesis." Through theoretical analysis and numerical simulation, the authors identify a novel learning phase transition. In this regime, the network stops merely memorizing individual patterns and begins to retrieve the underlying features used to generate them, demonstrating a natural capacity for unsupervised feature extraction.
Introduction
While the standard Hopfield Model explains how neural systems store and retrieve memories, it assumes patterns are completely random. In reality, data like images or text usually reside on a "hidden manifold"—a lower-dimensional structure within a high-dimensional space. This paper explores what happens when the simple Hebbian learning rule is applied to more realistic, structured data. By introducing the RFHM, the authors bridge the gap between classical associative memory and modern machine learning's focus on feature extraction.
Methods: The Random-Features Hopfield Model
The RFHM generates $P$ binary patterns of length $N$ by starting with Gaussian vectors in a latent space of dimension $D$. These are then projected and passed through a non-linearity. The model is analyzed in the thermodynamic limit using the replica method from statistical physics. Two key order parameters are tracked:
- Pattern Magnetization ($m$): Measures the retrieval of a specific training example (storage).
- Feature Magnetization ($\mu$): Measures the retrieval of the underlying latent features (learning).
Key Results: The Learning Phase Transition
The most striking discovery is the emergence of a "Learning" phase. When the number of patterns ($P$) is sufficiently high and the latent dimension ($D$) is relatively small, the network's attractors shift from individual patterns to the underlying features. This transition is a first-order jump in factor magnetization, as shown in the figure above.
The 0.138 Limit and Phase Competition
The authors found that the critical capacity for feature retrieval mirrors the classic Hopfield capacity ($\alpha_c \approx 0.138$). If the latent space complexity ($\alpha_D = D/N$) exceeds this value, the model fails to learn features and behaves like a standard, uncorrelated Hopfield network. Between the storage and learning phases lies a "spin-glass" region where the model fails to retrieve either patterns or features effectively.
Discussion and Future Work
The RFHM demonstrates that a simple Hebbian system can perform unsupervised feature extraction simply by being exposed to enough correlated examples. However, the current analysis relies on Replica Symmetry (RS), which may break down at very low temperatures or high correlations, requiring a more complex "Full Replica Symmetry Breaking" (FRSB) approach for perfect numerical alignment.
Future research directions include applying this framework to Modern Hopfield Networks (Dense Associative Memories), which utilize exponential interactions similar to the attention mechanisms in Transformers. Investigating the dynamical process of how a system settles into a feature attractor from a corrupted input also remains an open and promising area of study.
Summary updated