Abstract
This paper investigates the retrieval properties of associative memory models utilizing non-monotonic neurons through the application of Two-Parameter Dynamical Replica Theory (2-DRT). Unlike traditional monotonic models, these systems lack thermodynamic potential functions, complicating standard analysis. We demonstrate that 2-DRT accurately reproduces the storage capacity, basin of attraction, and retrieval dynamics observed in simulations. Furthermore, 2-DRT provides a superior estimate of the stability region for "superretrieval" states compared to the conventional Self-Consistent Signal-to-Noise Analysis (SCSNA).
Introduction
In the study of neural networks, associative memory models like the Hopfield network typically employ monotonic activation functions. However, non-monotonic neurons—where the output decreases for large inputs—have been shown to significantly enhance storage capacity and enlarge the basins of attraction. The primary theoretical hurdle is that these networks do not possess a Lyapunov (energy) function, rendering equilibrium statistical mechanics inapplicable. To address this, we employ Two-Parameter Dynamical Replica Theory (2-DRT) to describe the macroscopic asynchronous dynamics of the system.
Methodology: The 2-DRT Framework
The 2-DRT approach simplifies the complex $N$-neuron system by tracking two macroscopic order parameters: the overlap ($m$), representing the similarity to a target memory, and the noise ($r$), representing interference from other stored patterns. The theory relies on two fundamental assumptions: self-averaging and equipartitioning (the system is assumed to be uniformly distributed across all microscopic states consistent with the current macroscopic $m$ and $r$ values).
As seen in the figure above, 2-DRT tracks the evolution of $m$ and $r$ quite accurately across various initial conditions. While a slight "slowing down" effect is predicted by the theory in early stages due to minor violations of the equipartitioning assumption, the overall trajectory matches numerical simulations with $N = 2^{15}$ units remarkably well.
Key Results and Capacity Analysis
A critical finding is that 2-DRT provides a much more accurate prediction of the network's storage capacity ($\alpha$) as a function of the non-monotonicity parameter ($\theta$) than previous methods.
The comparison in the figure above highlights that while the Self-Consistent Signal-to-Noise Analysis (SCSNA) tends to overestimate the capacity, the 2-DRT predictions align closely with numerical simulations. This confirms that 2-DRT is a robust tool even for systems without a thermodynamic potential.
Superretrieval and Stability
The paper also explores "superretrieval" states, where the noise parameter $r$ vanishes over time (proportional to $1/\ln t$), leading to near-perfect recall. While SCSNA predicts the existence of these states, 2-DRT goes further by identifying their local stability conditions. We find that superretrieval is physically observable only within a specific region of the phase space ($2\alpha < \theta < 1$), explaining why these states are not always reached from standard starting points even when they theoretically exist.
Conclusion
2-DRT proves to be a powerful analytical framework for non-monotonic associative memory. It successfully captures the complex dynamics and storage limits of these networks where traditional equilibrium theories fail. Future work will focus on refining the theory to account for non-Markovian effects and potential replica symmetry breaking at high memory loads.
Summary updated