Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings

Francesco D'Amico, Dario Bocchi, Luca Maria Del Bono, Saverio Rossi, Matteo Negri

TL;DR

Maximizing pseudo-likelihood in energy-based models at zero temperature naturally implements an associative memory with large basins of attraction, even when the learned network couplings are asymmetric. As the training dataset size grows, the network transitions from simple memorization to a generalization regime where it forms stable attractors near unseen test data.

What Problem Does This Paper Address?

Energy-based probabilistic models (like Boltzmann machines) are powerful tools for learning data distributions, but training them via exact maximum likelihood is notoriously difficult. The culprit is the global normalization factor—the partition function—which requires summing over an exponential number of states and is computationally intractable.

Pseudo-likelihood (PL) is a popular, computationally efficient workaround that replaces global normalization with tractable local conditional probabilities. While PL is widely used in practice (for example, in predicting protein structures), we lack a deep theoretical understanding of why it works so well, and how its optimization process relates to overfitting, memorization, and generalization. This paper bridges that gap by connecting PL training directly to the physics of Associative Memories (AMs), such as Hopfield Networks.

Methods

The authors analyze a standard energy-based model with two-body interactions and binary variables ($x_i \in \{\pm 1\}$) trained by minimizing the negative log-pseudo-likelihood (NLpL) loss.

To study the network's behavior, they look at its recurrent dynamics in the zero-temperature limit ($\lambda \to \infty$). This turns a probabilistic sampling process into a deterministic, discrete-time dynamical system where nodes update based on the sign of their local fields. Crucially, the authors do not force the learned coupling matrix $J$ to be symmetric ($J_{ij} \neq J_{ji}$). They leverage the statistical mechanics theory of spherical perceptrons to analytically derive the distribution of classification margins and explain the network's storage capacity.

Datasets

The authors evaluate their approach across a diverse set of synthetic and real-world datasets spanning computer science, physics, and biology:

  • Uncorrelated Synthetic Data: Random binary patterns generated via a symmetric Rademacher distribution.
  • Correlated Synthetic Data: A random feature model designed to mimic the "Hidden Manifold Hypothesis" of real-world data.
  • MNIST: The classic handwritten digits dataset, binarized and downsampled to $14 \times 14$ pixels.
  • Protein Sequences: Natural amino acid sequences from two protein families: the DNA-Binding Domain (DBD) and Beta Lactamase.
  • Spin Glasses: Equilibrium configurations sampled from a 2D Edwards-Anderson spin glass model.

Key Insights and Ideas

  • PL as a Max-Margin Classifier: Minimizing the pseudo-likelihood loss is mathematically equivalent to training $N$ independent perceptrons (one for each variable). Because of the implicit bias of the logistic-like loss, the training naturally maximizes the classification margins. This maximization is precisely what creates stable fixed points with large basins of attraction.
FIG. 1. Maximizing pseudo-likelihood goes from Hebbian learning to perceptrons with maximum margins. Panel (a): Comparison between the analytical distribution at fixed λ (blue lines) and the numerical distribution with free norm at λfree(t) = λ (orange histogram), for different loads α and training epochs. While the two distributions come from different settings and do not exactly match, key properties for retrieval such as the mean, minimum, and variance vary similarly as λ increases. Panel (b): The distribution of margins in the perceptron spherical perceptron trained maximizing pseudo-likelihood, as a function of the parameter λ. The load is fixed to α = 0.4.
  • Asymmetry is a Benefit: Traditional Hopfield networks require symmetric couplings to guarantee a stable energy landscape. The authors show that PL-trained networks retain their natural asymmetry but still function as highly effective associative memories, often achieving larger basins of attraction than symmetric models.
  • The Memorization-to-Generalization Transition: By adjusting the model "load" $\alpha = P/N$ (the ratio of training examples to system size), the authors uncover a clear phase transition. At low loads, the network overfits and perfectly memorizes training samples as fixed points. At higher loads, individual training samples stop being fixed points; instead, the network develops collective attractors that represent the underlying data distribution, allowing it to successfully denoise completely unseen test data.

Key Results

  • Higher Capacity: For uncorrelated data, the PL-trained network achieves a storage capacity ($\alpha \approx 1$) well above the classical Hopfield limit ($\alpha \approx 0.14$).
  • Asymmetric Advantage: Asymmetric couplings yield larger basins of attraction than symmetric ones at higher network loads, meaning they are more robust to noise when retrieving memories.
Comparison of basin sizes between asymmetric and symmetric couplings across different model loads.
  • Real-World Generalization: On the MNIST dataset, as the training set size increases, both training and test images converge to fixed points with a high final overlap ($\sim 0.85$), showing that the network learns the prototypical structure of digits rather than just memorizing specific pixels.
  • Biological Relevance: For protein sequences, the zero-temperature dynamics successfully converge to stable configurations that share a high overlap ($\sim 0.55 - 0.60$) with natural test sequences, effectively capturing the valid evolutionary space of the protein family.

Limitations and Future Work

  • Zero-Temperature Limit: The core theoretical link established in this paper relies on the zero-temperature limit. Real-world applications of energy-based models operate at finite temperatures to sample diverse data. Extending this framework to analyze how these basins of attraction behave under thermal noise is a vital next step.
  • Simple Architectures: The study is limited to pairwise interactions without hidden units. To explain modern deep learning architectures (like Restricted Boltzmann Machines or Diffusion Models), the theory needs to be extended to include hidden variables, which complicates the landscape analysis.
  • Maximum Likelihood Comparison: It remains an open question whether exact maximum likelihood training shares this exact same implicit bias toward associative memory formation, or if this elegant behavior is a unique side-effect of the pseudo-likelihood approximation.

Summary updated

Abstract

Energy-based probabilistic models learned by maximizing the likelihood of the data are limited by the intractability of the partition function. A widely used workaround is to maximize the pseudo-likelihood, which replaces the global normalization with tractable local normalizations. Here we show that, in the zero-temperature limit, a network trained to maximize pseudo-likelihood naturally implements an associative memory: if the training set is small, patterns become fixed-point attractors whose basins of attraction exceed those of any classical Hopfield rule. We explain quantitatively this effect on uncorrelated random patterns. Moreover, we show that, for different structured datasets coming from computer science (random feature model, MNIST), physics (spin glasses) and biology (proteins), as the number of training examples increases the learned network goes beyond memorization, developing meaningful attractors with non-trivial correlations with test examples, thus showing the ability to generalize. Our results therefore reveal pseudo-likelihood works both as an efficient inference tool and as a principled mechanism for memory and generalization.
BibTeX
@misc{damico2025pseudo,
  author = {Francesco D'Amico and Dario Bocchi and Luca Maria Del Bono and Saverio Rossi and Matteo Negri},
  title = {{Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings}},
  year = {2025},
  eprint = {2507.05147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.05147},
}
Figure