Abstract
Understanding the role of regularization is a central question in statistical inference. Empirically, well-chosen regularization schemes often dramatically improve the quality of the inferred models by avoiding overfitting of the training data. We consider here the particular case of $L_2$ regularization in the maximum a posteriori (MAP) inference of generative pairwise graphical models. Based on analytical calculations on Gaussian multivariate distributions and numerical experiments on Gaussian and Potts models we study the likelihoods of the training, test, and ‘generated data’ (with the inferred models) sets as functions of the regularization strengths. We show in particular that, at its maximum, the test likelihood and the ‘generated’ likelihood, which quantifies the quality of the generated samples, have remarkably close values. The optimal value for the regularization strength is found to be approximately equal to the inverse sum of the squared couplings incoming on sites on the underlying network of interactions. Our results seem to be robust against changes in the structure of the ground-truth underlying interactions that generated the data, when small fluctuations of the posterior distribution around the MAP estimator are taken into account, and when $L_1$ regularization is considered (instead of $L_2$). Connections with empirical works on protein models learned from homologous sequences are discussed.
Introduction
Regularization is a fundamental tool in machine learning and statistical physics used to mitigate overfitting. By introducing a penalty—typically $L_1$ or $L_2$ norms—to the model parameters, one can effectively filter out noise and focus on the underlying structural patterns of the data. While empirical methods like cross-validation are standard for determining the optimal regularization strength ($\gamma$), they often lack a deep theoretical justification. This work explores the theoretical foundations of optimal regularization specifically for generative pairwise graphical models, which are used to model complex systems like protein sequences or physical configurations.
Methodology and Theoretical Framework
The study employs Maximum A Posteriori (MAP) inference to investigate how models are inferred under the assumption that interaction strengths should be small. The researchers utilize Gaussian multivariate distributions to derive exact analytical solutions in both the large-sample and single-sample limits. The key variables and symbols used in these derivations are summarized in the table below.
The core of the analysis involves tracking three distinct log-likelihoods as functions of the regularization strength $\gamma$:
- Training Likelihood ($L_{train}$): Measures the model's fit to the observed training data.
- Test Likelihood ($L_{test}$): Evaluates the model's predictive power on unseen data.
- Generated Likelihood ($L_{gen}$): Assesses how well the model fits data it has generated itself.
To validate these findings beyond Gaussian systems, numerical experiments were conducted on discrete-valued Potts models using Pseudo-Likelihood Maximization (PLM) and Adaptive Cluster Expansion (ACE).
The "Crossing" Coincidence
A central discovery of this research is the remarkable alignment between the optimal test performance and the model's internal consistency. As shown in Figure 1, the regularization strength that maximizes the test likelihood ($\gamma^{opt}$) is nearly identical to the point where the test likelihood and generated likelihood curves intersect ($\gamma^{cross}$).
This "crossing coincidence" suggests that a perfectly tuned generative model perceives its own generated samples as being just as realistic as unseen real-world data. Analytically, the authors demonstrate that in the limits of infinite data ($\alpha \to \infty$) and a single sample ($\alpha \to 0$), $\gamma^{opt}$ and $\gamma^{cross}$ converge to the same value.
The Inverse Strength Rule and Universality
For large datasets, the researchers derived a simple rule for the optimal regularization strength: $$\gamma_{opt} \approx \frac{n}{\sum_{i,j} (J_{ij}^{tr})^2}$$ In plain English: the stronger the true interactions in the system, the weaker the regularization you should apply. Stronger signals are harder for noise to corrupt, so you need less "smoothing." This relationship proved robust across different network topologies and even qualitatively held when transitioning from $L_2$ to $L_1$ regularization.
Conclusion and Future Directions
The findings provide a theoretical basis for empirical scalings used in fields like protein sequence modeling, where $\gamma$ is often scaled by the number of interacting neighbors. However, several challenges remain. Real biological data often contains "phylogenetic noise" due to evolutionary relationships, which violates the independent-sample assumption. Future work will aim to extend these analytical results to $L_1$ regularization and higher-order interaction models, such as Restricted Boltzmann Machines.
Summary updated