Large pseudocounts and<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mi>L</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:math>-norm penalties are necessary for the mean-field inference of Ising and Potts models

John P. Barton, Simona Cocco, Eleonora De Leonardis, Rémi MonassonPhysical 2014 · DOI · arXiv

Abstract

The mean-field (MF) approximation provides a computationally efficient method for inferring interactions in networks of correlated variables. However, its performance relies heavily on strong regularization, a requirement often misunderstood as a mere defense against sampling noise. This study demonstrates that large regularization—specifically pseudocounts and $L_2$-norm penalties—is fundamentally necessary to correct systematic biases inherent in the MF approximation itself. Even in the limit of infinite data, a finite regularization strength is required to achieve optimal inference.

The Systematic Bias of Mean-Field Inference

Inferring the underlying interactions ($J$) from observed correlations is the core of the "Inverse Ising" and "Inverse Potts" problems. While the Mean-Field (MF) approximation is favored for its speed, it assumes variables follow a Gaussian distribution. Because binary spins (Ising) or categorical states (Potts) are non-Gaussian, the MF approach systematically overestimates strong couplings.

Mean-field approximation $J^{MF}$ vs. true coupling $J$.

As shown in the figure above, the inferred coupling $J^{MF}$ deviates significantly from the true coupling $J$ as the interaction strength increases. This mathematical flaw persists regardless of the amount of data collected, necessitating a permanent "correction factor" in the form of regularization.

Regularization as a Necessary Correction

Standard statistical theory suggests that regularization should vanish as the number of samples ($B$) increases. However, in MF inference, the optimal regularization strength remains finite even as $B \to \infty$. The authors explore two primary schemes:

  1. Pseudocounts: Adding a uniform "fake" background to the observed data.
  2. $L_2$-norm penalties: Penalizing the magnitude of the interaction weights.

Through analytical study of the $m$-component spin model and numerical simulations of 1D lattices and random graphs, the researchers found that pseudocounts are particularly robust. They effectively "stretch" the inference where it is most needed, outperforming $L_2$ penalties in heterogeneous networks where node activity varies widely.

Key Results and Performance

The study reveals that for Ising models, a pseudocount intensity of $\alpha \approx 0.2$ is consistently optimal. In Potts models, which are significantly harder to infer, the required strength increases with the number of states ($q$), sometimes reaching $\alpha \approx 0.75$ for $q=20$.

Relative square error as a function of sampling depth for different pseudocount strengths $\alpha$.

The error analysis demonstrates that without regularization ($\alpha=0$), the relative square error for strong couplings ($J=2$) remains high even with many samples. Applying an optimal pseudocount ($\alpha=0.2$) drastically reduces this error and stabilizes the inference against sampling noise. This stability is crucial for practical applications, such as predicting protein-protein contacts, where the relative rank of interactions is more important than their absolute values.

Conclusion

The necessity of "abnormally strong" regularization in mean-field inference is not a failure of data collection but a requirement for correcting the approximation's intrinsic Gaussian bias. Pseudocounts offer a superior regularization strategy for complex, heterogeneous systems. While the Ising model is well-handled by these methods, Potts models remain challenging, particularly for highly conserved or extremely rare states, suggesting a need for future development of site-specific or link-dependent penalties.

Summary updated

Abstract

The mean-field (MF) approximation offers a simple, fast way to infer direct interactions between elements in a network of correlated variables, a common, computationally challenging problem with practical applications in fields ranging from physics and biology to the social sciences. However, MF methods achieve their best performance with strong regularization, well beyond Bayesian expectations, an empirical fact that is poorly understood. In this work, we study the influence of pseudocount and L(2)-norm regularization schemes on the quality of inferred Ising or Potts interaction networks from correlation data within the MF approximation. We argue, based on the analysis of small systems, that the optimal value of the regularization strength remains finite even if the sampling noise tends to zero, in order to correct for systematic biases introduced by the MF approximation. Our claim is corroborated by extensive numerical studies of diverse model systems and by the analytical study of the m-component spin model for large but finite m. Additionally, we find that pseudocount regularization is robust against sampling noise and often outperforms L(2)-norm regularization, particularly when the underlying network of interactions is strongly heterogeneous. Much better performances are generally obtained for the Ising model than for the Potts model, for which only couplings incoming onto medium-frequency symbols are reliably inferred.
BibTeX
@inproceedings{barton2014large,
  author = {John P. Barton and Simona Cocco and Eleonora De Leonardis and Rémi Monasson},
  title = {{Large pseudocounts and<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mi>L</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:math>-norm penalties are necessary for the mean-field inference of Ising and Potts models}},
  booktitle = {Physical},
  year = {2014},
  doi = {10.1103/physreve.90.012132},
}

Loading related content...

Figure