Reconstruction of Pairwise Interactions using Energy-Based Models

Christoph Feinauer, Carlo Lucibello

Abstract

Inferring pairwise interactions from data is a fundamental task in statistical mechanics, yet real-world datasets often contain higher-order correlations that bias standard pairwise models. We present a hybrid Energy-Based Model (EBM) that decomposes the system energy into an interpretable pairwise component and a flexible neural network component. This architecture allows the neural network to capture complex higher-order noise, enabling more accurate reconstruction of the underlying physical couplings. We demonstrate the effectiveness of this approach on synthetic Ising-like systems and protein sequence data, showing that the hybrid model consistently outperforms standard pairwise and pure neural network approaches in recovering true interaction strengths.

Introduction

In fields ranging from structural biology to condensed matter physics, researchers frequently employ pairwise models, such as the Ising or Potts models, to characterize the interactions between system components. The goal of inverse statistical mechanics is to infer these interaction strengths (couplings) from observed data. However, real-world systems often exhibit higher-order interactions involving three or more variables. When a simple pairwise model is fitted to such data, it often produces "effective" interactions that are distorted by these higher-order effects, obscuring the true underlying physics. This paper addresses the challenge of extracting clean pairwise information from data contaminated by complex, high-order noise.

Methods

The authors propose a hybrid Energy-Based Model (EBM) where the total energy of a configuration $s$ is defined as the sum of a pairwise term and a neural network term: $$E(s) = E_{pw}(s) + E_{nn}(s)$$ The term $E_{pw}$ contains the explicit coupling parameters $J_{ij}$ of interest, while $E_{nn}$ is a Multi-Layer Perceptron (MLP) designed to absorb higher-order interactions.

Hybrid model architecture combining explicit pairwise terms with a neural network component.

Training and Reconstruction

The model is trained using pseudolikelihood maximization, a technique that avoids the intractable partition function by predicting the value of a single variable conditioned on all others. This approach is analogous to the masking strategies used in modern transformer models. After training, the authors employ a Fourier expansion technique to extract the final "effective" pairwise interactions from the total energy function. This ensures that any pairwise patterns inadvertently learned by the neural network are correctly attributed to the final reconstructed couplings.

Results

The model was evaluated on synthetic datasets generated from Ising-like models with varying system sizes ($N=16, 64$) and varying strengths ($\gamma$) of higher-order interactions.

Reconstruction error ($\epsilon$) as a function of higher-order interaction strength ($\gamma$) for system sizes 16 and 64.

As shown in the performance plots, the hybrid model (both explicit and reconstructed versions) consistently achieves lower reconstruction error compared to "Pairwise Only" or "MLP Only" models. The advantage of the hybrid approach becomes more pronounced as the strength of higher-order interactions ($\gamma$) increases. Furthermore, the hybrid model demonstrated superior sample efficiency, requiring fewer data points to reach high accuracy than the pure MLP approach.

In real-world tests using the Pfam dataset for the WW domain protein, the hybrid model improved physical contact prediction by reducing false positives compared to standard Potts models. This suggests the model successfully filters out co-evolutionary noise that typically plagues traditional bioinformatics tools.

Discussion

The success of the hybrid model stems from a "division of labor": the neural network acts as a flexible sink for complex correlations, allowing the pairwise term to focus on the fundamental couplings. This architecture bridges the gap between the interpretability of simple physical models and the expressive power of deep learning. While the model faces challenges with specific "parity-like" interactions and computational scaling for very large systems, it represents a significant step toward more robust inverse statistical mechanics. Future work will focus on scaling these methods to larger protein families and exploring more complex interaction topologies.

Summary updated

Abstract

Pairwise models like the Ising model or the generalized Potts model have found many successful applications in fields like physics, biology, and economics. Closely connected is the problem of inverse statistical mechanics, where the goal is to infer the parameters of such models given observed data. An open problem in this field is the question of how to train these models in the case where the data contain additional higher-order interactions that are not present in the pairwise model. In this work, we propose an approach based on Energy-Based Models and pseudolikelihood maximization to address these complications: we show that hybrid models, which combine a pairwise model and a neural network, can lead to significant improvements in the reconstruction of pairwise interactions. We show these improvements to hold consistently when compared to a standard approach using only the pairwise model and to an approach using only a neural network. This is in line with the general idea that simple interpretable models and complex black-box models are not necessarily a dichotomy: interpolating these two classes of models can allow to keep some advantages of both.
BibTeX
@inproceedings{feinauer2021reconstruction,
  author = {Christoph Feinauer and Carlo Lucibello},
  title = {{Reconstruction of Pairwise Interactions using Energy-Based Models}},
  booktitle = {MSML},
  year = {2021},
}
Figure