Removing Phase Transitions from Gibbs Measures

Ian Fellows, Mark HandcockAISTATS 2017

Abstract

Gibbs measures, including Ising models and Exponential-family Random Graph Models (ERGMs), are foundational tools for modeling high-dimensional discrete data. However, these models frequently suffer from "model degeneracy," a phenomenon where the probability mass concentrates on extreme, unrealistic configurations, making statistical inference and simulation nearly impossible. This paper introduces Tapered Gibbs Measures, a principled extension of the maximum entropy framework that incorporates a quadratic penalty on extreme statistics. We prove that this approach can eliminate phase transitions, ensuring a stable "uniphase" distribution that allows for the successful estimation of complex models previously deemed unfittable.

The Challenge of Model Degeneracy

Gibbs measures are widely used in fields ranging from computer vision to social network analysis. They are typically derived using the principle of maximum entropy: finding the distribution that is the least informative while matching certain observed mean statistics.

Despite their utility, these models have a notorious "dark side" known as phase transitions or model degeneracy. In a social network model, for instance, a infinitesimal change in a parameter can cause the model to jump abruptly from predicting a sparse, realistic network to a completely "saturated" one where every possible connection exists. For researchers, this is a significant hurdle: Markov Chain Monte Carlo (MCMC) sampling often fails as the algorithm becomes trapped in these extreme modes, and the resulting model may place almost all its probability mass on configurations that bear no resemblance to the actual data.

Methodology: Tapered Gibbs Measures

The authors propose the Tapered Gibbs Measure to address this instability. While standard Gibbs measures maximize entropy subject to a constraint on the mean of sufficient statistics, the tapered version adds a second constraint: the variance of these statistics must also be bounded.

Using Karush-Kuhn-Tucker (KKT) multipliers, the resulting probability mass function includes an additional quadratic "tapering" term:

$$q(x|\theta, \beta) \propto \exp\left(\sum \theta_k g_k(x) - \sum \beta_k^{-2} (\mu_k - g_k(x))^2\right)$$

In this formulation, $\beta$ acts as a hyper-parameter controlling the strength of the force pulling the distribution toward the observed mean.

The Uniphase Guarantee

A key theoretical contribution of this work is the "Uniphase Guarantee" (Theorem 2). The authors provide a mathematical proof that for a sufficiently small $\beta$, the distribution is guaranteed to be unimodal, effectively removing local minima or saddle points in the configuration density. This ensures that the model remains stable and computationally tractable.

Figure 2: Histograms of statistics for the Ising model under varying levels of tapering. As the hyper-parameter β decreases (moving from bottom to top), the bimodal distribution characteristic of model degeneracy collapses into a stable, unimodal distribution centered at the observed data.

Key Results and Applications

Stabilizing the Ising Model

To demonstrate the effect, the authors applied the tapered measure to a classic Ising model representing political affiliation (simulated on an $n \times n$ toroidal lattice). In the standard model, the distribution was bimodal, predicting a community that was either "all liberal" or "all conservative," despite the underlying data being a 50/50 split. As shown in Figure 2, the Tapered Gibbs Measure successfully collapsed these extreme modes into a single, realistic Gaussian-like distribution centered at the observed data.

Fitting the "Unfittable": Network Science Collaboration

The approach was also tested on a real-world dataset of co-authorship among 1,589 scientists. This network is notoriously difficult for traditional ERGMs due to its size and structural complexity. Previous attempts to fit a 12-parameter ERGM to this data using standard Gibbs measures failed as the MCMC process diverged.

Using the tapered model, the authors successfully achieved convergence. The resulting model provided meaningful insights into network transitivity and degree distributions, matching the observed data with high precision.

Figure 3: Goodness-of-fit plots for the Network Science Collaboration network. The red dots represent observed network statistics (Degree and Edge-wise Shared Partners), while the shaded regions represent the range of values produced by simulations from the fitted Tapered Gibbs model.

Conclusion and Practical Heuristics

Tapered Gibbs Measures represent a significant advancement in the stability of high-dimensional discrete models. While choosing the optimal $\beta$ remains somewhat domain-dependent, the authors suggest a practical "2-Sigma" rule: setting $\beta$ to twice the square root of the expected variance ($r=2$) consistently yields stable results across various applications. By constraining variance, researchers can now fit complex, informative models that were previously impossible to estimate due to the "beast" of phase transitions.

Summary updated

BibTeX
@inproceedings{fellows2017removing,
  author = {Ian Fellows and Mark Handcock},
  title = {{Removing Phase Transitions from Gibbs Measures}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2017},
}
Figure