Asymptotic Normality of the Maximum Pseudolikelihood Estimator for Fully Visible Boltzmann Machines

Hien Duy Nguyen, Ian A. WoodIEEE TNNLS 2016 · DOI

TL;DR

This paper proves that the Maximum Pseudolikelihood Estimator (MPLE) for Fully Visible Boltzmann Machines (FVBMs) is not only consistent but also asymptotically normal. This theoretical breakthrough allows for the construction of reliable confidence intervals and hypothesis tests for FVBM parameters, which is crucial for their practical application.

What Problem Does This Paper Address?

Boltzmann Machines (BMs) are a class of neural networks used for modeling multivariate binary data. Estimating the parameters of these models, especially in the "fully visible" case (FVBM) where all units are observable, is challenging. Traditional Maximum Likelihood Estimation (MLE) becomes computationally intractable due to the complex normalizing constant in the model's probability distribution. While Maximum Pseudolikelihood Estimation (MPLE) has been proposed as a computationally feasible alternative and shown to be consistent, its statistical properties, particularly its asymptotic behavior, remained unproven. This paper addresses this gap by rigorously demonstrating the asymptotic normality of the MPLE for FVBMs, thereby providing a solid theoretical foundation for statistical inference.

Methods

The core of this paper is a theoretical proof of the asymptotic normality of the MPLE for FVBMs. The authors achieve this by leveraging established theorems from statistical inference, specifically Theorem 5.1 from [14] and Theorem 2.2 from [2]. They meticulously verify the necessary regularity conditions for these theorems to hold for the FVBM model. This involves:

  • Identifiability and Support: Demonstrating that the model parameters can be uniquely identified and that the probability distributions have a common support.
  • Smoothness and Derivatives: Proving that the conditional probabilities are smooth enough to admit third-order derivatives with respect to the parameters.
  • Score Function Properties: Verifying that the expected value of the first derivative of the log-pseudolikelihood (the score function) is zero, and that the expected value of the product of two score functions equals the negative of the expected value of the second derivative (the Fisher information).
  • Fisher Information Matrix: Showing that the Fisher information matrix is non-singular and positive-definite.
  • Boundedness of Higher-Order Derivatives: Establishing that the expected values of the third-order derivatives of the log-pseudolikelihood are bounded.

Once asymptotic normality is established, the paper outlines how to construct Wald-type confidence intervals and hypothesis tests using the estimated parameters and the derived Fisher information matrix.

Datasets

Not discussed in the paper. The study uses simulated data generated from a 3-variate FVBM with known parameters.

Key Insights and Ideas

  • Bridging the Gap: The paper successfully bridges a critical gap in the statistical understanding of MPLE for FVBMs by proving asymptotic normality. This moves beyond just consistency to enable practical statistical inference.
  • Theoretical Rigor: The rigorous verification of multiple, complex regularity conditions for established statistical theorems is a significant theoretical contribution.
  • Enabling Inference: The direct consequence of asymptotic normality is the ability to construct confidence intervals and perform hypothesis tests. This makes FVBMs more amenable to real-world statistical analysis.
  • Practical Implications: The paper demonstrates that the theoretical results hold even for relatively small sample sizes in their simulation study, suggesting practical utility.

Key Results

The primary result is the proof of asymptotic normality for the Maximum Pseudolikelihood Estimator (MPLE) of Fully Visible Boltzmann Machines (FVBMs). This means that as the sample size increases, the distribution of the MPLE approaches a normal distribution. Specifically, the paper shows that for a sample $X_1, \dots, X_n$ from a density (1) with true parameter $\theta^0$, the estimator $\hat{\theta}_n$ satisfies:

$$ \sqrt{n}(\hat{\theta}_n - \theta^0) \xrightarrow{d} \mathcal{N}(0, I_1(\theta^0)^{-1}) $$

where $I_1(\theta^0)$ is the Fisher information matrix.

The paper also demonstrates how this asymptotic normality can be used to:

  • Construct Asymptotic Confidence Intervals (CIs): For the $k$-th element of $\theta$, a $(1-\alpha) \times 100\%$ confidence interval is given by $\hat{\theta}_k \pm z_{1-\alpha/2} \sqrt{I_{1,kk}^{-1}(\hat{\theta}) / n}$, where $I_{1,kk}^{-1}(\hat{\theta})$ is the $k$-th diagonal element of the inverse Fisher information matrix evaluated at the estimate $\hat{\theta}$.
  • Perform Hypothesis Tests: A Wald-type test statistic is derived for testing hypotheses about subsets of parameters.

The simulation study supports these theoretical findings, showing that the constructed confidence intervals and hypothesis tests exhibit expected coverage percentages and false positive rates, respectively, even for moderate sample sizes.

Limitations and Future Work

  • Assumption A1: The paper relies on Assumption A1, which states that the Hessian of the log-pseudolikelihood is non-singular and negative-definite. While this assumption is necessary for the theoretical results and is assumed to hold if the model is identifiable, it is not proven theoretically in the paper. The authors state that if one assumes A1, then B5 (related to the Fisher information matrix) follows.
  • Hidden Units: The current work is limited to the "fully visible" case of Boltzmann Machines. The authors explicitly state that extending these results to the standard case with hidden units would be an interesting direction for future work.
  • Simulation Study Limitations: The authors note that the deviations from expected confidence and significance levels in the simulation study might be due to the number of repetitions not being sufficiently large to distinguish between sample sizes, or a slow probabilistic convergence. Larger sample sizes might be needed to observe clearer differences.

Summary updated

BibTeX
@article{nguyen2016asymptotic,
  author = {Hien Duy Nguyen and Ian A. Wood},
  title = {{Asymptotic Normality of the Maximum Pseudolikelihood Estimator for Fully Visible Boltzmann Machines}},
  journal = {IEEE TNNLS},
  year = {2016},
  doi = {10.1109/tnnls.2015.2425898},
}

Loading related content...

Figure