Bias-Variance Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions

Simon Luo, Mahito Sugiyama

Introduction

A fundamental challenge in probabilistic modeling is increasing a model's representational power without succumbing to overfitting. This paper investigates two distinct paradigms for enhancing complexity within the Boltzmann Machine family: the addition of latent variables, as seen in Restricted Boltzmann Machines (RBMs), and the explicit modeling of higher-order interactions between visible features, known as Higher-Order Boltzmann Machines (HBMs).

Structural comparison between (a) a Restricted Boltzmann Machine with hidden nodes and (b) a Higher-Order Boltzmann Machine represented as a poset of feature interactions.

The RBM approach introduces hidden nodes ($h_1, h_2, h_3$) to capture latent dependencies among visible units ($v_1, \dots, v_4$). In contrast, the HBM approach utilizes a hierarchical structure—often visualized as a Hasse diagram—to explicitly define interactions between subsets of features (e.g., pairs, triplets, or quadruplets). While both methods increase model capacity, they navigate the bias-variance trade-off differently, particularly in data-sparse regimes.

Information Geometric Framework

To rigorously compare these architectures, the authors employ an information geometric formulation. By treating probability distributions as points on a manifold, the total error between a model's estimate and the ground truth can be decomposed using a generalized Pythagorean theorem. This allows for a precise quantification of:

  1. Bias: The error resulting from the model's inability to represent the true distribution even with infinite data.
  2. Variance: The error resulting from sensitivity to specific fluctuations in finite training samples.

The study utilizes Gibbs sampling for inference and Annealed Importance Sampling (AIS) to estimate the partition function, which is computationally intensive for both high-order and hidden-layer models.

Empirical Analysis of the Bias-Variance Trade-Off

Using synthetic datasets where the ground truth is known, the researchers evaluated how total error, bias, and variance scale with sample size and model complexity.

Decomposition of total error into bias and variance for RBMs with varying numbers of hidden nodes across different sample sizes.

As shown in the RBM results, variance is the dominant source of error at small sample sizes, especially as the number of hidden nodes (and thus parameters) increases. However, as the sample size grows, variance decays significantly, and the total error eventually converges to a "bias floor" determined by the model's structural limitations.

A key finding of the paper is that HBMs exhibit greater stability than RBMs when training data is limited. While RBMs rely on hidden layers to "discover" latent patterns—a process prone to high variance—HBMs benefit from explicitly defined interaction terms. For small datasets, this structural prior leads to lower total error by significantly reducing variance compared to the RBM's stochastic discovery of features.

Conclusion and Future Directions

The study concludes that while RBMs and HBMs achieve comparable total error on large datasets, HBMs are superior in "small data" scenarios due to their lower variance. However, HBMs face scaling challenges; calculating interactions for high-dimensional data remains computationally expensive. Future work aims to extend this analysis to Deep Boltzmann Machines (DBMs) and validate these findings on complex real-world datasets such as genomic or image data, where higher-order dependencies are prevalent but difficult to model efficiently.

Summary updated

BibTeX
@inproceedings{luo2019bias,
  author = {Simon Luo and Mahito Sugiyama},
  title = {{Bias-Variance Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33014488},
}
Figure