Bias-Variance Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions
Lists
Only you see this
Abstract
Hierarchical probabilistic models are able to use a large number of parameters to create a model with a high representation power. However, it is well known that increasing the number of parameters also increases the complexity of the model which leads to a bias-variance trade-off. Although it is a classical problem, the bias-variance trade-off between hiddenlayers and higher-order interactions have not been well studied. In our study, we propose an efficient inference algorithm for the log-linear formulation of the higher-order Boltzmann machine using a combination of Gibbs sampling and annealed importance sampling. We then perform a bias-variance decomposition to study the differences in hidden layers and higher-order interactions. Our results have shown that using hidden layers and higher-order interactions have a comparable error with a similar order of magnitude and using higherorder interactions produce less variance for smaller sample size.
BibTeX
@inproceedings{luo2019bias,
author = {Simon Luo and Mahito Sugiyama},
title = {{Bias-Variance Trade-Off in Hierarchical Probabilistic Models Using Higher-Order Feature Interactions}},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2019},
doi = {10.1609/aaai.v33i01.33014488},
}| Hierarchical models as marginals of hierarchical models | International Journal of Approximate Reasoning · 2016 |
| Stochastic Gradient Estimate Variance in Contrastive Divergence and Persistent Contrastive Divergence | Workshops @ ICLR · 2014 |
| Transductive Boltzmann Machines | arXiv · 2018 |
| Learning Restricted Boltzmann Machines with Few Latent Variables | arXiv · 2020 |
| Hierarchical Variational Models | arXiv · 2015 |
| Mixture Representations for Inference and Learning in Boltzmann Machines | arXiv · 2013 |