Fully-Automatic Bayesian Piecewise Sparse Linear Models

Riki Eto, Ryohei Fujimaki, Satoshi Morinaga, Hiroshi TamanoAISTATS 2014

Abstract

This paper introduces a fully-automatic approach to constructing Piecewise Sparse Linear Models (PSLMs) by combining Hierarchical Mixtures of Experts (HMEs) with Factorized Asymptotic Bayesian (FAB) inference. The framework simultaneously determines optimal tree-based partitions and selects relevant features for local experts, resulting in models that are both highly interpretable and competitively accurate.

Introduction

In machine learning, a persistent challenge is the trade-off between the interpretability of simple models and the predictive power of complex "black-box" algorithms. Piecewise Linear Models (PLMs) offer a middle ground by dividing the feature space into regions, each handled by a local linear expert. However, selecting the optimal partition structure and identifying relevant features for each region traditionally requires manual tuning or exhaustive search. This paper addresses these challenges through a unified Bayesian framework that automates model selection without sensitive hyperparameters.

Methods

The authors propose FAB/HMEs, a framework that integrates several advanced techniques to automate model construction:

FAB Inference and Sparsity

By extending Factorized Asymptotic Bayesian (FAB) inference to HMEs, the model naturally induces $L_0$-regularization. This allows the algorithm to automatically "shrink" the model by pruning irrelevant experts (structural sparsity) and removing irrelevant features within experts (feature sparsity) during the learning process.

Bernoulli Gating

To improve computational efficiency, the standard sigmoid gating functions are replaced with Bernoulli gating functions based on step functions. This simplification accelerates the optimization process significantly.

The structure of a Hierarchical Mixture of Experts (HME) showing gating functions $g(x, \alpha)$ and local experts $p(y|x, \phi)$.

Projection E-step

A key contribution is the identification of a "training-prediction gap," where maximizing marginal likelihood does not always translate to optimal predictive performance. The authors introduce a Projection E-step to align the model's internal logic with its predictive behavior.

The Projection E-step algorithm and its geometric interpretation as a correction toward the predictive distribution.

Results and Evaluation

The framework was evaluated using synthetic data, nine benchmark regression datasets (including Abalone, Housing, and Puma32H), and a real-world energy demand forecasting task.

  • Model Compactness: FAB/HMEs+ produced significantly more compact models than traditional methods. On average, it utilized only 9.7 experts, compared to 97.0 for standard Regression Trees.
  • Predictive Accuracy: The model performed competitively with Bayesian Treed Linear Models (BTLMs) and outperformed standard HMEs. Notably, on the Puma32H dataset, it exceeded the performance of RBF-kernel Support Vector Regression (SVR).
  • Interpretability: In the energy forecasting task, the model automatically identified meaningful regimes, such as distinguishing between weekday and weekend energy consumption patterns.

Conclusion

The FAB/HMEs framework successfully automates the construction of piecewise sparse linear models. By leveraging FAB inference and the Projection E-step, the method achieves a balance of sparsity, accuracy, and interpretability. While currently optimized for regression, future work aims to extend this $L_0$ regularization framework to classification tasks involving non-quadratic loss functions.

Summary updated

BibTeX
@inproceedings{eto2014fully,
  author = {Riki Eto and Ryohei Fujimaki and Satoshi Morinaga and Hiroshi Tamano},
  title = {{Fully-Automatic Bayesian Piecewise Sparse Linear Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2014},
}
Figure