Abstract
This paper introduces a fully-automatic approach to constructing Piecewise Sparse Linear Models (PSLMs) by combining Hierarchical Mixtures of Experts (HMEs) with Factorized Asymptotic Bayesian (FAB) inference. The framework simultaneously determines optimal tree-based partitions and selects relevant features for local experts, resulting in models that are both highly interpretable and competitively accurate.
Introduction
In machine learning, a persistent challenge is the trade-off between the interpretability of simple models and the predictive power of complex "black-box" algorithms. Piecewise Linear Models (PLMs) offer a middle ground by dividing the feature space into regions, each handled by a local linear expert. However, selecting the optimal partition structure and identifying relevant features for each region traditionally requires manual tuning or exhaustive search. This paper addresses these challenges through a unified Bayesian framework that automates model selection without sensitive hyperparameters.
Methods
The authors propose FAB/HMEs, a framework that integrates several advanced techniques to automate model construction:
FAB Inference and Sparsity
By extending Factorized Asymptotic Bayesian (FAB) inference to HMEs, the model naturally induces $L_0$-regularization. This allows the algorithm to automatically "shrink" the model by pruning irrelevant experts (structural sparsity) and removing irrelevant features within experts (feature sparsity) during the learning process.
Bernoulli Gating
To improve computational efficiency, the standard sigmoid gating functions are replaced with Bernoulli gating functions based on step functions. This simplification accelerates the optimization process significantly.
Projection E-step
A key contribution is the identification of a "training-prediction gap," where maximizing marginal likelihood does not always translate to optimal predictive performance. The authors introduce a Projection E-step to align the model's internal logic with its predictive behavior.
Results and Evaluation
The framework was evaluated using synthetic data, nine benchmark regression datasets (including Abalone, Housing, and Puma32H), and a real-world energy demand forecasting task.
- Model Compactness: FAB/HMEs+ produced significantly more compact models than traditional methods. On average, it utilized only 9.7 experts, compared to 97.0 for standard Regression Trees.
- Predictive Accuracy: The model performed competitively with Bayesian Treed Linear Models (BTLMs) and outperformed standard HMEs. Notably, on the Puma32H dataset, it exceeded the performance of RBF-kernel Support Vector Regression (SVR).
- Interpretability: In the energy forecasting task, the model automatically identified meaningful regimes, such as distinguishing between weekday and weekend energy consumption patterns.
Conclusion
The FAB/HMEs framework successfully automates the construction of piecewise sparse linear models. By leveraging FAB inference and the Projection E-step, the method achieves a balance of sparsity, accuracy, and interpretability. While currently optimized for regression, future work aims to extend this $L_0$ regularization framework to classification tasks involving non-quadratic loss functions.
Summary updated