Abstract
In regression tasks the distribution of the data is often too complex to be fitted by a single model. In contrast, partition-based models are developed where data is divided and fitted by local models. These models partition the input space and do not leverage the input-output dependency of multimodal-distributed data, and strong local models are needed to make good predictions. Addressing these problems, we propose a binary tree-structured hierarchical routing mixture of experts (HRME) model that has classifiers as non-leaf node experts and simple regression models as leaf node experts. The classifier nodes jointly soft-partition the input-output space based on the natural separateness of multimodal data. This enables simple leaf experts to be effective for prediction. Further, we develop a probabilistic framework for the HRME model, and propose a recursive Expectation-Maximization (EM) based algorithm to learn both the tree structure and the expert models. Experiments on a collection of regression tasks validate the effectiveness of our method compared to a variety of other regression models.
Introduction
Real-world regression data is frequently "multi-modal," meaning that for a given input $x$, there may be multiple potential output behaviors $y$. Standard "divide and conquer" approaches, such as Decision Trees or Random Forests, typically partition the input space based on feature dimensions. However, if a specific region of the input space still contains multiple modes of behavior, these methods require highly complex local models to achieve accuracy. The Hierarchical Routing Mixture of Experts (HRME) addresses this by partitioning data based on the joint distribution of both inputs and outputs, effectively simplifying complex global problems into manageable unimodal sub-problems.
Methodology: Joint Partitioning and Tree Structure
The HRME model is structured as a probabilistic binary tree. It consists of two primary components:
- Non-leaf nodes (Routers): These contain binary classifiers ($h_\beta$) that act as a gating mechanism, routing data points to child nodes.
- Leaf nodes (Experts): These contain simple regression models ($r_\theta$), such as linear regression or Support Vector Regression (SVR).
The core innovation is the use of joint partitioning. By applying a threshold $t$ to the output variable $y$, the model converts the regression task into a temporary classification task during training. This allows the routers to learn how to separate different "modes" of data effectively. The final prediction $\widehat{y}$ is a weighted sum of the leaf experts' outputs, governed by the routing probabilities $q(z|x)$.
The model parameters and tree structure are learned simultaneously through a recursive Expectation-Maximization (EM) algorithm. In the E-step, the model estimates the probability of a data point belonging to a specific leaf. In the M-step, it optimizes the routers to maximize classification accuracy and the leaf experts to minimize regression error. The tree grows dynamically, adding subtrees only when they improve a global objective (the Q-value), thus allowing the model to discover its own optimal depth without manual pruning.
Experimental Evaluation
The HRME model was evaluated against synthetic benchmarks and real-world datasets from the UCI Machine Learning Repository, including HOUSING, CONCRETE, and KIN40K.
A key demonstration of the model's capability is the "3-LINES" synthetic dataset, which features three intersecting lines with varying noise levels. While a single linear model or a standard input-partitioning tree would struggle with the intersections, HRME identifies each line as a distinct mode and assigns an expert to each.
On complex real-world datasets like ENERGY and KIN40K, HRME consistently outperformed standard baselines, including Multi-Layer Perceptrons (MLP) and traditional Hierarchical Mixture of Experts (HME). By using soft probabilistic routing instead of hard cuts, HRME avoids the high-variance "staircase" effect common in Decision Trees, resulting in smoother and more accurate fits across varying data regimes.
Conclusion
HRME demonstrates that the complexity of multi-modal regression data can be managed more effectively by shifting the burden from the regression models to the routing mechanism. By jointly partitioning the input-output space, even simple linear experts can capture complex global trends. Future work will focus on optimizing the threshold search strategy and exploring depth-specific hyperparameter tuning to further enhance performance and scalability.
Summary updated