Abstract
Inspired by the feedforward multilayer perceptron (FF-MLP), decision tree (DT) and extreme learning machine (ELM), a new classification model, called the subspace learning machine (SLM), is proposed in this work. SLM first identifies a discriminant subspace, $S^0$, by examining the discriminant power of each input feature. Then, it uses probabilistic projections of features in $S^0$ to yield 1D subspaces and finds the optimal partition for each of them. This is equivalent to partitioning $S^0$ with hyperplanes. A criterion is developed to choose the best $q$ partitions that yield $2^q$ partitioned subspaces among them. We assign $S^0$ to the root node of a decision tree and the intersections of $2^q$ subspaces to its child nodes of depth one. The partitioning process is recursively applied at each child node to build an SLM tree. When the samples at a child node are sufficiently pure, the partitioning process stops and each leaf node makes a prediction. The idea can be generalized to regression, leading to the subspace learning regressor (SLR). Furthermore, ensembles of SLM/SLR trees can yield a stronger predictor. Extensive experiments are conducted for performance benchmarking among SLM/SLR trees, ensembles and classical classifiers/regressors.
Introduction
Classical machine learning models often face a fundamental trade-off between interpretability and predictive power. Decision Trees (DTs) are highly interpretable but frequently result in "weak" classifiers that overfit due to single-feature splitting. Conversely, Support Vector Machines (SVMs) and Multilayer Perceptrons (MLPs) offer high performance but often act as "black boxes," struggling with high-dimensional complexity and computational overhead. Extreme Learning Machines (ELMs) utilize random projections to address some of these issues, but their lack of guidance in feature selection can lead to practical inefficiencies.
The Subspace Learning Machine (SLM) is proposed as a hybrid architecture that bridges these gaps. By combining the hierarchical structure of decision trees with the feature-projection capabilities of MLPs, SLM partitions high-dimensional feature spaces using multiple discriminant directions simultaneously. This approach results in wider, shallower trees that are both computationally efficient and more resistant to overfitting than traditional deep trees.
Methodology
The core of SLM lies in its ability to hierarchically partition the input feature space $X$ into multiple discriminant subspaces through a multi-step process:
- Subspace Identification: The model identifies an initial discriminant subspace $S^0$ by filtering out features with low predictive power.
- Probabilistic Projection: Instead of splitting on a single axis or using purely random projections, SLM employs a probabilistic mechanism. Projection vectors are weighted toward features with higher "discriminant power," determined by a Discriminant Feature Test (DFT) based on entropy.
- N-ary Splitting: SLM nodes perform $n$-ary splits rather than binary ones. By selecting the $q$ best partitions, the model creates $2^q$ child nodes at each level, leading to a "wide and shallow" tree structure.
- Generalization and Ensembles: The framework extends to regression via the Subspace Learning Regressor (SLR), which utilizes Mean Squared Error (MSE) instead of entropy. To further boost performance, the authors introduced SLM Forest (bagging) and SLM Boost (additive boosting).
Experimental Evaluation
The performance of SLM and SLR was benchmarked against 15 datasets. For classification, the study used three synthetic 2D datasets (Circle-and-Ring, 2-New-Moons, 4-New-Moons) and six real-world benchmarks from the UCI repository, including Iris, Wine, and Banknote Authentication. Regression performance was evaluated using three synthetic "Make Friedman" datasets and three real-world sets such as Boston and California Housing.
Key Results and Discussion
Experimental results highlight several advantages of the SLM framework:
- Superior Accuracy and Convergence: SLM Forest and SLM Boost consistently outperformed industry standards like XGBoost and Random Forest across most datasets. Notably, SLM variants converged to optimal performance significantly faster, requiring fewer trees to reach peak accuracy.
- Model Efficiency: The baseline SLM model is remarkably compact. For instance, on the Banknote dataset, SLM achieved higher accuracy than an SVM while using only 40 parameters compared to the SVM's 1,322.
- Robustness through Shallowness: By allowing multiple decisions per node, SLM avoids the "death by a thousand cuts" typical of deep decision trees, making the model naturally more robust against overfitting.
Conclusion and Future Work
The Subspace Learning Machine offers a promising alternative to traditional classifiers and regressors by providing a "middle ground" of guided randomness and hierarchical partitioning. While the current model demonstrates high efficiency and accuracy, future work will focus on automating hyper-parameter tuning and exploring integration with unsupervised representation learning to handle ultra-high-dimensional data like raw images. Additionally, scaling the methodology to "Big Data" environments remains a key objective for future research.
Summary updated