Oblique and rotation double random forest

Mudasir Ahmad Ganaie, Muhammad Tanveer, Ponnuthurai N. Suganthan, Václav SnáselNeural Networks 2021 · DOI

TL;DR

This paper introduces two novel ensemble methods, "oblique" and "rotation" double random forests, designed to improve the accuracy and generalization capabilities of decision tree ensembles. By incorporating geometric properties of data and enhancing diversity among base learners, these methods demonstrate superior performance over existing random forest variants on a wide range of benchmark datasets.

What Problem Does This Paper Address?

Standard Random Forests (RF) and their variants, while powerful, have limitations. Decision trees in RF typically create axis-parallel splits, which can be suboptimal for capturing the geometric structure of data. Furthermore, standard RF might not grow trees deep enough to achieve optimal performance. The authors aim to address these issues by proposing methods that:

  1. Capture Geometric Properties: Standard RF's axis-parallel splits ignore the inherent geometric relationships within the data.
  2. Enhance Tree Depth: The standard RF might not grow trees sufficiently deep, potentially leading to suboptimal performance.
  3. Increase Diversity: While RF uses bagging and random subspace, further enhancing diversity among base learners can lead to better generalization.

Methods

The paper proposes two main approaches:

  • Rotation Double Random Forest (DRaF): This method enhances diversity by applying transformations (specifically Principal Component Analysis - PCA, and Linear Discriminant Analysis - LDA) to randomly selected feature subspaces at each non-leaf node. This rotation creates more diverse projections of the data, leading to more varied base learners. Unlike standard RF, where bootstrap samples are used at the root, DRaF uses bootstrap samples at each non-terminal node, allowing for deeper trees.
  • Oblique Double Random Forest (MPDRaF): This approach addresses the limitation of axis-parallel splits by using multivariate decision trees. At each non-leaf node, a Multisurface Proximal Support Vector Machine (MPSVM) is employed to find optimal separating hyperplanes, effectively capturing the geometric structure of the data. To combat potential issues with small sample sizes as trees grow deeper, the authors incorporate regularization techniques like Tikhonov regularization, axis-parallel split regularization, and a null space approach.

Both proposed methods build upon the "double random forest" concept, which trains base learners on the original dataset rather than bootstrap samples, leading to larger and potentially more accurate trees.

Datasets

The proposed models were evaluated on:

  • 121 UCI datasets: A standard benchmark suite for evaluating machine learning algorithms.
  • Real-world fisheries datasets: To assess performance on practical, domain-specific data.

Key Insights and Ideas

  • Geometry Matters: Ignoring the geometric structure of data with axis-parallel splits is a significant limitation for decision tree ensembles. Oblique splits, as implemented with MPSVM, can better capture these relationships.
  • Diversity is Key (and Can Be Enhanced): Rotating feature subspaces (PCA/LDA) at each node, in addition to standard ensemble techniques, significantly increases the diversity of base learners, leading to improved generalization.
  • Deeper Trees, Better Performance: The "double random forest" concept, which allows for deeper trees by using bootstrap sampling at non-terminal nodes, consistently shows improved performance.
  • Regularization for Robustness: As decision trees grow deeper, sample size issues can arise. Employing regularization techniques (Tikhonov, null space) is crucial for maintaining performance and robustness.

Key Results

  • Superior Accuracy: The proposed DRaF-LDA, MPDRaF-P, and DRaF-PCA models achieved superior average accuracy compared to baseline models like standard RF and standard double random forest.
  • Better Ranking: Statistical analysis (Friedman test, Nemenyi post-hoc test) indicated that the proposed models (DRaF-LDA, DRaF-PCA, MPDRaF-P) consistently received better average ranks across the 121 datasets, signifying better generalization performance.
  • Increased Tree Size: The proposed double random forest variants generated larger trees (more nodes) compared to standard RF variants, which correlates with improved performance.
  • Computational Complexity: While PCA/LDA transformations add computational overhead, MPSVM-based oblique trees are generally efficient. The proposed methods are competitive with existing advanced ensemble techniques.

Limitations and Future Work

  • Computational Cost: While effective, the proposed methods, particularly those involving PCA/LDA transformations at each node, can increase computational time compared to simpler RF variants.
  • Future Directions: The authors suggest extending this work to regression and time-series forecasting problems. They also propose a benchmarking study comparing their methods, standard RF variants, double RF variants, and XGBoost to guide model selection.

Summary updated

BibTeX
@inproceedings{ganaie2021oblique,
  author = {Mudasir Ahmad Ganaie and Muhammad Tanveer and Ponnuthurai N. Suganthan and Václav Snásel},
  title = {{Oblique and rotation double random forest}},
  booktitle = {Neural Networks},
  year = {2021},
  doi = {10.1016/j.neunet.2022.06.012},
}
Figure