AugBoost: Gradient Boosting Enhanced with Step-Wise Feature Augmentation

Philip Tannor, Lior RokachIJCAI 2019 · DOI

TL;DR

This paper introduces AugBoost, a novel method that enhances Gradient Boosted Decision Trees (GBDT) by augmenting features between boosting iterations. By incorporating new, learned features (either from neural networks or unsupervised methods like PCA/RP), AugBoost significantly outperforms standard GBDT and existing feature augmentation techniques on a variety of classification tasks.

What Problem Does This Paper Address?

Gradient Boosted Decision Trees (GBDT) are a powerful and widely used machine learning algorithm known for achieving state-of-the-art results. However, a limitation of GBDT is that the set of features used to train the decision trees remains constant throughout all boosting iterations. While other ensemble methods like Random Forests have explored feature augmentation, GBDT's unique characteristic of updating the target variable at each iteration suggests an opportunity for step-wise feature augmentation. This paper aims to bridge this gap by introducing and evaluating methods to dynamically augment features within the GBDT framework, seeking to improve predictive performance.

Methods

The core innovation is AugBoost, which integrates feature augmentation into the GBDT training loop. Instead of using the same original features for every decision tree, AugBoost generates new features at specific intervals and concatenates them with the original features before training each new tree.

The paper proposes three main augmentation strategies:

  • AugBoost-ANN: This method uses an Artificial Neural Network (ANN) to learn an embedding of the original features. Features are extracted from the ANN's last hidden layer and then combined with the original features. This is a supervised approach, as the ANN is trained on the (updated) target variable.
  • AugBoost-PCA: This approach uses Principal Component Analysis (PCA) to rotate and transform subsets of the original features. PCA is an unsupervised method, meaning it doesn't use the target variable. The transformed features are then concatenated with the original ones.
  • AugBoost-RP: Similar to AugBoost-PCA, this method uses Random Projections (RP) to transform feature subsets. RP is also an unsupervised technique. The new features are concatenated with the original features.

A key implementation detail is the NBA parameter, which controls how often feature augmentation occurs. Instead of augmenting features in every single iteration, augmentation is performed every NBA iterations. This significantly speeds up training, especially for computationally intensive methods like AugBoost-ANN, while still allowing the boosting process to leverage the new information.

Datasets

The paper evaluated the proposed methods on 20 classification datasets sourced from the UCI repository, Kaggle, and the Keel dataset repository. These datasets vary in size, number of features, and number of classes, providing a broad testbed for the algorithms.

Key Insights and Ideas

  • Augmenting features between boosting iterations is beneficial: Unlike methods that augment features only once before training an ensemble, step-wise augmentation within GBDT yields significant improvements.
  • Supervised augmentation (ANN) can be powerful: AugBoost-ANN, which leverages the target variable to guide feature learning, shows strong performance, suggesting that supervised feature extraction can be highly effective in the context of boosting.
  • Unsupervised augmentation also works well: AugBoost-PCA and AugBoost-RP demonstrate that even without direct supervision, rotating or projecting features can enhance GBDT's performance.
  • Concatenation is key: Combining the newly augmented features with the original features appears to be a robust strategy, allowing models to benefit from both handcrafted and learned representations.
  • Regularization effect: The paper suggests that AugBoost methods might have a reduced tendency to overfit compared to standard GBDT, as indicated by learning curves.

Key Results

The AugBoost methods consistently outperformed the classic GBDT (specifically, the LightGBM implementation) and RPBoost across the 20 classification tasks.

  • In 11 out of 20 cases, all three AugBoost variations outperformed LightGBM.
  • AugBoost-ANN and AugBoost-PCA significantly outperformed LightGBM (p < 0.05 and p < 0.1, respectively).
  • AugBoost-ANN and AugBoost-PCA significantly outperformed RPBoost (p < 0.05).
  • AugBoost-RP also showed an advantage over RPBoost.
  • Statistical tests (Friedman and Nemenyi post-hoc) confirmed the significance of these improvements.

Limitations and Future Work

  • ANN Architecture: The ANN architecture used in AugBoost-ANN was a generic, "one-size-fits-all" design. The authors suggest that more sophisticated ANN architectures, potentially chosen via meta-learning, could further improve performance, especially given the high computational cost of AugBoost-ANN.
  • Feature Selection: The paper concatenated new features with original ones. Future work could explore applying feature selection to this combined set before training each DT.
  • Alternative Augmentation: The authors propose exploring other feature augmentation techniques, such as autoencoders, t-SNE, and LDA, and even combining different augmentation methods across iterations.
  • Regression Tasks: While the paper mentions applicability to regression, specific implementation changes might be needed for optimal results.
  • GBDT Implementation: The experiments used scikit-learn's GBDT as a base. Embedding AugBoost into more advanced GBDT libraries like XGBoost or LightGBM could potentially yield even better results.

Summary updated

BibTeX
@inproceedings{tannor2019augboost,
  author = {Philip Tannor and Lior Rokach},
  title = {{AugBoost: Gradient Boosting Enhanced with Step-Wise Feature Augmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/493},
}
Figure