xgboost: Extreme Gradient Boosting

Tianqi Chen, Tong He, Michaël Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, Mu Li, Junyuan Xie, Min Lin, Yifeng Geng, Yutian Li, Jiaming Yuan, David CortesOpenalex Unknown 2014 · DOI · OpenAlex

Abstract: Extreme Gradient Boosting (XGBoost) is an efficient implementation of the gradient boosting framework originally proposed by Chen & Guestrin (2016). This package serves as the R interface, providing high-performance linear model solvers and tree learning algorithms. By leveraging parallel computation on a single machine, XGBoost can achieve speeds over 10 times faster than existing gradient boosting implementations. It supports a wide range of objective functions, including regression, classification, and ranking, and is designed for extensibility, allowing users to define custom objectives.

1. Introduction

In the landscape of machine learning for tabular data, gradient boosting has emerged as a dominant technique. However, early implementations often struggled with scalability, computational speed, and the handling of sparse data. The XGBoost framework addresses these challenges by adopting a "system" approach to boosting, prioritizing computational efficiency and memory optimization. This R package provides a robust interface to the core C++ library, enabling data scientists to handle millions of rows efficiently while maintaining flexibility across diverse problem domains.

2. Computational Framework

The core of XGBoost is an iterative functional gradient descent algorithm that builds an ensemble of weak learners, typically shallow decision trees. Each subsequent learner is trained to correct the residual errors of the previous ensemble.

Key technical innovations within the R package include:

  • The DMatrix: A specialized internal data structure optimized for memory efficiency and speed. It natively handles sparse matrices, which is essential for high-dimensional datasets.
  • Booster Types: While tree-based models (gbtree) are standard, the package also supports linear models (gblinear) and DART (Dropout additive regression trees).
  • Parallelization and GPU Support: Utilizing OpenMP, the package automatically parallelizes tree construction across CPU cores. It also offers CUDA-enabled GPU support for further acceleration.
Documentation snippet for the 'gblinear' booster type.

3. Data and Implementation

The XGBoost manual utilizes several benchmark datasets to demonstrate its functionality. These include the classic Iris and mtcars datasets for classification and regression, as well as the Titanic dataset for handling categorical features and missing values.

A primary example used for training and cross-validation is the Agaricus dataset, which involves classifying mushrooms as poisonous or edible.

Reference to the Agaricus test dataset within the package documentation.

4. Key Innovations and Interpretability

XGBoost introduces several features that enhance both performance and model understanding:

  • Sparsity-Aware Splitting: The algorithm learns optimal default directions for missing values at each node split, improving robustness on real-world "messy" data.
  • Model Serialization: To ensure cross-platform and cross-version compatibility, the manual emphasizes using native functions like xgb.save() rather than standard R serialization (saveRDS()), as models are pointers to C++ objects.
  • SHAP Integration: The package integrates SHapley Additive exPlanations (SHAP), allowing users to decompose individual predictions into feature contributions, moving beyond global feature importance to local interpretability.

5. Performance and Limitations

XGBoost is recognized for its significant speed advantages, often cited as being up to 10 times faster than competing packages. The introduction of the QuantileDMatrix enables histogram-based tree building, which reduces memory overhead and training time by binning continuous features.

Despite its strengths, the framework presents certain challenges. The vast hyperparameter space requires extensive tuning, often necessitating automated optimization tools. Additionally, recent updates (post-version 2.1.0) introduced structural changes that require careful management of legacy models. Future development focuses on enhancing multi-target model support through the multi_strategy parameter.

Summary updated

Abstract

Extreme Gradient Boosting, which is an efficient implementation of the gradient boosting framework from Chen &amp; Guestrin (2016) &lt;<a href="https://doi.org/10.1145%2F2939672.2939785" target="_top">doi:10.1145/2939672.2939785</a>&gt;. This package is its R interface. The package includes efficient linear model solver and tree learning algorithms. The package can automatically do parallel computation on a single machine which could be more than 10 times faster than existing gradient boosting packages. It supports various objective functions, including regression, classification and ranking. The package is made to be extensible, so that users are also allowed to define their own objectives easily.
BibTeX
@inproceedings{chen2014xgboost,
  author = {Tianqi Chen and Tong He and Michaël Benesty and Vadim Khotilovich and Yuan Tang and Hyunsu Cho and Kailong Chen and Rory Mitchell and Ignacio Cano and Tianyi Zhou and Mu Li and Junyuan Xie and Min Lin and Yifeng Geng and Yutian Li and Jiaming Yuan and David Cortes},
  title = {{xgboost: Extreme Gradient Boosting}},
  booktitle = {Openalex Unknown},
  year = {2014},
  doi = {10.32614/cran.package.xgboost},
}
Figure