lightgbm: Light Gradient Boosting Machine

Yu Shi, Guolin Ke, Damien Soukhavong, James Lamb, Meng Qi, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu, Nikita Titov, David Cortés‐PoloOpenalex Unknown 2020 · DOI · OpenAlex

Abstract

Tree based algorithms can be improved by introducing boosting frameworks. 'LightGBM' is one such framework, based on Ke, Guolin et al. (2017). This package offers an R interface to work with it. It is designed to be distributed and efficient with the following advantages:

  1. Faster training speed and higher efficiency.
  2. Lower memory usage.
  3. Better accuracy.
  4. Parallel learning supported.
  5. Capable of handling large-scale data. In recognition of these advantages, 'LightGBM' has been widely-used in many winning solutions of machine learning competitions. Comparison experiments on public datasets suggest that 'LightGBM' can outperform existing boosting frameworks on both efficiency and accuracy, with significantly lower memory consumption. In addition, parallel experiments suggest that in certain circumstances, 'LightGBM' can achieve a linear speed-up in training time by using multiple machines.

Introduction

LightGBM is a high-performance gradient boosting framework designed for speed, memory efficiency, and massive scalability. Traditional Gradient Boosting Decision Trees (GBDT) often encounter a "scalability wall" when processing millions of rows and high-dimensional features. Standard GBDTs become prohibitively slow because they must scan every data point for every feature to calculate information gain for potential splits. The LightGBM framework addresses these challenges by optimizing training speed, memory footprint, and inference latency.

Methods and Algorithmic Innovations

The R interface is built upon several novel techniques introduced in the foundational LightGBM research:

  • Histogram-based Learning: Discretizes continuous features into discrete bins (histograms), significantly speeding up split-finding and reducing memory usage.
  • Leaf-wise Tree Growth: Grows trees by expanding the leaf that reduces the most loss, rather than traditional level-wise growth, often leading to higher accuracy.
  • R-Specific Optimizations: Leverages data.table and sparse Matrix objects. It also introduces lgb.configure_fast_predict for low-latency inference.
  • Parallel Learning: Supports distributed learning across multiple machines for linear speed-up.

Datasets and Benchmarking

The package includes several classic datasets for benchmarking and demonstration. A notable example is the Agaricus (Mushroom) Data Set, a UCI repository staple used for binary classification.

The Agaricus dataset is included in the package for training and testing.

Other datasets include the Bank Marketing Data Set, used to predict telemarketing success. These benchmarks demonstrate LightGBM's superiority in efficiency and accuracy over existing frameworks like XGBoost.

Key Insights: Inference and Memory Management

A practical feature of the R package is its explicit handling of prediction latency. By pre-configuring model objects, users can bypass the overhead of repeated parameter parsing.

Parameters for prediction functions, including CSR input handling and iteration control.

Furthermore, memory is treated as a "first-class citizen." Functions like lgb.drop_serialized allow users to manually prune memory used by underlying C++ objects. The package also supports direct categorical feature handling, avoiding the need for manual one-hot encoding, which often leads to better splits and reduced dimensionality.

Results and Limitations

LightGBM has become a cornerstone of winning solutions in machine learning competitions like Kaggle due to its efficiency and scalability. However, users should be aware of serialization complexity; because of the C++ backend, models require specific functions like lgb.save or lgb.load to persist across R sessions. Additionally, the extensive list of hyperparameters requires expertise or automated tuning to find optimal configurations for specific datasets.

Summary updated

Abstract

Tree based algorithms can be improved by introducing boosting frameworks. 'LightGBM' is one such framework, based on Ke, Guolin et al. (2017) &lt;<a href="https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision" target="_top">https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision</a>&gt;. This package offers an R interface to work with it. It is designed to be distributed and efficient with the following advantages: 1. Faster training speed and higher efficiency. 2. Lower memory usage. 3. Better accuracy. 4. Parallel learning supported. 5. Capable of handling large-scale data. In recognition of these advantages, 'LightGBM' has been widely-used in many winning solutions of machine learning competitions. Comparison experiments on public datasets suggest that 'LightGBM' can outperform existing boosting frameworks on both efficiency and accuracy, with significantly lower memory consumption. In addition, parallel experiments suggest that in certain circumstances, 'LightGBM' can achieve a linear speed-up in training time by using multiple machines.
BibTeX
@inproceedings{shi2020lightgbm,
  author = {Yu Shi and Guolin Ke and Damien Soukhavong and James Lamb and Meng Qi and Thomas Finley and Taifeng Wang and Wei Chen and Weidong Ma and Qiwei Ye and Tie-Yan Liu and Nikita Titov and David Cortés‐Polo},
  title = {{lightgbm: Light Gradient Boosting Machine}},
  booktitle = {Openalex Unknown},
  year = {2020},
  doi = {10.32614/cran.package.lightgbm},
}
Figure