TF Boosted Trees: A Scalable TensorFlow Based Framework for Gradient Boosting

Natalia Ponomareva, Soroush Radpour, Gilbert Hendry, Salem Haykal, Thomas Colthurst, Petr Mitrichev, Alexander Grushetsky

TL;DR

TF Boosted Trees (TFBT) is an open-source framework that brings the power of gradient boosting to the TensorFlow ecosystem. By introducing a novel "layer-by-layer" boosting procedure and a parameter-server architecture, it enables efficient, distributed training on massive datasets that exceed worker memory, all while leveraging TensorFlow’s automatic differentiation for easy custom loss functions.

What Problem Does This Paper Address?

Gradient Boosted Trees (GBTs) are the "workhorses" of tabular data, dominating competitions like Kaggle. However, scaling them to industrial-sized datasets presents several hurdles:

  1. Memory Constraints: Many existing implementations require the entire dataset to fit into memory, which is impossible for web-scale data.
  2. Complexity of Customization: Implementing a custom loss function usually requires the user to manually derive and code first and second-order derivatives (gradients and Hessians).
  3. Architectural Rigidity: Traditional GBTs build one full tree at a time. This can lead to "weak" deeper nodes because they are trained on increasingly smaller subsets of data without refreshing global statistics.
  4. Infrastructure Costs: Training large models on stable, high-priority instances is expensive. There is a need for systems that can run reliably on cheaper, "preemptible" (stateless) virtual machines.

Methods

TFBT introduces several technical innovations to solve these problems:

  • Layer-by-Layer Boosting: Unlike standard boosting which builds a complete tree before moving to the next, TFBT can build trees one layer at a time across the entire ensemble. It recalculates gradients and Hessians for every new layer. This allows deeper nodes to "see" the errors of the previous layer more clearly, acting as a prior that leads to stronger trees and faster convergence.
  • Parameter Server (PS) Architecture: To handle high-dimensional data, TFBT uses a PS approach. Workers process small mini-batches of data to build local "quantile sketches" (summaries of feature distributions). These are pushed to the PS, which aggregates them to find the best splits. This "online" approach means the full dataset never needs to reside in memory.
  • StampedResource: To manage the chaos of asynchronous distributed training, the authors created a "stamping" system. Every resource (tree ensemble, gradients, etc.) has an int64 version token. This ensures that workers are always performing updates against a consistent version of the model, preventing stale data from corrupting the training process.
  • TensorFlow Integration: Because it's built on TF, it inherits Automatic Differentiation. Users only need to define a loss function; the system handles the math for the gradients and Hessians automatically.

Datasets

The paper discusses the system's design and architecture rather than providing a benchmark suite. While it references the success of GBTs on datasets like Kaggle and KDDCup, specific experimental datasets used for the internal validation of TFBT are not discussed in the paper.

Key Insights and Ideas

  • Trees as "Neural" Layers: By building layer-by-layer and recalculating statistics, TFBT treats tree growth more like the training of a neural network. This helps mitigate the "data fragmentation" problem where deep nodes in traditional trees become unreliable due to small sample sizes.
  • Statelessness is a Feature: By design, the workers in TFBT hold no state. If a worker is preempted or fails, it can simply be replaced. It just grabs the latest "stamp" from the Parameter Server and continues with a new mini-batch.
  • Multi-class Efficiency: Most GBT libraries handle multi-class problems by building $N$ trees for $N$ classes (one-vs-rest). TFBT supports storing per-class scores at each leaf, significantly reducing the total number of trees required.

Key Results

  • Convergence & Accuracy: The authors state that the layer-by-layer approach generally leads to faster convergence and less overfitting for deeper trees compared to standard stochastic gradient boosting.
  • Scalability: The system is designed to scale to high-dimensional data where "All-Reduce" architectures (like those used in some versions of XGBoost) typically struggle.
  • Deployment: Because it is part of the TensorFlow ecosystem, TFBT models can be exported and served on multiple platforms, including mobile, via TF Serving.

Limitations and Future Work

While the paper highlights the strengths of the Parameter Server approach for high-dimensional data, it notes that Map-Reduce and All-Reduce versions of GBTs can be slow to scale in those specific environments. However, the paper does not provide a detailed quantitative comparison of training time or communication overhead between TFBT and competitors like LightGBM or XGBoost. Future work is implied in the "contrib" status of the library at the time of writing, suggesting ongoing refinement of the per-layer boosting algorithms.

Summary updated

Abstract

TF Boosted Trees (TFBT) is a new open-sourced framework for the distributed training of gradient boosted trees. It is based on TensorFlow, and its distinguishing features include a novel architecture, automatic loss differentiation, layer-by-layer boosting that results in smaller ensembles and faster prediction, principled multi-class handling, and a number of regularization techniques to prevent overfitting.
BibTeX
@inproceedings{ponomareva2017tf,
  author = {Natalia Ponomareva and Soroush Radpour and Gilbert Hendry and Salem Haykal and Thomas Colthurst and Petr Mitrichev and Alexander Grushetsky},
  title = {{TF Boosted Trees: A Scalable TensorFlow Based Framework for Gradient Boosting}},
  booktitle = {ECML/PKDD},
  year = {2017},
  url = {https://openreview.net/forum?id=KWDszu7SQA},
}
Figure