Quantile Extreme Gradient Boosting for Uncertainty Quantification

Xiaozhe Yin, Masoud Fallah Shorshani, Rob McConnell, Scott Fruin, Yao-Yi Chiang, Meredith FranklinarXiv 2023 · DOI · arXiv

This paper introduces QXGBoost, a novel adaptation of the popular Extreme Gradient Boosting (XGBoost) algorithm designed specifically for uncertainty quantification. By integrating a smoothed quantile regression objective, the authors enable XGBoost to produce robust prediction intervals rather than just single-point estimates.

Bridging the Gap in XGBoost

In high-stakes fields like environmental science, a single-number prediction (a point estimate) is often insufficient. Decision-makers need to know the "uncertainty" or the range in which the true value likely falls. While other algorithms like LightGBM and Gradient Boosting Machines (GBM) have established "quantile" versions to provide these ranges, XGBoost has historically lagged behind.

The technical hurdle is that XGBoost’s optimization engine requires a loss function that is twice-differentiable (it needs both a first and a second derivative). Standard quantile loss (often called "pinball loss") is shaped like a sharp "V"—it isn't differentiable at the tip, and its second derivative is zero everywhere else, which causes XGBoost's mathematical framework to fail.

Methods: The Huber-Smoothed Quantile Loss

The core innovation is the implementation of a Huber-smoothed quantile loss function.

  • The Huber Norm: The authors replace the sharp "V" of the standard quantile loss with a smooth parabolic curve near the origin. This "Huberization" ensures the function is continuously differentiable.
  • Second-Order Approximation: By providing a non-zero second derivative (Hessian) in the transition region, the authors satisfy XGBoost’s requirement for a second-order Taylor expansion during optimization.
  • Hyperparameter Tuning: The method introduces a new parameter, the Huber threshold ($\nu$). If $ u$ is too large, the model loses its quantile properties; if it's too small, it mimics the non-differentiable pinball loss. The authors found that smaller values (under 2) generally work best to maintain the integrity of the quantiles.
The Huber approximation smooths the quantile loss function to make it twice-differentiable, a requirement for XGBoost optimization.

Experimental Setup

The authors evaluated QXGBoost using two primary sources:

  1. Simulated Data: A heteroscedastic sine wave ($y = 1.5 \cdot x \cdot \sin(x) + \epsilon$) where the noise level changes depending on the input $x$. This tests the model's ability to adapt to varying levels of uncertainty.
  2. Traffic Noise Data: A real-world environmental dataset from Long Beach, CA. It contains 6,647 geo-referenced observations of A-weighted equivalent noise levels ($L_{Aeq}$) with 44 predictor variables, including meteorology, land use, and road features.

Key Results and Insights

The authors used the Coverage Width-based Criterion (CWC), a metric that penalizes models for having poor coverage or excessively wide intervals, alongside the Prediction Interval Normalized Average Width (PINAW).

Performance comparison showing QXGBoost achieving superior CWC and narrower intervals compared to GBM and LightGBM.
  • Superior Efficiency: On the traffic noise dataset, QXGBoost achieved a significantly lower CWC than its competitors, outperforming both Quantile LightGBM and Quantile GBM. This indicates a better balance between "Coverage" (how often the true value is inside the interval) and "Width" (how narrow and useful that interval is).
  • Narrower Intervals: QXGBoost consistently produced smaller PINAW values, meaning it provided the most precise "tight" bounds while still capturing the target number of data points.
  • Robustness to Overfitting: In traffic noise experiments, QXGBoost showed better generalization than Quantile LightGBM. While LightGBM often had higher training coverage, QXGBoost maintained more consistent performance on the testing set.
  • Point Accuracy: Even when focused on intervals, QXGBoost maintained a lower Root Mean Square Error (1.89) for point predictions compared to GBM and LightGBM (both 2.04).

Limitations and Future Work

Despite its success, the model struggled at the extreme ends of the traffic noise spectrum (very low or very high noise) due to data sparsity. Furthermore, to achieve a "hard" 90% coverage, the authors currently use manual padding of the intervals. Future work will focus on optimizing the objective function to hit these coverage targets automatically and simplifying the tuning of the Huber threshold ($\nu$), which currently requires careful cross-validation.

Summary updated

Abstract

As the availability, size and complexity of data have increased in recent years, machine learning (ML) techniques have become popular for modeling. Predictions resulting from applying ML models are often used for inference, decision-making, and downstream applications. A crucial yet often overlooked aspect of ML is uncertainty quantification, which can significantly impact how predictions from models are used and interpreted. Extreme Gradient Boosting (XGBoost) is one of the most popular ML methods given its simple implementation, fast computation, and sequential learning, which make its predictions highly accurate compared to other methods. However, techniques for uncertainty determination in ML models such as XGBoost have not yet been universally agreed among its varying applications. We propose enhancements to XGBoost whereby a modified quantile regression is used as the objective function to estimate uncertainty (QXGBoost). Specifically, we included the Huber norm in the quantile regression model to construct a differentiable approximation to the quantile regression error function. This key step allows XGBoost, which uses a gradient-based optimization algorithm, to make probabilistic predictions efficiently. QXGBoost was applied to create 90\% prediction intervals for one simulated dataset and one real-world environmental dataset of measured traffic noise. Our proposed method had comparable or better performance than the uncertainty estimates generated for regular and quantile light gradient boosting. For both the simulated and traffic noise datasets, the overall performance of the prediction intervals from QXGBoost were better than other models based on coverage width-based criterion.
BibTeX
@misc{yin2023quantile,
  author = {Xiaozhe Yin and Masoud Fallah Shorshani and Rob McConnell and Scott Fruin and Yao-Yi Chiang and Meredith Franklin},
  title = {{Quantile Extreme Gradient Boosting for Uncertainty Quantification}},
  year = {2023},
  eprint = {2304.11732},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.11732},
}
Figure