Probabilistic Forecasting: A Level-Set Approach

Hilaf Hasson, Bernie Wang, Tim Januschowski, Jan GasthausNeurIPS 2021 · OpenReview

TL;DR

The Level Set Forecaster (LSF) is a simple yet powerful wrapper algorithm that transforms standard point-estimation models (like XGBoost) into probabilistic ones. By grouping training data with similar predictions into "level sets," LSF generates full predictive distributions that rival state-of-the-art deep learning models in accuracy while remaining significantly faster than traditional methods like Quantile Regression Forests (QRFs).

The Challenge of Probabilistic Forecasting

In fields ranging from retail and IoT to medicine, knowing the single "most likely" future value—a point forecast—is rarely sufficient. Decision-makers need to understand uncertainty, which requires knowing the range of possible outcomes and their associated probabilities.

While deep learning models (such as DeepAR) naturally handle probabilistic forecasting well, tree-based methods (like XGBoost or LightGBM) remain the gold standard for tabular and time-series data due to their robustness, accuracy, and ease of use. However, turning these point-estimators into probabilistic models has historically been difficult. Existing solutions, such as Quantile Regression Forests (QRFs), are often computationally expensive and cannot easily leverage modern, highly optimized gradient-boosting libraries. This paper bridges that gap with a method that is both theoretically sound and computationally efficient.

The Level Set Forecaster (LSF)

The authors introduce the Level Set Forecaster (LSF) based on an elegant intuition: if a model predicts similar values for two different inputs, those inputs likely belong to the same "level set" and should share a similar uncertainty profile. The algorithm operates in three steps:

  1. Training: Train any standard point estimator (e.g., XGBoost) on the target dataset.
  2. Binning: Group the training data points whose predicted values are "close" to each other into discrete bins.
  3. Inference: For a new test point, generate the point prediction from the base model and identify the corresponding bin of training data with similar predictions. The actual observed values ($y$-targets) in that bin form the empirical predictive distribution for the new point.

The paper provides a specific implementation called XLSF (LSF applied to XGBoost) and a Tree-based Time Series Wrapper (TTSW) to handle time-series data by converting it into a tabular format using lag features.

Key Insights and Theoretical Contributions

  • Output-Space Neighborhoods: The core innovation is realizing that you do not need to alter how a model learns to extract probabilities. Instead, you can look at the "neighborhood" of its predictions in the output space rather than the input feature space.
  • Theoretical Milestone: The authors provide the first consistency proof for Quantile Regression Forests (QRFs) under the standard CART-splitting criterion, filling a long-standing gap in machine learning theory.
  • Non-Parametric Flexibility: Because LSF uses empirical samples from the training data to form the distribution, it makes no assumptions about the noise distribution (e.g., it does not assume Gaussian noise).

Datasets and Key Results

The paper evaluates LSF across tabular regression tasks (10 datasets including facebook, meps, and bike) and time-series forecasting benchmarks (including the M5 Competition dataset, electricity, and traffic).

Performance comparison of XLSF, QRF, and Conformalized Predictions
  • Computational Efficiency: On tabular datasets, XLSF proved to be drastically faster than QRFs. As shown in the performance table, on the facebook2 dataset, XLSF took approximately 14.8 seconds while QRF took over 103 seconds, all while maintaining competitive or superior accuracy.
  • Deep Learning Rivalry: In time-series benchmarks, the tree-based TTSW (XLSF) performed as well as, or better than, state-of-the-art deep learning models like DeepAR and CNN-QR, particularly on datasets like electricity and m4_daily.
  • M5 Competition: Simply "wrapping" the winning point-forecast solution of the M5 competition with LSF made it competitive with the top 5 specialized entries in the uncertainty track.

Limitations and Future Work

Like all tree-based methods, LSF is constrained by the range of its training data and cannot extrapolate trends beyond historical minimums and maximums. Additionally, the model's performance is sensitive to the min_bin_size hyperparameter: bins that are too small yield noisy distributions, while bins that are too large lose precision.

Future work may explore applying LSF to more diverse base learners beyond trees—such as k-nearest neighbors or neural networks—to provide a "second opinion" on uncertainty. Refining the partitioning algorithm to optimize how bins are merged also remains an active area for further research.

Summary updated

Abstract

Large-scale time series panels have become ubiquitous over the last years in areas such as retail, operational metrics, IoT, and medical domain (to name only a few). This has resulted in a need for forecasting techniques that effectively leverage all available data by learning across all time series in each panel. Among the desirable properties of forecasting techniques, being able to generate probabilistic predictions ranks among the top. In this paper, we therefore present Level Set Forecaster (LSF), a simple yet effective general approach to transform a point estimator into a probabilistic one. By recognizing the connection of our algorithm to random forests (RFs) and quantile regression forests (QRFs), we are able to prove consistency guarantees of our approach under mild assumptions on the underlying point estimator. As a byproduct, we prove the first consistency results for QRFs under the CART-splitting criterion. Empirical experiments show that our approach, equipped with tree-based models as the point estimator, rivals state-of-the-art deep learning models in terms of forecasting accuracy.
BibTeX
@inproceedings{hasson2021probabilistic,
  author = {Hilaf Hasson and Bernie Wang and Tim Januschowski and Jan Gasthaus},
  title = {{Probabilistic Forecasting: A Level-Set Approach}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=VD3TMzyxKK},
}

Loading related content...

Figure