Variational Inference for Nonparametric Bayesian Quantile Regression

Sachinthaka Abeywardana, Fabio Ramos

Abstract

Quantile regression offers a robust alternative to standard mean regression by modeling specific percentiles of a response distribution, which is particularly valuable for skewed or heteroscedastic data. This paper introduces a nonparametric Bayesian framework for quantile regression that leverages Gaussian Processes (GPs) and Variational Inference (VB). By framing the pinball loss function as an Asymmetric Laplace Distribution (ALD) likelihood and utilizing a Gaussian scale mixture representation, we develop a stable and efficient inference algorithm. Experimental evaluations on real-world datasets, such as bone mineral density and motorcycle crash data, demonstrate that the proposed VB approach provides better convergence and higher accuracy for extreme quantiles than traditional Expectation Propagation (EP) methods.

Introduction

Standard regression models typically estimate the conditional mean, which can be misleading when data distributions are skewed, multimodal, or heteroscedastic. Quantile regression addresses this by estimating various quantiles, providing a more complete statistical picture of the relationship between variables. While Bayesian quantile regression is desirable for its principled uncertainty quantification, existing methods often rely on computationally expensive MCMC or unstable Expectation Propagation (EP) approximations. This work proposes a flexible GP-based model optimized via a Variational Bayesian EM algorithm, ensuring monotonic convergence and robust performance across complex datasets.

Methodology

The proposed framework centers on the Asymmetric Laplace Distribution (ALD). In quantile regression, the pinball loss function is used to define quantiles; this loss is mathematically equivalent to the log-likelihood of an ALD. To facilitate efficient Bayesian inference, the authors employ a Gaussian scale mixture trick, representing the ALD as a mixture of Gaussians by introducing a latent variable $w$.

Variational Bayesian Inference

Rather than using slow MCMC sampling, the authors derive a Variational Bayesian (VB) approximation. This approach provides an analytical lower bound on the marginal likelihood, which is maximized using an Expectation-Maximization (EM) algorithm. Unlike EP, which lacks theoretical convergence guarantees and can be unstable, the VB framework ensures monotonic convergence to a local optimum, making it more reliable for practical applications.

Experimental Evaluation

The performance of the VB-GP model was compared against EP-based methods on several benchmark datasets.

Bone Density Dataset

The Bone Density dataset involves predicting mineral density as a function of age, characterized by non-symmetric noise and varying data density.

Comparison of EP and VB methods on the Bone Density dataset.

As shown in Figure 3, the VB method (b) provides a more consistent fit across the age range compared to EP (a). The superimposed view (c) highlights the differences in quantile estimates, where VB demonstrates superior stability, particularly in regions where data is sparse.

Motorcycle Dataset

The Motorcycle dataset measures helmet acceleration during a crash, exhibiting significant heteroscedasticity over time.

Comparison of VB and EP on the Motorcycle crash dataset.

In Figure 6, the VB method (a) effectively captures the complex, time-varying variance of the acceleration. Notably, the VB approach excels at estimating extreme quantiles (e.g., 0.99), which are critical for safety analysis but often poorly handled by parametric or EP-based methods (b).

Key Insights

  • Handling Non-Symmetry: The ALD-based likelihood allows the model to accurately represent skewed distributions without assuming Gaussian noise, a common limitation in standard GP regression.
  • Theoretical Stability: The VB framework's use of a lower bound ensures stable optimization of hyperparameters, providing a significant advantage over the often unstable EP method.
  • Nonparametric Flexibility: By utilizing Gaussian Processes, the model learns complex, nonlinear patterns directly from the data without requiring manual parametric specification.

Conclusion

This paper presents a robust and flexible Bayesian approach to quantile regression. By combining Gaussian Processes with Variational Inference, the model achieves high accuracy and theoretical stability. While the current implementation scales at $O(N^3)$, future work will focus on scaling the method to larger datasets using sparse GP techniques and incorporating constraints to prevent quantile crossing.

Summary updated

Abstract

Quantile regression deals with the problem of computing robust estimators when the conditional mean and standard deviation of the predicted function are inadequate to capture its variability. The technique has an extensive list of applications, including health sciences, ecology and finance. In this work we present a non-parametric method of inferring quantiles and derive a novel Variational Bayesian (VB) approximation to the marginal likelihood, leading to an elegant Expectation Maximisation algorithm for learning the model. Our method is nonparametric, has strong convergence guarantees, and can deal with nonsymmetric quantiles seamlessly. We compare the method to other parametric and non-parametric Bayesian techniques, and alternative approximations based on expectation propagation demonstrating the benefits of our framework in toy problems and real datasets.
BibTeX
@inproceedings{abeywardana2015variational,
  author = {Sachinthaka Abeywardana and Fabio Ramos},
  title = {{Variational Inference for Nonparametric Bayesian Quantile Regression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i1.9465},
}
Figure