Abstract
Quantile regression offers a robust alternative to standard mean regression by modeling specific percentiles of a response distribution, which is particularly valuable for skewed or heteroscedastic data. This paper introduces a nonparametric Bayesian framework for quantile regression that leverages Gaussian Processes (GPs) and Variational Inference (VB). By framing the pinball loss function as an Asymmetric Laplace Distribution (ALD) likelihood and utilizing a Gaussian scale mixture representation, we develop a stable and efficient inference algorithm. Experimental evaluations on real-world datasets, such as bone mineral density and motorcycle crash data, demonstrate that the proposed VB approach provides better convergence and higher accuracy for extreme quantiles than traditional Expectation Propagation (EP) methods.
Introduction
Standard regression models typically estimate the conditional mean, which can be misleading when data distributions are skewed, multimodal, or heteroscedastic. Quantile regression addresses this by estimating various quantiles, providing a more complete statistical picture of the relationship between variables. While Bayesian quantile regression is desirable for its principled uncertainty quantification, existing methods often rely on computationally expensive MCMC or unstable Expectation Propagation (EP) approximations. This work proposes a flexible GP-based model optimized via a Variational Bayesian EM algorithm, ensuring monotonic convergence and robust performance across complex datasets.
Methodology
The proposed framework centers on the Asymmetric Laplace Distribution (ALD). In quantile regression, the pinball loss function is used to define quantiles; this loss is mathematically equivalent to the log-likelihood of an ALD. To facilitate efficient Bayesian inference, the authors employ a Gaussian scale mixture trick, representing the ALD as a mixture of Gaussians by introducing a latent variable $w$.
Variational Bayesian Inference
Rather than using slow MCMC sampling, the authors derive a Variational Bayesian (VB) approximation. This approach provides an analytical lower bound on the marginal likelihood, which is maximized using an Expectation-Maximization (EM) algorithm. Unlike EP, which lacks theoretical convergence guarantees and can be unstable, the VB framework ensures monotonic convergence to a local optimum, making it more reliable for practical applications.
Experimental Evaluation
The performance of the VB-GP model was compared against EP-based methods on several benchmark datasets.
Bone Density Dataset
The Bone Density dataset involves predicting mineral density as a function of age, characterized by non-symmetric noise and varying data density.
As shown in Figure 3, the VB method (b) provides a more consistent fit across the age range compared to EP (a). The superimposed view (c) highlights the differences in quantile estimates, where VB demonstrates superior stability, particularly in regions where data is sparse.
Motorcycle Dataset
The Motorcycle dataset measures helmet acceleration during a crash, exhibiting significant heteroscedasticity over time.
In Figure 6, the VB method (a) effectively captures the complex, time-varying variance of the acceleration. Notably, the VB approach excels at estimating extreme quantiles (e.g., 0.99), which are critical for safety analysis but often poorly handled by parametric or EP-based methods (b).
Key Insights
- Handling Non-Symmetry: The ALD-based likelihood allows the model to accurately represent skewed distributions without assuming Gaussian noise, a common limitation in standard GP regression.
- Theoretical Stability: The VB framework's use of a lower bound ensures stable optimization of hyperparameters, providing a significant advantage over the often unstable EP method.
- Nonparametric Flexibility: By utilizing Gaussian Processes, the model learns complex, nonlinear patterns directly from the data without requiring manual parametric specification.
Conclusion
This paper presents a robust and flexible Bayesian approach to quantile regression. By combining Gaussian Processes with Variational Inference, the model achieves high accuracy and theoretical stability. While the current implementation scales at $O(N^3)$, future work will focus on scaling the method to larger datasets using sparse GP techniques and incorporating constraints to prevent quantile crossing.
Summary updated