Abstract
Quantile Regression (QR) provides a way to approximate a single conditional quantile. To have a more informative description of the conditional distribution, QR can be merged with deep learning techniques to simultaneously estimate multiple quantiles. However, the minimisation of the QR-loss function does not guarantee non-crossing quantiles, which affects the validity of such predictions and introduces a critical issue in certain scenarios. In this article, we propose a generic deep learning algorithm for predicting an arbitrary number of quantiles that ensures the quantile monotonicity constraint up to the machine precision and maintains its modelling performance with respect to alternative models. The presented method is evaluated over several real-world datasets obtaining state-of-the-art results as well as showing that it scales to large-size data sets.
Introduction
Quantile Regression (QR) is a fundamental technique for estimating the conditional distribution of a response variable. Unlike standard regression which predicts a single point estimate (usually the mean), QR allows for the estimation of various percentiles, providing a comprehensive view of uncertainty. When integrated with deep learning, models like Implicit Quantile Networks (IQN) can theoretically learn the entire quantile function $\Phi(\tau, \mathbf{x})$.
However, a persistent challenge in deep QR is the "crossing quantile" phenomenon. Because neural networks lack inherent monotonicity constraints, they often predict higher quantiles to be lower than lower quantiles (e.g., the 90th percentile being lower than the 50th). This violation of basic probability theory renders uncertainty estimates unreliable and mathematically invalid. Existing solutions often rely on soft penalties or rigid architectural constraints that limit model expressivity.
Methodology: Integration of the Partial Derivative
The proposed framework addresses the crossing problem by leveraging the Fundamental Theorem of Calculus. Instead of directly predicting the quantile value, the model predicts the partial derivative of the quantile function with respect to the quantile level $\tau$.
The core components of the architecture are:
- Positive Derivative Modeling: A neural network $\phi_w$ outputs the derivative, constrained to be strictly positive using an activation function like
softplusorReLU. - Chebyshev Integration: To recover the quantile function, the derivative is integrated. The authors employ a truncated Chebyshev polynomial expansion, which is a standard numerical technique for efficient integration.
- Efficient Computation: By utilizing the Discrete Cosine Transform (DCT), the model computes polynomial coefficients and integrals with $\Theta(d \log d)$ complexity, where $d$ is the polynomial degree.
- The Quantile Function: The final prediction is formulated as: $$P(\tau, \mathbf{x}) = K_w(\mathbf{x}) + \int_0^\tau \phi_w(t, \mathbf{x}) dt$$ where $K_w(\mathbf{x})$ is a learned constant of integration representing a base quantile or the mean.
Experimental Evaluation
The method was tested on a synthetic multi-modal "Glasses" dataset and eight UCI benchmark datasets, including Housing, Naval, and Protein. These benchmarks are the standard for testing regression generalization.
Key Results
- Guaranteed Monotonicity: In UCI benchmarks, the proposed models (
Ours-q0andOurs-Mean) achieved zero crossing quantiles at evaluation points. In contrast, the baseline IQN model produced over 100,000 crossings in the Naval dataset. - Predictive Performance: The model achieved state-of-the-art log-likelihood scores, outperforming IQN and other constrained variants like PCDN. This demonstrates that enforcing monotonicity does not come at the cost of predictive accuracy.
- Scalability: The DCT-based integration ensures minimal computational overhead, making the approach suitable for large-scale datasets.
Conclusion
By shifting the learning objective to the partial derivative of the quantile function, this research provides a mathematically rigorous and computationally efficient solution to the crossing quantile problem. The use of Chebyshev polynomials and DCT allows for machine-precision monotonicity without sacrificing the flexibility of deep neural networks. Future work includes extending this framework to Distributional Reinforcement Learning, where valid quantile estimates are crucial for stable agent training.
Reference: Brando, A., Gimeno, J., Rodríguez-Serrano, J. A., & Vitrià, J. (2022). Deep Non-Crossing Quantiles through the Partial Derivative. AISTATS.
Summary updated