Non-Crossing Quantile Regression

144 concepts106 papers1 proposal

Standard regression models are built to answer a single question: "What is the expected value of the outcome?" By predicting the conditional mean, ordinary least squares and standard neural networks collapse the rich, messy uncertainty of the real world into a single average.

Quantile Regression (QR) offers a more complete picture. Instead of predicting just the average, it estimates specific percentiles—such as the 10th, 50th, or 90th percentiles—of the conditional distribution. This allows practitioners to map out the full range of potential outcomes, which is essential for high-stakes decision-making under uncertainty.

However, when standard machine learning models are tasked with predicting multiple quantiles, they routinely run into a fundamental mathematical paradox known as quantile crossing.


The Quantile Crossing Paradox

By definition, a cumulative distribution function (CDF) must be monotonically non-decreasing. If you ask for the 90th percentile ($\tau = 0.90$) of house prices in a neighborhood, that value must logically be greater than or equal to the 50th percentile ($\tau = 0.50$).

Quantile crossing occurs when a model violates this rule, predicting that a higher percentile is lower than a lower percentile.

<img src="https://images.unsplash.com/photo-1551288049-bebda4e38f71?auto=format&fit=crop&w=800&q=80" alt="Data visualization abstract" style="width:100%; max-width:600px; margin: 20px auto; display:block; border-radius: 8px;"/>

To understand why this happens, consider how standard quantile regression models are trained. To estimate a quantile $\tau$, models minimize the pinball loss (or asymmetric absolute loss). When we want to predict a set of multiple quantiles, the most straightforward approach is to optimize a separate model—or a separate output head of a neural network—for each target quantile level $\tau$.

Because these models or output heads are optimized independently, they lack any awareness of each other. The model estimating the 90th percentile operates completely blind to the model estimating the 80th percentile. Several factors routinely drive these independent curves to intersect:

  • Sampling Noise: In regions of the input space with sparse data, random sampling fluctuations can easily cause a lower quantile estimate to jump above a higher one, as noted by Tang et al..
  • Optimization Variance: In deep neural networks, stochastic gradient descent introduces optimization noise. As Liu et al. demonstrate, this noise can cause different confidence layers to overlap illogically.
  • Model Misspecification: If the true underlying data distribution is highly complex or heteroscedastic (where variance changes across the input space), a linear or insufficiently flexible model will force straight lines or simple curves to cross in edge regions of the input space, a phenomenon detailed by Carlier et al..

Why Non-Monotonicity Breaks Downstream Systems

When quantiles cross, the implied cumulative distribution function collapses. A crossing event means that the derivative of the inverse CDF becomes negative, yielding a mathematically invalid probability density function (PDF) with negative probabilities.

This structural failure makes the model's outputs untrustworthy for downstream applications:

Distributional Reinforcement Learning

In distributional RL, agents do not just learn the expected reward of an action; they learn the entire distribution of future returns. As Zhou et al. explain, when estimated return distributions exhibit crossing quantiles, the resulting probability density becomes invalid. This lack of structural integrity introduces noisy gradients, destabilizes training, and prevents risk-sensitive agents from accurately evaluating worst-case scenarios.

Probabilistic Forecasting

In fields like energy grid management or supply chain logistics, predicting demand boundaries is vital. Koochali et al. point out that quantile crossing prevents the derivation of a coherent probability density function, making it impossible for operators to calculate reliable risk metrics.

Conformal Prediction and Calibration

Methods like Conformalized Quantile Regression rely on predicted quantiles to construct rigorous prediction intervals. If the lower bound exceeds the upper bound, the intervals become nonsensical. Yaniv Romano notes that while heuristic post-processing can sort these bounds after the fact, it fails to address the underlying lack of monotonicity constraints during the model's actual training phase.


Strategies for Enforcing Monotonicity

To eliminate this paradox, researchers have developed several paradigms to ensure that predicted quantiles strictly respect their natural ordering. These solutions range from simple post-hoc corrections to rigid architectural constraints.

Quantile Monotonization Strategies
│
├── Post-Hoc Rearrangement (Sorting outputs after training)
│
├── Incremental Architectures (Predicting base + positive offsets)
│
├── Functional & Convex Constraints (Using monotonic splines or ICCVs)
│
└── Optimal Transport (Mapping multivariate quantiles via Brenier maps)

1. Post-Hoc Rearrangement

The simplest approach is to let independent models cross during training, and then apply a mathematical "rearrangement" operator during inference. As Chao Cheng notes, sorting the estimated values restores a logically consistent distribution. While computationally efficient for small samples, this approach acts merely as a band-aid; it does not fix the underlying gradients during training, meaning the model never truly learns the joint structure of the distribution.

2. Architectural Monotonicity (The Incremental Approach)

Instead of predicting absolute quantile values independently, neural networks can be designed to predict a baseline quantile followed by strictly positive increments.

For instance, Zhou et al. introduce an elegant architectural constraint using a combination of a Softmax layer and a Cumulative Sum. If the network outputs a vector of positive values $\Delta_1, \Delta_2, \dots, \Delta_n$, the quantiles are reconstructed as:

$$q(\tau_i) = q(\tau_0) + \sum_{j=1}^i \Delta_j$$

Because each $\Delta_j > 0$, $q(\tau_i)$ is mathematically guaranteed to be greater than $q(\tau_{i-1})$, completely eliminating the possibility of crossing.

3. Functional Constraints and Monotonic Splines

Another approach models the entire quantile function continuously rather than predicting a discrete set of points. Jan Gasthaus et al. and Danielle C. Maddix et al. leverage monotonic regression splines within recurrent neural networks. By constraining the spline coefficients to be positive, the network models a continuous, smooth, and strictly increasing inverse CDF. This allows users to query any arbitrary quantile level at inference time without retraining the model.

4. Convex Neural Networks and Partial Derivatives

By exploiting the properties of Input Convex Neural Networks (ICNNs), models can guarantee monotonicity through calculus. If a network's output is strictly convex with respect to its quantile input $\tau$, its partial derivative $\frac{\partial f}{\partial \tau}$ is guaranteed to be monotonically increasing. Kelvin Kan et al. and Axel Brando et al. utilize this property to generate temporally consistent, non-crossing sample paths by structuring the forecast as the gradient of a convex potential.

5. Joint Optimization and Multi-Task Learning

Instead of isolating the quantile tasks, models can optimize them jointly within a shared mathematical space. Maxime Sangnier frames joint quantile regression using vector-valued Reproducing Kernel Hilbert Spaces (RKHSs). By treating the estimation of different quantiles as a multi-task learning problem, the optimization framework curbs crossing behavior by sharing structural information across the entire quantile spectrum simultaneously.

6. Moving Beyond One Dimension: Vector Quantile Regression

When moving from a single target variable to multivariate targets (e.g., predicting both the future price and volume of an asset), the concept of a "quantile" becomes geometric. Guillaume Carlier et al. and Jesse Sun et al. solve multivariate quantile crossing by leveraging Optimal Transport theory. By learning a "Brenier map"—the gradient of a strictly convex function—these frameworks map a simple reference distribution to a complex target distribution, guaranteeing that the multivariate quantile surfaces remain nested and mathematically consistent.

Key Foundational Work

Linear discriminant model for information retrieval (2005) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 103 papers
  2. Version 1 · 2026-10-04 · 140 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_quantile_crossing_549d26dc,
  title = {quantile crossing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quantile-crossing/rdr_549d26dc8187420980b980628a58f96a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_quantile_crossing_549d26dc_v2,
  title = {quantile crossing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quantile-crossing/rdr_549d26dc8187420980b980628a58f96a/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake