Abstract
Conformal methods create prediction bands that control average coverage assuming solely i.i.d. data. Besides average coverage, one might also desire to control conditional coverage, that is, coverage for every possible value of the covariates. This paper introduces two new conformal prediction methods, Dist-split and CD-split, which leverage conditional density estimators to provide reliable prediction bands with asymptotic conditional coverage guarantees.
Introduction
In supervised learning, providing a single point estimate is often insufficient for high-stakes decision-making. We require a range of values—a prediction band—that likely contains the true response. Conformal prediction has emerged as a leading framework for this task because it offers "distribution-free" guarantees, meaning the validity of the bands does not depend on the underlying data distribution.
However, a significant limitation of standard conformal prediction is that it typically only guarantees marginal coverage. For a target coverage of 90%, the bands will be correct 90% of the time on average across the entire population. This can lead to systematic under-coverage for specific subgroups. Ideally, we seek conditional coverage, where the 90% guarantee holds for every specific input $x$. While perfect conditional coverage is impossible without restrictive assumptions, this paper proposes methods that achieve asymptotic conditional coverage, improving local accuracy as more data is collected.
Methodology
The authors propose two primary algorithms that move beyond simple regression-based error margins by utilizing conditional density estimation.
Dist-split
Dist-split estimates the conditional cumulative distribution function (CDF) of the data. It utilizes the probability integral transform: if a random variable is plugged into its own CDF, the resulting values are uniformly distributed. By calibrating the quantiles of these transforms on a hold-out set, Dist-split creates valid prediction intervals.
CD-split
CD-split is designed for maximum flexibility. Instead of being restricted to single intervals, it can produce arbitrary regions, such as the union of multiple disconnected intervals. It targets Highest Density Regions (HDR)—the smallest possible sets that contain the required probability mass. This is particularly effective for multi-modal data where the response might fall into two distinct "humps."
The Profile Distance
To maintain performance in high-dimensional spaces, the authors introduce a novel profile distance. Traditional local methods often suffer from the "curse of dimensionality" because Euclidean neighborhoods become empty as the number of features increases. The profile distance instead groups inputs that have similar predicted density shapes. This allows the model to pool information from "neighbors" in the density space, even if they are far apart in the original feature space, effectively ignoring irrelevant covariates.
Experimental Evaluation
The methods were evaluated across several challenging synthetic scenarios, including asymmetric noise, heteroscedasticity (varying noise levels), and bimodal distributions. All regression experiments utilized 20 covariates, many of which were intentionally irrelevant to test the robustness of the profile distance metric.
Key Results
Superior Conditional Control
In complex scenarios, particularly bimodal and heteroscedastic ones, CD-split and Dist-split demonstrated significantly lower conditional coverage absolute deviation compared to standard baselines like Reg-split (which uses a point estimate plus a global margin). As shown in the figure below, the proposed methods maintain stable coverage across the range of $x$, whereas Reg-split's coverage degrades significantly in certain regions.
Efficiency and Adaptability
CD-split consistently produced the smallest (most efficient) prediction regions. In bimodal data, it achieved this by "skipping" the low-probability space between modes, whereas traditional methods were forced to draw a single, large, and wasteful interval. Furthermore, the methods proved resilient to high dimensionality, effectively filtering out irrelevant features that typically degrade the performance of local regression methods.
Conclusion and Future Work
Dist-split and CD-split provide a robust middle ground between marginal and conditional coverage. By shifting the focus from point regression to density estimation and introducing the profile distance, these methods offer flexible, distribution-free bands that adapt to the local structure of the data. While the efficiency of these bands depends on the quality of the underlying density estimator, the coverage guarantee remains valid. Future research may explore adapting the profile distance to other statistical tasks that currently rely on standard Euclidean metrics.
Summary updated