Abstract
In multi-target regression, traditional uncertainty quantification methods often produce overly conservative, convex prediction regions that fail to capture the complex, non-convex dependencies between target variables. We present Volume-Sorted Prediction Set (VSPS), a novel framework that combines conditional normalizing flows with conformal prediction to generate tight, shape-agnostic uncertainty regions. By leveraging the Jacobian determinant of the learned transformation, VSPS identifies high-density areas in the response space, producing prediction sets that are mathematically guaranteed to meet a user-specified coverage probability while significantly reducing the volume of the predicted region compared to state-of-the-art baselines.
Introduction
Predicting multiple correlated outcomes simultaneously—such as the efficacy and side-effect profile of a new pharmaceutical—is a fundamental challenge in modern machine learning. While point predictions provide a best estimate, high-stakes applications require rigorous uncertainty quantification (UQ). Standard UQ approaches typically treat targets independently, resulting in rectangular prediction boxes, or force joint distributions into simple geometric shapes like ellipsoids.
However, real-world data often exhibits complex, non-convex structures, such as "V-shapes," curves, or even disconnected clusters. Forcing these distributions into convex hulls leads to "empty space" within the prediction set, reducing the informativeness of the model. The Volume-Sorted Prediction Set (VSPS) addresses this by creating flexible, non-convex regions that adapt to the underlying data topology while maintaining formal coverage guarantees.
Methodology
1. Conditional Normalizing Flows (CNFs)
VSPS utilizes CNFs to learn a bijective (invertible) and differentiable mapping between a complex multi-target response distribution and a simple latent space (typically a standard Gaussian). Because the mapping is smooth and invertible, the model can use the Jacobian determinant to calculate the exact change in volume between the original space and the latent space. This allows for exact density evaluation, a significant improvement over methods like Variational Autoencoders (VAEs) that only optimize a lower bound.
2. Volume-Sorted Sampling
To construct the prediction set, the framework samples points from the latent space and maps them back to the original target space. These samples are then sorted based on their Jacobian determinants. Points with higher determinants correspond to regions of higher probability density in the original space. This "density over distance" approach ensures that the prediction set prioritizes areas where data is most likely to reside, rather than simply buffering around a mean prediction.
3. Conformal Calibration and Shape Construction
The prediction region is defined as a union of balls centered around the top $K$ high-density samples. The radius of these balls, denoted as $\gamma$, is calibrated using a held-out calibration dataset. This step ensures that the resulting region satisfies the marginal coverage guarantee (e.g., a 90% probability that the true value falls within the set). Furthermore, VSPS includes an optimization algorithm to select the ideal number of samples ($K$) to minimize the total volume of the prediction set without violating the coverage constraint.
Experimental Results
Synthetic Validation
The researchers first tested VSPS on a synthetic 2D "V-shaped" distribution. Traditional Naïve Quantile Regression (QR) produced large, inefficient boxes to cover the non-convex shape. In contrast, VSPS generated a tight, curved region that followed the data distribution perfectly. Quantitatively, VSPS produced prediction regions over 10 times smaller than Naïve QR while maintaining the same 90% coverage.
Real-World Benchmarks
VSPS was evaluated against several baselines, including ST-DQR and Naïve QR, across diverse datasets:
- Bio: Physicochemical properties of protein tertiary structures.
- Blog: Blog feedback data.
- House: King County house sales prices.
- MEPS (2019–2021): Medical Expenditure Panel Survey data.
Across all datasets, VSPS consistently achieved the smallest prediction region sizes. On the "Bio" dataset, for instance, VSPS regions were approximately 50% smaller than those of the nearest competitor (ST-DQR). Importantly, the model maintained reliable marginal coverage and demonstrated superior conditional coverage, remaining robust across different subsets of input data.
Discussion and Limitations
While VSPS offers a significant leap in efficiency and flexibility, it introduces certain trade-offs. The process of sampling, calculating Jacobians, and sorting for every test instance is more computationally intensive than standard interval calculations. Additionally, while the "union of balls" approach is highly effective for low-to-moderate dimensions (2D or 3D), higher-dimensional response spaces may eventually require more complex shape templates to combat the curse of dimensionality.
Finally, the effectiveness of VSPS is intrinsically linked to the quality of the underlying Normalizing Flow. While the conformal calibration step ensures that the coverage guarantee is always met (by increasing the radius $\gamma$ if the flow is poor), the efficiency (tightness) of the set depends on the flow's ability to accurately model the data distribution.
Conclusion
Volume-Sorted Prediction Set (VSPS) represents a significant advancement in multi-target regression. By shifting the focus from distance-based scores to density-based volume sorting, it provides a mathematically rigorous yet highly flexible framework for uncertainty quantification. Its ability to adapt to any data topology makes it a powerful tool for complex, real-world predictive tasks where precision and reliability are paramount.
Summary updated