Calibrated Multivariate Regression with Localized PIT Mappings

Lucas Kock, G. S. Rodrigues, Scott A. Sisson, Nadja Klein, David J. NottarXiv 2024 · DOI · arXiv

Abstract

Calibration ensures that predicted uncertainties align with observed outcomes, a critical requirement for reliable decision-making. While univariate recalibration is well-studied, multivariate calibration remains a significant challenge, particularly for misspecified models or mixed-type data. This paper introduces a novel post-hoc recalibration approach using localized mappings of Probability Integral Transform (PIT) values. By constructing these mappings via K-Nearest Neighbors (KNN) or Normalizing Flows (NF), we provide a flexible, model-agnostic solution. We demonstrate the efficacy of our method on synthetic benchmarks, currency exchange rate forecasting, and a public health study on childhood malnutrition in India.

Introduction

In probabilistic forecasting, calibration is the gold standard for reliability. A well-calibrated model ensures that if a 90% confidence interval is predicted, the true value falls within that interval 90% of the time. While robust tools exist for calibrating single variables, real-world scenarios often demand multivariate predictions. For instance, financial institutions must assess the joint risk of multiple currencies fluctuating simultaneously, and healthcare providers must evaluate the co-occurrence of multiple symptoms.

Achieving multivariate calibration is notoriously difficult. Most standard models rely on simplifying assumptions—such as Gaussian distributions—that fail to capture complex dependencies. Furthermore, many datasets are "mixed," containing both continuous and discrete variables. This research addresses these gaps by providing a post-hoc method to recalibrate joint predictions without requiring the retraining of the underlying base model.

Methodology: Localized PIT Mappings

The core of the proposed approach is the Localized PIT Mapping technique. This process functions as a "corrective lens" applied to a pre-trained base model through three distinct stages:

1. The PIT Transformation

Predictions from a base model are first transformed into Probability Integral Transform (PIT) values. In a perfectly calibrated system, these values should be uniformly distributed between 0 and 1. Deviations from uniformity indicate miscalibration, such as overconfidence or bias.

2. Localization

Unlike global recalibration methods that apply a single fix to an entire dataset, this method focuses on local neighborhoods within the feature space. This is based on the insight that a model may be well-calibrated for certain types of inputs but highly inaccurate for others. By localizing the fix, the researchers can address specific areas of model failure.

3. Mapping Strategies

The researchers propose two distinct flavors for the mapping process:

  • K-Nearest Neighbors (KNN): This approach identifies similar historical examples and uses their PIT values to adjust current predictions. It is computationally efficient and easy to implement but is limited by the sample size of the validation set.
  • Normalizing Flows (NF): This version utilizes a deep learning architecture (specifically Real-NVP) to learn a complex, bijective mapping between biased PIT values and the true distribution. While more computationally intensive, it allows for the generation of infinite samples from the corrected distribution, providing greater flexibility for complex joint distributions.

Experimental Evaluation

The effectiveness of the approach was tested across diverse domains:

Currency Exchange Rates

Using five years of daily exchange rates for major currencies (AUD, CNY, EUR, GBP, SGD) relative to the USD, the researchers evaluated a base LSTM (Long Short-Term Memory) neural network. The base model was found to be "under-dispersed," meaning its confidence intervals were too narrow and failed to capture the true volatility of the market. The KNN-based recalibration successfully widened these bands during volatile periods, restoring statistical reliability to the forecasts.

Childhood Malnutrition in India

This application involved a "mixed" response task using data from the Demographic and Health Surveys (DHS). The model predicted both a continuous indicator (wasting) and a binary indicator (fever). The base model significantly overestimated the risk of wasting. By applying Normalizing Flow recalibration, the researchers corrected these risk estimates and, crucially, were able to accurately calculate the joint risk of a child suffering from both conditions—a task the independent base models could not perform reliably.

Key Insights and Discussion

  • Model-Agnosticism: The method is entirely "model-free." It can be applied to any architecture, from simple linear regressions to complex deep neural networks, without needing to know the internal mechanics of the base model.
  • Handling Mixed Data: By employing randomized PIT values, the method seamlessly integrates discrete and continuous data, overcoming a common hurdle in multivariate analysis.
  • Preserving Interpretability: This approach allows researchers to maintain simple, interpretable base models for their primary analysis while using the recalibration layer to ensure the resulting uncertainty estimates are trustworthy.

Limitations and Future Work

While the results are promising, several areas for improvement remain. The KNN approach is constrained by the size of the validation data, and the theoretical mathematical properties of these PIT-based mappings require further formalization. Additionally, while the method ensures calibration (reliability), it does not explicitly optimize for sharpness (the narrowness of the prediction intervals). Future research will explore balancing these two factors while reducing the computational overhead associated with training Normalizing Flows for recalibration tasks.

Conclusion

Localized PIT Mappings offer a powerful, flexible framework for correcting the uncertainty estimates of multivariate models. By focusing on local feature neighborhoods and providing tools to handle mixed-type data, this method ensures that complex joint predictions are not only accurate in their mean estimates but also statistically reliable in their expressions of risk.

Summary updated

Abstract

Calibration ensures that predicted uncertainties align with observed uncertainties. While there is an extensive literature on recalibration methods for univariate probabilistic forecasts, work on calibration for multivariate forecasts is much more limited. This paper introduces a novel post-hoc recalibration approach that addresses multivariate calibration for potentially misspecified models. Our method involves constructing local mappings between vectors of marginal probability integral transform values and the space of observations, providing a flexible and model free solution applicable to continuous, discrete, and mixed responses. We present two versions of our approach: one uses K-nearest neighbors, and the other uses normalizing flows. Each method has its own strengths in different situations. We demonstrate the effectiveness of our approach on two real data applications: recalibrating a deep neural network's currency exchange rate forecast and improving a regression model for childhood malnutrition in India for which the multivariate response has both discrete and continuous components.
BibTeX
@misc{kock2024calibrated,
  author = {Lucas Kock and G. S. Rodrigues and Scott A. Sisson and Nadja Klein and David J. Nott},
  title = {{Calibrated Multivariate Regression with Localized PIT Mappings}},
  year = {2024},
  eprint = {2409.10855},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.10855},
}
Figure