MVG-CRPS: A Robust Loss Function for Multivariate Probabilistic Forecasting

Vincent Zhihao Zheng, Lijun Sun

Abstract

Multivariate Gaussian (MVG) distributions are central to modeling correlated continuous variables in probabilistic forecasting. Neural forecasting models typically parameterize the mean vector and covariance matrix of the distribution using neural networks, optimizing with the log-score (negative log-likelihood) as the loss function. However, the sensitivity of the log-score to outliers can lead to significant errors in the presence of anomalies. Drawing on the continuous ranked probability score (CRPS) for univariate distributions, we propose MVG-CRPS, a strictly proper scoring rule for MVG distributions. MVG-CRPS admits a closed-form expression in terms of neural network outputs, thereby integrating seamlessly into deep learning frameworks. Experiments on real-world datasets across multivariate autoregressive and univariate sequence-to-sequence (Seq2Seq) forecasting tasks show that MVG-CRPS improves robustness, accuracy, and uncertainty quantification in probabilistic forecasting.

Introduction

In probabilistic forecasting, the goal is to predict a probability distribution rather than a single point estimate to capture inherent uncertainty. For multivariate data—such as predicting power demand across hundreds of sensors—the Multivariate Gaussian (MVG) distribution is the standard choice.

Currently, most models are trained using the log-score (Negative Log-Likelihood). However, the log-score is notoriously sensitive to outliers. Because it penalizes errors exponentially, a single anomalous data point can skew the model, leading to inflated variance and poor calibration. While the Energy Score (ES) offers a robust alternative, it lacks a closed-form solution for MVG distributions, requiring computationally expensive Monte Carlo sampling during training. MVG-CRPS is designed to bridge this gap, offering the robustness of the Energy Score with the computational efficiency of the log-score.

The MVG-CRPS Method

The core innovation of MVG-CRPS is the use of a whitening transformation to derive a closed-form expression for the multivariate Continuous Ranked Probability Score. The process involves:

  1. Decorrelation: The model predicts a mean vector and a covariance matrix. Using Singular Value Decomposition (SVD), the authors perform a PCA whitening transformation, rotating and scaling the multivariate data into a space where all variables are independent standard Gaussians.
  2. Univariate Application: With independent variables, the standard univariate CRPS formula—which has a simple, differentiable closed-form expression—can be applied to each component individually.
  3. Strict Propriety: The authors mathematically prove that this scoring rule is "strictly proper," ensuring the loss is minimized if and only if the predicted distribution matches the true data distribution.

Experimental Results

The authors evaluated MVG-CRPS using the GluonTS library on two main tasks: Multivariate Autoregressive Forecasting (using models like GPVar and Transformers) and Univariate Seq2Seq Forecasting (using N-HiTS).

Efficiency and Accuracy

MVG-CRPS consistently outperformed the log-score in accuracy. For instance, on the electricity dataset, it achieved a CRPS score of 0.0249, a significant improvement over the log-score's 0.0419. Crucially, MVG-CRPS provides a massive speedup over sampling-based methods. On the elec_au dataset, training with the Energy Score took 717 minutes, whereas MVG-CRPS completed in just 29 minutes.

Figure 1: Comparison of CRPS vs. training time across different loss functions on the elec_au dataset. MVG-CRPS achieves high accuracy with significantly lower training time than the Energy Score.

Qualitative Robustness

Visual analysis demonstrates that models trained with the log-score often produce "exploded" covariance matrices when encountering outliers, leading to overly wide and unreliable prediction intervals. In contrast, MVG-CRPS models remain calibrated, producing sharper and more reliable uncertainty estimates.

Figure 2: Qualitative comparison of forecasting performance between Log-Score and MVG-CRPS. MVG-CRPS provides more reliable and sharper prediction intervals across different time series (TS).

Conclusion

MVG-CRPS offers a robust, efficient, and mathematically grounded alternative to standard loss functions in multivariate forecasting. By utilizing a linear penalty rather than an exponential one, it maintains stability in the presence of real-world data noise. Future work may extend this approach to non-Gaussian settings using copulas or optimize the SVD whitening process for extremely high-dimensional joint forecasting tasks.

Summary updated

Abstract

Multivariate Gaussian (MVG) distributions are central to modeling correlated continuous variables in probabilistic forecasting. Neural forecasting models typically parameterize the mean vector and covariance matrix of the distribution using neural networks, optimizing with the log-score (negative log-likelihood) as the loss function. However, the sensitivity of the log-score to outliers can lead to significant errors in the presence of anomalies. Drawing on the continuous ranked probability score (CRPS) for univariate distributions, we propose MVG-CRPS, a strictly proper scoring rule for MVG distributions. MVG-CRPS admits a closed-form expression in terms of neural network outputs, thereby integrating seamlessly into deep learning frameworks. Experiments on real-world datasets across multivariate autoregressive and univariate sequence-to-sequence (Seq2Seq) forecasting tasks show that MVG-CRPS improves robustness, accuracy, and uncertainty quantification in probabilistic forecasting.
BibTeX
@misc{zheng2024mvg,
  author = {Vincent Zhihao Zheng and Lijun Sun},
  title = {{MVG-CRPS: A Robust Loss Function for Multivariate Probabilistic Forecasting}},
  year = {2024},
  eprint = {2410.09133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.09133},
}
Figure