Conditional Temporal Neural Processes with Covariance Loss

Boseon Yoo, Jiwoo Lee, Janghoon Ju, Seijun Chung, Soyeon Kim, Jaesik Choi

Introduction

In the field of deep learning for regression, the standard objective is to minimize the distance between a model's prediction and the ground-truth value, typically using Mean Squared Error (MSE). While effective for point-wise accuracy, this approach often overlooks the underlying dependencies between target variables—how they co-vary over time or space. Modeling these dependencies is a hallmark of Gaussian Processes (GPs) and Conditional Neural Processes (CNPs), which provide a probabilistic framework for capturing uncertainty and correlation. However, GPs are computationally expensive, and CNPs often require specialized architectures that are not easily integrated into existing state-of-the-art models.

This paper addresses this gap by introducing Covariance Loss, a novel regularization term designed to instill "prior knowledge" of target dependencies into any neural network. By forcing the network to learn basis functions that reflect the empirical covariance of the targets, the authors bridge the gap between standard deep learning and the robust modeling capabilities of GPs without requiring a complete architectural overhaul.

The Covariance Loss Framework

The core contribution of this work is a loss function that can be "plugged in" to existing temporal or spatial neural networks. The total loss consists of two primary components:

  1. Mean Squared Error (MSE): The traditional term that ensures the model's predictions are close to the actual values.
  2. Covariance Regularization: A term that minimizes the difference between the covariance matrix of the learned basis functions $\Phi(X)$ and the empirical covariance matrix of the target variables $Y$.
Overview of the Covariance Loss framework comparing standard training (w/o Cov) to the proposed method (w/ Cov), highlighting the regularization of basis functions against target covariance.

Mathematically, this forces the network to find a feature space where the basis functions have a linear relationship with the target variables' covariance. This alignment ensures that the model captures the "shape" of the data dynamics rather than just fitting individual points. This is conceptually equivalent to the optimization scheme used in GPs, where the kernel function defines the relationship between points.

Experimental Evaluation

The authors evaluated the effectiveness of Covariance Loss across a diverse set of tasks:

  • Spatio-Temporal Traffic Forecasting: Using datasets like PeMSD7(M) (California highway velocity), METR-LA (Los Angeles traffic speed), and PeMS-BAY (Bay Area velocity).
  • Image and Graph Classification: Testing on MNIST, CIFAR-10, and graph-based citation networks like PubMed and Cora.
  • Synthetic Regression: A 1D line curve dataset to benchmark against original CNP implementations.

In the traffic forecasting domain, Covariance Loss was integrated into established models like Spatio-Temporal Graph Convolutional Networks (STGCN) and Graph WaveNet (GWNET).

Theoretical and Empirical Insights

A key insight from the paper is the "Zero Effect" Constraint. Through constraint analysis, the authors demonstrate that Covariance Loss encourages the network to minimize the "cross-terms" (the interactions between different basis functions).

Distribution of cross-term effectiveness on predictions, demonstrating how Covariance Loss (STGCN-Cov) encourages interactions to converge toward zero compared to the baseline (STGCN).

As shown in the figure above, the effectiveness of cross-terms in the STGCN-Cov model is tightly centered around zero. This results in a more stable and interpretable model where each basis function effectively learns a scaled version of the target dynamics, reducing the influence of spurious correlations and noise. Furthermore, this alignment makes the model significantly more robust to "noisy nodes." In graph-based traffic networks, if a single sensor fails, a standard model's error can propagate through the graph. Because Covariance Loss "remembers" the expected co-movement of nodes, it can effectively ignore the outlier and maintain prediction integrity.

Results and Performance

The empirical results demonstrate consistent improvements across all benchmarks:

  • Traffic Accuracy: On the PeMSD7(M) dataset, STGCN-Cov reduced the 15-minute Mean Absolute Error (MAE) from 2.25 to 2.20. More importantly, the model showed a superior ability to detect the onset of rush hour congestion earlier than baseline models.
  • Classification Boost: In image classification, simply replacing the standard loss with Covariance Loss improved CIFAR-10 accuracy from 88.73% to 91.37%.
  • Noise Resilience: In stress tests where noise was added to sensors, the performance of standard STGCN plummeted after 5 noisy nodes. The Covariance Loss version remained stable even with 15 noisy nodes, showcasing its utility in real-world, imperfect sensor environments.

Conclusion and Future Work

While Covariance Loss offers significant benefits, it is not without limitations. The authors noted that performance can degrade if there is a distribution mismatch—when the dependencies learned during training do not hold in the test set. Additionally, the hyperparameter $\lambda$, which controls the weight of the covariance term, requires careful tuning to balance point-wise accuracy with covariance matching.

Future research directions include exploring more sophisticated distance measures for the covariance matrix, such as the Wishart distance, which may offer better theoretical properties than the Frobenius norm at the cost of increased computational complexity. Nonetheless, Covariance Loss represents a powerful, architecture-agnostic tool for improving the robustness and accuracy of neural networks in complex regression tasks.

Summary updated

BibTeX
@inproceedings{yoo2021conditional,
  author = {Boseon Yoo and Jiwoo Lee and Janghoon Ju and Seijun Chung and Soyeon Kim and Jaesik Choi},
  title = {{Conditional Temporal Neural Processes with Covariance Loss}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure