Conditional Temporal Neural Processes with Covariance Loss

Boseon Yoo, Jiwoo Lee, Janghoon Ju, Seijun Chung, Soyeon Kim, Jaesik Choi

Introduction

The authors introduce Covariance Loss, a novel regularization term that forces neural networks to learn the underlying dependencies between target variables, much like Gaussian Processes (GPs) do. By minimizing the difference between the covariance of the network's basis functions and the empirical covariance of the targets, models become significantly more robust to noise and better at capturing complex spatial-temporal patterns.

What Problem Does This Paper Address?

In standard regression tasks, neural networks are typically trained using Mean Squared Error (MSE), which focuses on predicting the "mean" or average value of a target. However, this approach often ignores the dependencies between target variables—how one output changes in relation to another.

Gaussian Processes (GPs) and Conditional Neural Processes (CNPs) are excellent at modeling these dependencies, but they can be computationally expensive or require specific, rigid architectures. The authors identify a gap: how can we bring the "dependency-awareness" of GPs into general neural network architectures without completely changing how we build and train them?

Methods

Covariance Loss Architecture

The core contribution is the Covariance Loss. It is designed as a plug-and-play regularization term that can be added to the standard MSE loss:

  1. Basis Function Mapping: The network is viewed as a feature extractor that produces a "basis function" $\Phi(X)$.
  2. Covariance Matching: The loss calculates the MSE between the covariance matrix of these basis functions and the empirical covariance matrix of the actual target variables ($YY^\top$).
  3. The Objective: The total loss becomes: $$\text{Total Loss} = \text{Prediction MSE} + \lambda \cdot \text{Covariance MSE}$$ where $\lambda$ is an importance factor.

By optimizing this, the network is forced to find a feature space where the relationships between features mirror the relationships between the outputs. While theoretically, a "Wishart distance" might be more mathematically rigorous for comparing covariance matrices, the authors chose MSE to keep computational complexity low and make the method more applicable to standard deep learning workflows.

Datasets

The authors evaluated their method across a diverse range of tasks:

  • Image Classification: MNIST and CIFAR-10.
  • Graph/Node Classification: PubMed and Cora.
  • Spatio-Temporal Traffic Forecasting: PeMSD7(M), METR-LA, and PEMS-BAY (large-scale highway traffic datasets).
  • Synthetic: A 1D line curve dataset used in original CNP papers to test interpolation and extrapolation.

Key Insights and Ideas

  • Structural Robustness: Because the model learns the relationship between variables (e.g., if traffic sensor A goes down, traffic sensor B usually goes up), it can make accurate predictions even when individual observations are noisy or missing.
  • The "Cross-Term" Constraint: The paper provides a fascinating mathematical analysis showing that Covariance Loss forces the "cross-terms" in the prediction to converge to zero. This effectively aligns the basis functions directly with the target variables. In their experiments, 87% of cross-terms in a "Covariance-optimized" model had zero effect, compared to only 33% in a standard model.
Effectiveness of cross-terms
  • Dynamic vs. Mean Activation: In 1D regression, standard CNPs often stick to the "mean" of the data. Models with Covariance Loss were shown to pursue the dynamics of the data more aggressively, leading to better curve fitting in complex scenarios.

Key Results

  • Improved Accuracy: In traffic forecasting (PeMSD7), the "STGCN-Cov" model outperformed the baseline STGCN across almost all metrics. For example, it reduced the 15-minute MAE from 2.25 to 2.20.
  • Noise Tolerance: The authors tested models by adding noise to an increasing number of sensors. While the baseline model's performance plummeted after 5 noisy nodes, the model with Covariance Loss remained stable up to 15 noisy nodes.
  • Classification Gains: Even in classification, where dependencies are often ignored, Covariance Loss provided a boost. On CIFAR-10, accuracy improved from 88.73% to 91.37%.
  • Early Detection: In traffic tasks, the proposed loss allowed models to detect the beginning and end of "rush hour" congestion significantly earlier and more accurately than baseline models.

Limitations and Future Work

  • Distribution Shifts: The authors noted one "negative example" where Covariance Loss made predictions worse. This happened when there was a significant difference in the data distribution between the training and testing sets. If the learned dependencies don't hold in the real world, the regularization can be counterproductive.
  • Computational Overhead: While more efficient than some GP alternatives, calculating the covariance matrix for every batch still adds a computational cost that scales with the batch size.
  • Hyperparameter Tuning: The importance factor $\lambda$ is a critical hyperparameter that requires careful tuning to balance mean accuracy with dependency modeling.

Future work could involve exploring more sophisticated ways to measure the distance between covariance matrices or applying this loss to even larger-scale generative models.

Summary updated

Abstract

We introduce a novel loss function, Covariance Loss, which is conceptually equivalent to conditional neural processes and has a form of regularization so that is applicable to many kinds of neural networks. With the proposed loss, mappings from input variables to target variables are highly affected by dependencies of target variables as well as mean activation and mean dependencies of input and target variables. This nature enables the resulting neural networks to become more robust to noisy observations and recapture missing dependencies from prior information. In order to show the validity of the proposed loss, we conduct extensive sets of experiments on real-world datasets with state-of-the-art models and discuss the benefits and drawbacks of the proposed Covariance Loss.
BibTeX
@inproceedings{yoo2021conditional,
  author = {Boseon Yoo and Jiwoo Lee and Janghoon Ju and Seijun Chung and Soyeon Kim and Jaesik Choi},
  title = {{Conditional Temporal Neural Processes with Covariance Loss}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure