GP-ConvCNP: Better generalization for conditional convolutional Neural Processes on time series data

Jens Petersen, Gregor Köhler, David Zimmerer, Fabian Isensee, Paul F. Jäger, Klaus H. Maier-HeinUAI 2021 · OpenReview

Abstract

Neural Processes (NPs) are a family of conditional generative models that are able to model a distribution over functions, in a way that allows them to perform predictions at test time conditioned on a set of context points. While Convolutional Conditional Neural Processes (ConvCNPs) improved these models by introducing translation equivariance, they often struggle with generalization and lack the ability to produce diverse samples. We present GP-ConvCNP, a hybrid model that replaces the deterministic density estimate in ConvCNPs with a Gaussian Process (GP) posterior. This modification reintroduces the ability to sample diverse functions and significantly improves extrapolation and robustness to distribution shifts in time series data.

Introduction

The goal of Neural Processes (NPs) is to learn a "distribution over functions," enabling them to infer an entire underlying process from a limited set of context observations. Recent advancements like the ConvCNP utilized Convolutional Neural Networks (CNNs) to achieve translation equivariance—the property that a pattern's recognition is independent of its temporal location. However, ConvCNPs face significant challenges: they struggle to extrapolate patterns beyond the training range, are sensitive to distribution shifts, and lose the generative capability to draw diverse samples from the predicted distribution.

Methodology: The GP-ConvCNP Architecture

GP-ConvCNP addresses these limitations by integrating the mathematical rigor of Gaussian Processes into the ConvCNP framework.

Replacing the Density Estimator

In a standard ConvCNP, context points are mapped to a grid using a simple Nadaraya-Watson kernel density estimator. GP-ConvCNP replaces this deterministic mapping with a Gaussian Process (GP) posterior. By using a GP, the model provides a principled way to handle local logic and uncertainty at the context points.

Distributional Input and Implicit Smoothing

Instead of feeding a single point estimate to the CNN, GP-ConvCNP passes the GP’s mean and covariance. During training, the model employs Monte-Carlo integration, sampling from the GP to provide inputs to the CNN. This approach implicitly "smooths" the data, acting as a regularizer that prevents the neural network from overfitting to specific context points.

Maintaining Equivariance

The model preserves the translation equivariance of the ConvCNP by utilizing stationary kernels (such as the RBF/Squared Exponential kernel) within the GP. This ensures that the high-level global patterns learned by the CNN remain location-invariant.

Evaluation and Results

The model was evaluated across synthetic tasks, real-world time series, and image benchmarks.

Synthetic and Qualitative Performance

GP-ConvCNP demonstrates a unique ability to learn complex patterns. A key finding is its capacity to extrapolate periodic signals even when the underlying GP uses a non-periodic kernel. The CNN effectively learns the cyclical structure that the base kernel cannot capture.

Qualitative comparison of NP variants across synthetic tasks including Matern kernels, periodic signals, and step functions.

Real-World Extrapolation and Robustness

On weather data (hourly temperature measurements), GP-ConvCNP significantly outperformed its predecessors in extrapolation tasks. When predicting future temperatures based on past context, it achieved a predictive log-likelihood (LL) of -1.190, compared to -1.261 for ConvCNP and -1.938 for Attentive NPs.

Extrapolation performance on real-world weather data, showing the model's ability to predict future temperature cycles from a 15-day context.

In distribution shift scenarios, such as the Hudson Bay lynx-hare predator-prey dataset, GP-ConvCNP proved remarkably robust. While standard ConvCNPs failed when tested on real-world data after being trained on simulations (LL of -23.540), GP-ConvCNP maintained a stable LL of -5.382.

Discussion and Future Work

GP-ConvCNP successfully bridges the gap between the flexibility of neural networks and the probabilistic foundations of GPs. By treating the GP as a "smart encoder," the model handles local uncertainty while the CNN captures global structure.

However, the inclusion of a GP introduces a computational complexity of $O(N^3)$, making GP-ConvCNP approximately 1.5x slower than the standard ConvCNP. Future research could mitigate this by incorporating approximate GP methods like inducing points or KISS-GP to scale to larger context sets. Additionally, exploring non-stationary or deep kernels within the GP framework could further enhance performance on non-smooth data like step functions.

Summary updated

Abstract

Neural Processes (NPs) are a family of conditional generative models that are able to model a distribution over functions, in a way that allows them to perform predictions at test time conditioned ...
BibTeX
@inproceedings{petersen2021gp,
  author = {Jens Petersen and Gregor Köhler and David Zimmerer and Fabian Isensee and Paul F. Jäger and Klaus H. Maier-Hein},
  title = {{GP-ConvCNP: Better generalization for conditional convolutional Neural Processes on time series data}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=E4N2kFuSZ4u},
}
Figure