Abstract
Neural Processes (NPs) are a family of conditional generative models that are able to model a distribution over functions, in a way that allows them to perform predictions at test time conditioned on a set of context points. While Convolutional Conditional Neural Processes (ConvCNPs) improved these models by introducing translation equivariance, they often struggle with generalization and lack the ability to produce diverse samples. We present GP-ConvCNP, a hybrid model that replaces the deterministic density estimate in ConvCNPs with a Gaussian Process (GP) posterior. This modification reintroduces the ability to sample diverse functions and significantly improves extrapolation and robustness to distribution shifts in time series data.
Introduction
The goal of Neural Processes (NPs) is to learn a "distribution over functions," enabling them to infer an entire underlying process from a limited set of context observations. Recent advancements like the ConvCNP utilized Convolutional Neural Networks (CNNs) to achieve translation equivariance—the property that a pattern's recognition is independent of its temporal location. However, ConvCNPs face significant challenges: they struggle to extrapolate patterns beyond the training range, are sensitive to distribution shifts, and lose the generative capability to draw diverse samples from the predicted distribution.
Methodology: The GP-ConvCNP Architecture
GP-ConvCNP addresses these limitations by integrating the mathematical rigor of Gaussian Processes into the ConvCNP framework.
Replacing the Density Estimator
In a standard ConvCNP, context points are mapped to a grid using a simple Nadaraya-Watson kernel density estimator. GP-ConvCNP replaces this deterministic mapping with a Gaussian Process (GP) posterior. By using a GP, the model provides a principled way to handle local logic and uncertainty at the context points.
Distributional Input and Implicit Smoothing
Instead of feeding a single point estimate to the CNN, GP-ConvCNP passes the GP’s mean and covariance. During training, the model employs Monte-Carlo integration, sampling from the GP to provide inputs to the CNN. This approach implicitly "smooths" the data, acting as a regularizer that prevents the neural network from overfitting to specific context points.
Maintaining Equivariance
The model preserves the translation equivariance of the ConvCNP by utilizing stationary kernels (such as the RBF/Squared Exponential kernel) within the GP. This ensures that the high-level global patterns learned by the CNN remain location-invariant.
Evaluation and Results
The model was evaluated across synthetic tasks, real-world time series, and image benchmarks.
Synthetic and Qualitative Performance
GP-ConvCNP demonstrates a unique ability to learn complex patterns. A key finding is its capacity to extrapolate periodic signals even when the underlying GP uses a non-periodic kernel. The CNN effectively learns the cyclical structure that the base kernel cannot capture.
Real-World Extrapolation and Robustness
On weather data (hourly temperature measurements), GP-ConvCNP significantly outperformed its predecessors in extrapolation tasks. When predicting future temperatures based on past context, it achieved a predictive log-likelihood (LL) of -1.190, compared to -1.261 for ConvCNP and -1.938 for Attentive NPs.
In distribution shift scenarios, such as the Hudson Bay lynx-hare predator-prey dataset, GP-ConvCNP proved remarkably robust. While standard ConvCNPs failed when tested on real-world data after being trained on simulations (LL of -23.540), GP-ConvCNP maintained a stable LL of -5.382.
Discussion and Future Work
GP-ConvCNP successfully bridges the gap between the flexibility of neural networks and the probabilistic foundations of GPs. By treating the GP as a "smart encoder," the model handles local uncertainty while the CNN captures global structure.
However, the inclusion of a GP introduces a computational complexity of $O(N^3)$, making GP-ConvCNP approximately 1.5x slower than the standard ConvCNP. Future research could mitigate this by incorporating approximate GP methods like inducing points or KISS-GP to scale to larger context sets. Additionally, exploring non-stationary or deep kernels within the GP framework could further enhance performance on non-smooth data like step functions.
Summary updated