Sequential Neural Processes

Gautam Singh, Jaesik Yoon, Youngsung Son, Sungjin Ahn

TL;DR

Sequential Neural Processes (SNP) extend Neural Processes by incorporating a temporal state-transition model, allowing them to effectively model dynamically changing stochastic processes. By applying this framework to 3D computer vision, the authors introduce the Temporal Generative Query Network (TGQN), the first 4D generative model capable of learning and rendering the temporal dynamics of 3D scenes from partial visual observations.

What Problem Does This Paper Address?

While deep neural networks excel at fast predictions, they are notoriously rigid: changing the target function usually requires a costly retraining phase. Gaussian Processes (GPs) offer a flexible alternative by directly inferring target processes from observations, but they scale poorly computationally. Neural Processes (NPs) elegantly bridge this gap, combining the speed of neural networks with the flexibility and uncertainty-awareness of GPs.

However, standard NPs treat observations as static or exchangeable, failing to account for situations where the underlying process evolves over time. For instance, if you are modeling a moving agent or a dynamic 3D scene where objects slide, rotate, or interact, a standard NP treats each time-step in isolation or lumps them together without temporal structure. This paper addresses this limitation by explicitly modeling the temporal dependencies in a sequence of stochastic processes.

Methods

The authors introduce Sequential Neural Processes (SNP), which merges NPs with State-Space Models (SSMs). Instead of modeling a single static latent variable, SNP utilizes a recurrent state-space structure where the latent state at the current time-step ($z_t$) depends on both the history of past latents ($z_{<t}$) and the current context observations ($C_t$).

Figure 1: Generative and infer- ence (shown as dashed edges) models in TGQN.

When scaling SNP to 4D scene modeling (3D space + time), they develop the Temporal Generative Query Network (TGQN). TGQN builds on the original Generative Query Network (GQN) but introduces a Temporal-ConvDRAW architecture. This allows the model to condition its spatial representations on a deterministic recurrent state ($ht$) updated via a Convolutional LSTM (ConvLSTM), capturing how the scene changes over time based on agent actions.

Overcoming Transition Collapse

A major technical hurdle introduced by SNP is transition collapse. Because the sequence of past latents ($z_{<t}$) contains rich historical information, the model's prior path often learns to ignore the current context ($C_t$). To force the model to attend to new observations, the authors propose a novel training objective called Posterior Dropout (PD). During training, a Bernoulli coin toss randomly determines whether a time-step will sample from the posterior or the prior transition, preventing the model from over-relying on history and neglecting immediate context.

Datasets

The model was evaluated across three distinct types of data:

  1. 1D Gaussian Process Regression: A synthetic dataset of evolving mathematical functions where kernel hyper-parameters gradually shift over time with added Gaussian noise.
  2. 2D Color Shapes Dataset: A custom canvas containing two moving shapes (triangles, squares, or circles) that follow specific occlusion rules and undergo random, rule-based color swaps.
  3. 3D Dynamic Environments: Generated using MuJoCo and OpenAI Gym, consisting of two setups:
  4. Color Cube: A cube with different colors on each face that translates and rotates based on discrete actions.
  5. Multi-Object: An arena containing a randomly colored sphere, cylinder, and cube. Both 3D environments featured deterministic and "jittery" (stochastic motion) variants.

Key Insights and Ideas

  • Decoupling Dynamics from Representation: By separating what the scene looks like from how it changes over time, TGQN can retain long-term knowledge (like the color of a hidden face of a cube) even when that object moves out of the current camera view.
  • Generalization Beyond Training Horizons: Because the temporal transitions are modeled autoregressively, the network can successfully predict states far beyond the sequence lengths it encountered during training.
  • Uncertainty as a Tool for Meta-Learning: When an object's state is unobserved, SNP samples from a broad prior distribution (capturing uncertainty). The moment a new context frame arrives, it instantly "re-synchronizes" its internal belief to match reality.

Key Results

SNP and TGQN consistently outperformed standard NP and GQN baselines across all tasks in both prediction (forecasting without context) and tracking (updating beliefs with partial context) regimes.

Figure 4: Comparison of generations of SNP with NP or GQN and comparison between SNP with and without posterior-dropout (PD). The latents are rolled-out from the prior conditioned on the context. For 1D regression, we report target NLL. For 2D and 3D settings, we report pixel MSE per image generated at each time-step.
  • 1D Regression: As time progressed, standard NPs degraded significantly in accuracy, whereas SNP successfully accumulated and retained historical knowledge to maintain low Negative Log-Likelihood (NLL).
  • 4D Scene Rendering: TGQN achieved massive drops in both NLL and pixel Mean Squared Error (MSE) compared to GQN. For instance, on the 2D Color Shapes tracking task, GQN's error worsened over time, while TGQN's error decreased as it observed more context.
Quantitative performance table
  • Qualitative Excellence: As shown below, TGQN maintains sharp, clear images and tracks object faces accurately over time, whereas GQN's predictions quickly blur out due to its inability to handle sequential transitions.
Figure 3: TGQN demonstration in Color-Cube Environment. Left: The contexts and actions provided in t < 5. Top Right: Scene maps showing the queried camera locations and the true cube and the wall colors. Bottom Right: TGQN predictions in 5 ≤t ≤10.

Limitations and Future Work

The authors note that like standard NPs, SNPs can still be susceptible to under-fitting the context data. They suggest that incorporating attention mechanisms (similar to Attentive Neural Processes) could help the model better target relevant context points. Additionally, finding ways to improve data efficiency—reducing the number of observations required to meta-learn new environments—remains an open challenge. Finally, deploying an SNP framework within reinforcement learning agents to help them adapt to changing environments is a highly promising future direction.

Summary updated

Abstract

Neural Processes combine the strengths of neural networks and Gaussian processes to achieve both flexible learning and fast prediction in stochastic processes. However, a large class of problems comprises underlying temporal dependency structures in a sequence of stochastic processes that Neural Processes (NP) do not explicitly consider. In this paper, we propose Sequential Neural Processes (SNP) which incorporates a temporal state-transition model of stochastic processes and thus extends its modeling capabilities to dynamic stochastic processes. In applying SNP to dynamic 3D scene modeling, we introduce the Temporal Generative Query Networks. To our knowledge, this is the first 4D model that can deal with the temporal dynamics of 3D scenes. In experiments, we evaluate the proposed methods in dynamic (non-stationary) regression and 4D scene inference and rendering.
BibTeX
@inproceedings{singh2019sequential,
  author = {Gautam Singh and Jaesik Yoon and Youngsung Son and Sungjin Ahn},
  title = {{Sequential Neural Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=gW4quV8ehqc},
}
Figure