Sequential Neural Processes

Gautam Singh, Jaesik Yoon, Youngsung Son, Sungjin Ahn

Neural Processes (NPs) combine the strengths of neural networks and Gaussian processes to achieve flexible, few-shot learning and fast prediction. However, they traditionally assume static data. When environments evolve over time—like a moving car or a bouncing ball—standard NPs treat each observation as an independent puzzle rather than a continuous sequence.

Presented at NeurIPS 2019, Sequential Neural Processes (SNP) bridges this gap by incorporating a temporal state-transition model. By merging the few-shot learning of NPs with the sequence-modeling power of State-Space Models (SSMs), SNP enables AI to model "stochastic processes of stochastic processes." This breakthrough paves the way for the first "4D" generative model capable of imagining how 3D scenes change over time, significantly outperforming static baselines in both accuracy and long-term consistency.

The Challenge of Dynamic Stochastic Processes

To understand SNP, it helps to look at its foundational components:

  1. Neural Processes (NPs): These offer uncertainty awareness and quick adaptation to new data (meta-learning) but lack an inherent understanding of time.
  2. State-Space Models (SSMs): These excel at modeling sequences and trajectories but struggle with high-dimensional, non-linear data like raw images or complex 3D scenes.

The gap lies in modeling temporally dependent structures while maintaining the ability to learn from just a few observations. If a robot looks at a room where a ball is rolling, a standard NP can represent the room but cannot "remember" the ball's previous location to predict its trajectory. SNP tackles this exact problem.

Core Innovations: SNP and TGQN

The authors propose two major architectural innovations to handle dynamic environments, alongside a novel training technique:

  • Sequential Neural Processes (SNP): Unlike a standard NP that generates a single latent variable $z$ for a task, SNP generates a sequence of latents $z_1, z_2, \dots, z_T$. Each $z_t$ depends on the previous state and the current context (new observations). This creates a "meta-transfer learning" effect, allowing the model to use past knowledge to update its belief about the current state.
  • Temporal Generative Query Networks (TGQN): The authors apply SNP to the Generative Query Network (GQN) framework. While the original GQN could render static 3D scenes from new angles, TGQN is a true "4D" model. It represents a 3D scene and predicts how that representation should change over time based on actions (e.g., "move left" or "rotate").
TGQN Roll-out
  • Posterior Dropout: A common failure in sequential latent models is "transition collapse," where the model ignores new observations and relies purely on its memory of the past. The authors introduce "posterior dropout"—randomly forcing the model to rely on its internal transition model during training—to ensure it learns to integrate new context effectively.

Key Insights and Generalization

By decoupling what the scene looks like (representation) from how it changes (dynamics), the model generalizes exceptionally well. It learns the "physics" of the world once and applies it to various room layouts.

One of the most impressive findings is TGQN's ability to handle unseen elements and generalize beyond its training horizon. For instance, in a "Color Cube" experiment, the model might not initially see the back of a rotating cube. SNP maintains a distribution of possible colors for that hidden face and "snaps" to the correct one the moment it becomes visible, without forgetting previously seen faces. Furthermore, if trained on sequences of length 10, TGQN can still produce plausible, sharp images at step 30, whereas non-sequential models quickly blur into noise.

Experimental Results

The model was evaluated across 1D regression tasks, 2D dynamic scenes, and new 3D dynamic scene benchmarks (Color Cube and Multi-Object environments, including "jittery" unpredictable versions).

Performance Metrics
  • Quantitative Superiority: In 3D scene prediction, TGQN achieved significantly lower Negative Log-Likelihood (NLL) and Pixel Mean Squared Error (MSE) than the standard GQN. For the 2D "Color Shapes" task, TGQN's NLL was roughly 10x better than the baseline.
  • Tracking vs. Prediction: SNP excels in both "tracking" (updating beliefs with constant new data) and "prediction" (imagining the future with no new data). In tracking tasks, SNP's accuracy actually improved over time as it accumulated more context, while the baseline NP's accuracy stagnated or worsened.

Limitations and Future Directions

Like many Neural Process variants, SNP can sometimes "under-fit" the context, failing to capture every minute detail of an observation. The authors suggest that integrating attention mechanisms (similar to Attentive Neural Processes) could mitigate this issue. Additionally, while SNP is highly capable, it still requires a fair amount of data to "meta-learn" the transition rules of a new environment.

Looking forward, the authors point toward using SNP as a "world model" for Reinforcement Learning (RL) agents. By accurately modeling dynamic stochastic processes, SNP could allow robots to plan actions in uncertain, changing environments much more effectively.

Summary updated

Abstract

Neural Processes combine the strengths of neural networks and Gaussian processes to achieve both flexible learning and fast prediction in stochastic processes. However, a large class of problems comprise underlying temporal dependency structures in a sequence of stochastic processes that Neural Processes (NP) do not explicitly consider. In this paper, we propose Sequential Neural Processes (SNP) which incorporates a temporal state-transition model of stochastic processes and thus extends its modeling capabilities to dynamic stochastic processes. In applying SNP to dynamic 3D scene modeling, we introduce the Temporal Generative Query Networks. To our knowledge, this is the first 4D model that can deal with the temporal dynamics of 3D scenes. In experiments, we evaluate the proposed methods in dynamic (non-stationary) regression and 4D scene inference and rendering.
BibTeX
@inproceedings{singh2019sequential,
  author = {Gautam Singh and Jaesik Yoon and Youngsung Son and Sungjin Ahn},
  title = {{Sequential Neural Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=gW4quV8ehqc},
}
Figure