Abstract
Neural Processes (NPs) are powerful tools for meta-learning static stochastic processes, but they struggle when the underlying environment evolves over time. This paper introduces Sequential Neural Processes (SNP) to model sequences of temporally changing stochastic processes. By extending this framework, we develop Temporal Generative Query Networks (TGQN)—the first 4D generative model capable of inferring, tracking, and rendering dynamic 3D scenes over time. Our evaluations across 1D regression and 3D scene generation tasks demonstrate that SNP and TGQN significantly outperform static baselines in both predictive accuracy and visual consistency.
Introduction
Neural Processes (NPs) combine the flexibility of Gaussian processes with the computational efficiency of neural networks, making them ideal for meta-learning stochastic processes from sparse context observations. However, real-world applications often involve dynamic environments where rules or states change—such as a camera observing moving objects in a 3D room.
Standard NPs and their 3D counterpart, Generative Query Networks (GQN), lack explicit temporal modeling. When an object moves, a standard GQN's learned mapping becomes obsolete. An ideal model should instead transfer past knowledge (e.g., object identity) while updating its belief about the current state (e.g., position) using new observations. This paper addresses this gap by marrying NPs with State-Space Models.
Methodology
Sequential Neural Processes (SNP)
Instead of a single latent variable, SNP models a sequence of latent variables $z_t$ over time. The latent state at any time step is conditioned on current context observations and the history of past latent states. This architecture enables meta-transfer learning: the model transfers general knowledge from the past while meta-learning the specifics of the current time step.
Temporal Generative Query Networks (TGQN)
TGQN applies the SNP framework to 3D scene modeling. It utilizes a recurrent architecture (ConvLSTM and a Temporal-ConvDRAW module) to update a 3D scene representation over time based on camera trajectories and object dynamics. This allows the model to maintain sharp, consistent object properties over long sequences, whereas standard GQNs often suffer from "forgetting" or blurring as they attempt to compress a sequence into a single vector.
Posterior Dropout
A key challenge identified is transition collapse, where the model's transition prior becomes so rich that it ignores new context observations. To mitigate this, the authors introduce Posterior Dropout. During training, the model is randomly forced to sample from its prior, incentivizing the network to actively integrate new context data to correct its trajectory.
Experiments and Results
The models were evaluated on several synthetic datasets: 1D GP Regression with evolving hyperparameters, 2D Color Shapes with stochastic rules, and 3D MuJoCo environments (Color Cube and Multi-Object) featuring both deterministic and jittery (stochastic) motions.
Key Findings
- Superior 1D Tracking: In 1D regression, SNP's accuracy improves as context accumulates over time. In contrast, standard NP accuracy deteriorates as it fails to reconcile conflicting past and present data.
- Enhanced 3D Scene Generation: TGQN vastly outperforms GQN in both Negative Log-Likelihood (NLL) and pixel Mean Squared Error (MSE). For instance, in the deterministic Color Cube task, TGQN achieved an NLL of 226 compared to GQN's 380.
- Long-Horizon Generalization: While trained on sequences of length $T=10$, TGQN successfully generated coherent predictions up to $T=30$, whereas GQN generations quickly degraded.
Conclusion
Sequential Neural Processes provide a robust framework for modeling dynamic stochastic processes. By decoupling dynamics from representation, TGQN maintains high-fidelity 3D scene consistency over time. While the model faces a trade-off between NLL and visual quality due to Posterior Dropout, it represents a significant step toward RL agents that can effectively operate in meta-learning settings within evolving environments.
Summary updated