Spatiotemporal Modeling with 4D Gaussian Splatting

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2149 papers159 conceptsfrozen 2026-10-04

The emergence of 3D Gaussian Splatting (3DGS) revolutionized neural rendering by replacing the slow, implicit volumes of NeRFs with explicit, differentiable point-based primitives. However, the original formulation was static. To capture the world in motion—a field often called 4D reconstruction—researchers have extended these primitives into the temporal dimension.

The core challenge of 4D modeling is maintaining spatial-temporal consistency: ensuring that a point on a moving object remains the same point across time, rather than "flickering" into existence or changing its identity. 4D Gaussian Splatting solves this by treating time either as a transformation of space or as a fundamental coordinate of the geometry itself.

The Deformation Field: Animating the Canonical State

The most intuitive way to model motion is to separate "what an object looks like" from "how it moves." This is the approach taken by frameworks like DreamGaussian4D and Align Your Gaussians.

In this paradigm, the system maintains a canonical space—a static set of 3D Gaussians representing the object at a baseline pose. To render a specific timestamp, a neural deformation field (often a small MLP or a voxel grid) predicts spatial offsets, rotations, and scale changes for each Gaussian. Huan Ling and colleagues demonstrate that by decoupling static geometry from temporal evolution, we can animate complex objects while retaining the lightning-fast rasterization of the original 3DGS.

Yuyang Yin and the 4DGen team refined this by using multi-resolution voxel modules to encode these deformations. This structure provides a spatial prior that prevents the Gaussians from "drifting" chaotically, ensuring that the motion feels physically plausible across frames.

4D Primitives: Geometry in Spacetime

A more radical approach treats time not as a change in 3D space, but as a fourth dimension of the primitive itself. In Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting, Zeyu Yang and Li Zhang define Gaussians using a 4D mean and a 4D covariance matrix.

Imagine a 3D ellipsoid. Now imagine that ellipsoid "tilting" into the time dimension. A tilt in the $x-t$ plane effectively represents velocity; the Gaussian's position in 3D space changes as the time coordinate advances. By optimizing these 4D ellipsoids, the scene dynamics emerge naturally from the rendering loss without needing an explicit deformation network.

To handle appearance changes—like a car's specular reflection moving as it drives—these models often integrate temporal Fourier series or spherical harmonics that evolve with time. This allows a single primitive to change not just its position, but its color and opacity as the sequence progresses.

Velocity and Lifespan: Modeling Physical Flow

For applications like autonomous driving and robotics, motion isn't just a visual effect; it's a physical property. Researchers like Zhen Xu in 4DGT and Hao Li in VDG have moved toward assigning explicit physical attributes to Gaussians:

  • Learnable Velocity: Each Gaussian carries a vector $(v_x, v_y, v_z)$. Its position at time $t$ is simply $P_{base} + v \cdot t$.
  • Temporal Lifespan: Primitives are given a "visibility peak" and a duration. This allows the model to represent objects that enter and leave the frame, such as a pedestrian walking across a street.

This framing is particularly powerful for scene understanding. By identifying which Gaussians have zero velocity versus high velocity, models like DIAL-GS by Chensheng Peng can automatically segment moving cars from static backgrounds without any human labels.

From Optimization to Feed-Forward Generation

The first generation of 4DGS required minutes or hours of optimization per scene. The field is now shifting toward Large Reconstruction Models (LRMs) that can infer 4D Gaussians in a single forward pass.

Jiawei Ren and the L4GM team demonstrated that a Transformer trained on massive synthetic datasets can predict the entire spatiotemporal trajectory of a 3D object from a single video in under a second. Similarly, 4D-LRM by Ziqiao Ma uses spatiotemporal Gaussian primitives to enable "any view at any time" rendering, effectively turning a sparse set of images into a continuous 4D world model.

Why It Works: The Power of Explicit Correspondence

The reason 4D Gaussian Splatting has largely superseded NeRF-based dynamic modeling is explicit correspondence. In a volumetric NeRF, the "identity" of a point is lost in a continuous field of density. If an object moves, the network must learn to decrease density in one area and increase it in another.

In contrast, a 4D Gaussian is a discrete "particle" of the scene. When that particle moves, its identity, texture, and properties move with it. This makes the representation inherently more efficient for tracking (Shape of Motion), more robust for robotic manipulation (ManiGaussian), and significantly faster to render, often exceeding 100 FPS.

Papers in this version

Showing 60 of 149 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 110 papers
  2. Version 1 · 2026-10-04 · 149 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_spatiotemporal_gaussian_primitives_b782553a,
  title = {spatiotemporal Gaussian primitives},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatiotemporal-gaussian-primitives/rdr_b782553a02514d638d9d213a3426ad9e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatiotemporal_modeling_with_4d_gaussian_b782553a_v1,
  title = {Spatiotemporal Modeling with 4D Gaussian Splatting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatiotemporal-modeling-with-4d-gaussian-splatting/rdr_b782553a02514d638d9d213a3426ad9e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}