Reconstructing the real world in 3D has undergone a massive paradigm shift. While Neural Radiance Fields (NeRFs) dominated the landscape by encoding scenes within the continuous weights of a multi-layer perceptron (MLP), they suffered from a fundamental bottleneck: rendering required casting millions of rays and evaluating the neural network hundreds of times per pixel, making real-time performance incredibly difficult to achieve.
The introduction of 3D Gaussian Splatting (3DGS) bypassed this bottleneck by using explicit, point-based primitives—millions of 3D ellipsoids parameterized by position, rotation, scale, opacity, and color. Because these primitives can be projected directly onto the 2D image plane and rasterized in parallel, 3DGS unlocked real-time rendering speeds.
However, the real world is rarely static. Extending explicit 3D primitives into the temporal dimension (4D) introduces a major challenge: how do we efficiently represent motion, deformation, and topological changes over time without sacrificing the real-time rendering benefits of the original static formulation? As noted in a comprehensive survey by Wu et al. (Recent advances in 3D Gaussian splatting), explicit geometric representations have proven uniquely effective at capturing complex animations compared to continuous neural volumes.
To bridge the gap between static 3D points and dynamic 4D worlds, researchers have converged on two primary paradigms: canonical deformation fields and natively spatiotemporal primitives.
Tracking Motion via Canonical Deformation Fields
The most intuitive way to animate a static scene is to decouple identity from motion. In a deformation-based architecture, the model maintains a fixed set of 3D Gaussians in a "canonical space." To render the scene at a specific timestamp $t$, a separate neural network or mathematical function acts as a coordinate warp, predicting spatial offsets ($\Delta x, \Delta R, \Delta S$) that transform the canonical primitives into their deformed states before they are passed to the rasterizer.
[Canonical Gaussians] ---> [Deformation Field (MLP/Bases)] ---> [Deformed Gaussians] ---> [Real-Time Rasterizer]
(Fixed Geometry) (Input: Time t) (Rendered Frame t)
This approach preserves structural consistency because the underlying primitives remain persistent across frames. For instance, Lin et al. (Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models) utilize a learned deformation field to predict temporal offsets for spatial and geometric attributes, allowing the model to maintain object identity and structural consistency while enabling fluid motion across time. Similarly, Liang et al. (GauFRe: Gaussian Deformation Fields for Real-Time Dynamic Novel View Synthesis) employ a forward-warping deformation field that transforms the entire point cloud once per frame, enabling highly efficient rasterization for dynamic rendering.
To capture highly complex or non-rigid movements—such as those found in surgical environments or volumetric capture—the parameterization of the deformation field becomes critical:
- Polynomial and Fourier Bases: Rather than relying on heavy MLPs, Chen et al. (SurgicalGS: Dynamic 3D Gaussian Splatting for Accurate Robotic-Assisted Surgical Scene Reconstruction) parameterize transformations using a combination of Fourier and polynomial basis functions to capture intricate, non-rigid tissue deformations.
- Trajectory Smoothness: Ma et al. (Detail Enhanced Gaussian Splatting for Large-Scale Volumetric Capture) use polynomial expansions to model the trajectory and property changes of primitives over time. This mathematical constraint ensures temporal smoothness in rotation, scale, and opacity, preventing the high-frequency visual jitter common in per-frame optimization.
Embedding Time Directly into 4D Primitives
While deformation fields work exceptionally well for smooth, continuous motions, they introduce an extra computational step per frame. An alternative paradigm eliminates the deformation network entirely by embedding time directly into the mathematical definition of the Gaussian primitive itself.
Instead of treating time as an external variable that warps space, the primitives are formulated as natively spatiotemporal objects. One way to do this is by assigning physical kinematic properties, such as velocity and angular velocity, to each Gaussian.
Hur et al. (UFO-4D: Unposed Feedforward 4D Reconstruction from Two Images) augment 3D Gaussian primitives with velocity vectors to jointly model geometry and motion within a single feedforward pass. This concept is expanded by Xu et al. (4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos), who integrate velocity, angular velocity, and finite lifespans into the primitives. By assigning infinite lifespans and zero velocity to stationary elements, their framework unifies moving objects and static backgrounds into a single, elegant representational format.
Alternatively, time can be treated as a fully continuous coordinate dimension integrated with spatial coordinates. Ma et al. (4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time) represent dynamic scenes using spatiotemporal Gaussian primitives that incorporate temporal mean and covariance parameters. In a similar vein, Kaixuan Zhang (Dynamic Novel View Synthesis in High Dynamic Range) demonstrates that letting Gaussian parameters evolve continuously within a four-dimensional space allows the model to represent complex motion without relying on discrete frame sequences.
Managing Lifecycles and Topological Changes
A major limitation of standard deformation fields is their assumption of topological permanence. If an object splits in two, disappears from the scene, or is permanently altered (such as tissue being cut during surgery), a continuous spatial warp cannot easily model the transition without creating severe "ghosting" artifacts.
To handle these transient effects, researchers introduce the concept of a primitive lifecycle, modulating a Gaussian's opacity as a function of time.
Opacity (α)
^
1.0| ___________
| / \
| / \ <--- Modulated by Radial Basis Functions or Sigmoids
0.0+----+---------------+----> Time (t)
Birth Death
Li et al. (Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis) control the lifespan of each primitive using a one-dimensional radial basis function (RBF). By modulating opacity based on the temporal coordinate, the model naturally represents transient scene elements like smoke, fire, or moving reflections that exist only for brief windows.
For irreversible topological changes, Shan et al. (Deformable Gaussian Splatting for Efficient and High-Fidelity Reconstruction of Surgical Scenes) assign a learnable temporal lifecycle to each primitive using Gaussian basis functions. This allows the system to activate or deactivate specific primitives over time, effectively simulating material removal and exposure without requiring complex spatial warping. To handle streaming data where objects continuously enter and leave the frame, Wu et al. (StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams) model visibility as a continuous function of time using sigmoid-based activations, allowing the system to handle dynamic occlusions and scene changes on the fly.
Why It Works: The Intuition of Explicit 4D Rasterization
The core reason 4D Gaussian Splatting succeeds where implicit volumetric methods struggle comes down to where the "heavy lifting" occurs in the pipeline.
In an implicit dynamic model, determining the color of a single pixel requires querying a neural network hundreds of times along a ray, and repeating this process for every frame.
In contrast, 4D Gaussian Splatting shifts the temporal computation to the very beginning of the pipeline. For any given timestamp $t$, the model evaluates a lightweight temporal equation—whether that means passing a timestamp through a small deformation network, calculating a linear offset via a velocity vector, or evaluating a 1D temporal RBF. Once this quick temporal adjustment is made, the primitives are frozen in their specific 3D states for that instant. From that point forward, the rendering engine utilizes the highly optimized, standard 3DGS tile-based rasterizer.
By decoupling the temporal modeling from the pixel-level rendering, 4D Gaussian Splatting achieves the best of both worlds: explicit, high-fidelity tracking of non-rigid motion and blazing fast, real-time frame rates.