Most foundational breakthroughs in 3D reconstruction and view synthesis operate under a strict mathematical contract: the world must stand perfectly still. From classic Structure-from-Motion (SfM) to modern Neural Radiance Fields (NeRF) introduced by Mildenhall et al., algorithms assume multi-view consistency. This assumption dictates that if multiple cameras observe the same 3D coordinate in space, they must be looking at the exact same physical entity with the same properties.
Early benchmark datasets like ScanNet, curated by Dai et al., accelerated the field by providing massive volumes of static indoor scans. However, the real world is inherently dynamic. When an algorithm encounters moving pedestrians, passing cars, or changing illumination, the static world assumption violates physics, causing optimization pipelines to fail catastrophically.
The Mechanics of Failure: Why Moving Objects Break 3D Models
To understand why dynamic scenes break traditional 3D reconstruction, we must look at how these models optimize their representations.
In a volumetric framework like NeRF, a neural network maps a 3D position and a 2D viewing direction to a specific color and density. If a pedestrian walks through a scene while a camera rig captures video, a single spatial coordinate will contain empty air at time $t_1$, but a pedestrian's jacket at time $t_2$. When the optimization pipeline tries to reconcile these conflicting views along a single ray, it encounters a contradiction. Because the model lacks a concept of time, it attempts to average these observations. The result is a severe degradation of image quality, characterized by cloudy, semi-transparent "ghosting" artifacts where the moving object used to be.
Traditional NeRF Ray Sampling:
Ray at t1: [ Empty Space ] ───> Photometric Loss expects Background Color
Ray at t2: [ Moving Object] ───> Photometric Loss expects Object Color
Result: Network compromises by creating semi-transparent "ghost" artifacts.
In explicit geometric representations like 3D Gaussian Splatting (3DGS), the failure mode is even more destructive due to a mechanism called adaptive densification. 3DGS algorithms periodically check where the reconstruction error is high and split or clone existing Gaussians to capture finer details.
When a transient object—such as a vehicle—moves across the frame, the model perceives a massive spike in photometric error at those pixels. As detailed by the authors of RobustSplat, the densification process mistakenly interprets this temporary error as a complex geometric structure that it missed. The algorithm aggressively spawns new 3D Gaussians to fit the transient object. Once the vehicle moves out of the frame, these Gaussians remain suspended in mid-air, permanently scarring the background reconstruction.
As Deming Li notes in MS-GS, attempting to reconstruct "in-the-wild" photo collections without accounting for these transient elements forces the model to overfit to temporary clutter, destroying the geometric integrity of the permanent scene.
Decoupling Space from Time
To reconstruct environments that change, researchers have shifted from purely 3D representations to 4D spatiotemporal modeling. The core intuition is to mathematically decouple the permanent, static background from the transient, dynamic foreground.
Continuous Spatiotemporal Tracking
One approach is to treat time as a continuous coordinate alongside space. For instance, Rempe et al. introduced CaSPR to learn canonical spatiotemporal point cloud representations. Instead of treating each frame as an isolated snapshot, CaSPR maps points to a normalized spatiotemporal space, allowing the model to track how an object's shape and position evolve continuously over time, even from sparse or irregularly sampled data.
Native 4D Decomposition
Historically, separating moving objects from the background required heavy preprocessing, such as running off-the-shelf 2D object detectors or video segmentation models to mask out dynamics. However, if the 2D segmentation model misses a frame or mislabels an object, those errors propagate directly into the 3D geometry.
To solve this bottleneck, Hu et al. developed Split4D. Instead of relying on external masks, Split4D decomposes a scene into static and dynamic components natively during optimization. By utilizing a streaming feature learning strategy, the framework identifies which elements exhibit temporal consistency (the background) and which elements vary across frames (the moving objects), achieving clean 4D decomposition without explicit video segmentation.
Decomposed 4D Pipeline (e.g., Split4D):
Multi-View Video ──> [ Streaming Feature Learning ] ──┬──> Static Gaussians (Background)
└──> Dynamic Gaussians (Time-varying)
Transient Filtering via Delayed Densification
Alternatively, if the goal is simply to recover a pristine model of a static environment (such as a clean 3D map of a city street filled with traffic), the model must actively ignore dynamic elements.
RobustSplat achieves this by altering the core 3DGS optimization loop. By intelligently delaying the densification process and employing a multi-resolution approach, the framework can distinguish between a permanent background detail (which remains consistent across multiple viewpoints over time) and a transient object (which appears suddenly and disappears). The model filters out the transient elements before they can trigger the creation of erroneous Gaussians, preserving a clean background.
The Downstream Impact on Scene Understanding
Overcoming the static scene assumption is not just a requirement for photorealistic view synthesis; it is a critical bottleneck for semantic scene understanding and robotics.
When we attempt to "lift" 2D semantic or panoptic labels into a 3D representation—as seen in frameworks like Panoptic NeRF developed by Fu et al. or Panoptic Lifting by Siddiqui et al.—the underlying architecture relies on multi-view label consistency. If a vehicle moves, its 2D semantic label shifts across pixels in subsequent frames, violating the geometric constraints of the 3D neural field and corrupting the semantic map.
To mitigate the noise introduced by these real-world inconsistencies, Zhu et al. introduced PCF-Lift, which represents lifted 3D features as probabilistic multivariate Gaussian distributions rather than deterministic vectors. While this probabilistic fusion allows the model to gracefully handle the noise and discrepancies inherent to 2D labels, the authors emphasize that fully adapting these lifting frameworks to handle the explicit temporal trajectories of moving agents remains an active and vital frontier in 3D computer vision.