Most 3D reconstruction methods assume the world holds still while the camera moves. When people, cars, or objects move through a capture, current methods respond in one of two ways. Distractor-removal methods detect the moving content from rendering errors and delete it. Dynamic-reconstruction methods model everything as moving and track it all. Neither uses what a point tracker can give them. A point tracker is a model that follows the same surface point across video frames. If such tracks are lifted to 3D, one can test each point directly for whether it stayed in the same 3D place over time. Points that stayed put belong to the static background. Points that moved belong to moving content, and their tracks already supply the start of a persistent 3D trajectory.
We propose to build a reconstruction that makes this test the organizing mechanism. Each 3D Gaussian (a small colored ellipsoid, the primitive of Gaussian Splatting) is created from a lifted track and given an explicit lifespan. It is labeled static or moving by a statistical test on its triangulated positions over time, not by how badly it renders. Moving particles are born when their tracks first appear, which removes the requirement that every object be present in the first frame. They are regularized only against neighbors whose trajectories agree with theirs. The claim is that a geometric motion test does two jobs that photometric residuals do poorly. It separates static from moving content in the conditions where color is unreliable: lighting changes, textureless walls, and stationary people. It also yields dense, persistent tracks for the moving content that distractor filters currently throw away.
Core Claim
Distractor filters treat moving content purely as noise. RobustSplat delays Gaussian cloning and builds masks from low-to-high-resolution rendering residuals. 3DGS-HPC classifies 16×16 image patches using a mixture model on color residuals plus DINOv2 feature distances. Both label content by rendering error, and both discard the motion once they have found it. The file records three places where this goes wrong. Illumination shifts fool residual-based decompositions (RobustSplat, DeGauss). Stationary people get deleted. Static objects seen in only a few views get misclassified as transient (3DGS-HPC).
Persistent trackers have the opposite blind spot. Dynamic 3D Gaussians treats every Gaussian as a moving particle with fixed color and opacity, regularized by local rigidity and long-term isometry (preserving distances between neighbors over time). It pays motion parameters for the whole background and cannot handle objects that appear after the first frame. TrackerSplat shows that initializing Gaussian motion from 2D tracks, triangulated across views, avoids the failure of gradient-only optimization under fast motion. It still applies those updates everywhere, without asking which points are static.
The new element here is the step between these two lines of work. Tracks are used first to decide static versus moving, and only then to drive motion.
For a candidate particle $i$ with triangulated positions $x_i(t)$ over its visible frames $T_i$, we compute a static score. It is the uncertainty-weighted residual of the best single fixed position:
$$ s_i = \min_{p}\ \sum_{t \in T_i} \big(x_i(t)-p\big)^\top \Sigma_i(t)^{-1}\big(x_i(t)-p\big), $$
Here $p$ is a candidate fixed 3D location and $\Sigma_i(t)$ is the triangulation covariance at time $t$, obtained from camera geometry and track confidence. Under the static hypothesis with Gaussian triangulation noise, $s_i$ follows a chi-square distribution with $3(|T_i|-1)$ degrees of freedom. The threshold is therefore a significance level, not a tuned residual percentile.
Two design choices follow from the score. Static particles collapse to a single position and use ordinary static Gaussian Splatting, with delayed growth borrowed from RobustSplat. Moving particles keep per-frame positions and rotations inside their lifespan. Their rigidity and isometry penalties apply only to neighbors whose trajectory residuals are mutually consistent. This softens the "artificial spring" failure that global isometry causes when objects separate.
Research Program
The system has three stages that are refined jointly.
The first stage extracts dense 2D tracks with DOT, as TrackerSplat does. It lifts these tracks to 3D by multi-view triangulation where several views overlap, and by rendered depth from the current static model otherwise. Each lifted track carries a covariance.
The second stage runs the static test and assigns each particle a lifespan: the interval during which its track is visible, extended by the photometric optimization when rendering evidence supports it. When a new cluster of moving tracks appears at a time $t>0$, particles are spawned at their triangulated positions at that moment. This is the mechanism that removes the first-frame constraint shared by Dynamic 3D Gaussians and Instruct-4DGS.
The third stage is photometric refinement. Moving particles follow TrackerSplat's approach of solving for motion from tracks before the gradient step. The static/moving label may be revisited during refinement, but only through the motion test on refined positions, never through rendering residuals alone.
Two analyses go with the method. The first measures when the geometric test fails. Long-baseline triangulation is weak at depth, so the test loses power far from the cameras, and moving cast shadows may be tracked as if they were surface texture. The second measures how much of the tracking gain comes from removing background parameters versus from track-based initialization.
Baselines
The naive reference points are vanilla static Gaussian Splatting, which shows the floaters and ghosting that motivate the problem, and Dynamic 3D Gaussians run with every Gaussian treated as moving. On PanopticSports, Dynamic 3D Gaussians reports a median 2D trajectory error of 1.57 pixels, an average 3D tracking error of 2.21 cm, and a novel-view PSNR of 28.7. These are the tracking numbers to beat on that dataset.
For background recovery, the strongest competitors are RobustSplat and 3DGS-HPC. They are purpose-built for this task and label content photometrically. 3DGS-HPC reports PSNR gains of 1.39 to 5.54 dB over native Gaussian Splatting on RobustNeRF. For RobustSplat's absolute PSNR there is no reported result found on Lacuna. On NeRF On-the-go there is no reported result found on Lacuna for RobustSplat, 3DGS-HPC, or DeGauss, so we will rerun all three with released code.
For decomposition, the strongest method is Split4D, which gives Gaussians velocities and identity features learned in streaming order. It reports an mIoU of 0.893 on Multi-Human, against 0.592 for SA4D. DeGauss is the second decomposition baseline; it pairs a static branch with a deformable foreground branch through a probabilistic mask.
The most dangerous baseline is a pipeline assembled from existing parts. DeGauss (or RobustSplat) masks separate the scene, and TrackerSplat or Dynamic 3D Gaussians then reconstructs only the masked foreground. Beating this pipeline is the real test of whether joint, track-based labeling matters.
Fairness conditions are the same for all methods. Every method gets the same input frames, camera poses, DOT tracks where applicable, a single RTX 4090-class GPU, and an equal wall-clock budget. Each baseline gets the same number of hyperparameter trials on a held-out sequence.
Two ablations isolate the key ingredient. The first keeps the full system but replaces the geometric static score with RobustSplat-style residual labels. The second disables birth at $t>0$.
Beating the baselines means three things. On background recovery, we must match or exceed the best distractor filter's PSNR on RobustNeRF and On-the-go. On tracking, we must beat Dynamic 3D Gaussians and the pipeline on 3D trajectory error in the late-entry and moving-camera protocols described below. Neither result may come at a loss of more than a small margin on the other.
Why not just…?
Why not just threshold ego-motion-compensated 2D flow to find moving pixels? On a fixed multi-camera rig, any pixel that moves in 2D is moving in 3D, so flow thresholding may match our test there, and we will report that honestly. The difference appears when cameras move. Then 2D displacement mixes camera motion and object motion, and separating the two requires exactly the 3D consistency test we propose. We include "flow threshold after compensation with estimated poses" as a baseline in the moving-camera protocol. If it matches the geometric test there too, the labeling contribution collapses, and the paper would reduce to the late-entry and scoped-rigidity results.
Why not just chain existing tools: mask with DeGauss or SAM, reconstruct the background statically, and track the foreground with Dynamic 3D Gaussians? The file offers two reasons to expect this to fall short. 3DGS-HPC documents that semantic masks miss shadows and moved furniture and delete stationary people. Split4D reports that lift-from-2D decomposition (SA4D, 0.592 mIoU) drifts badly in contact zones. Even with good masks, Dynamic 3D Gaussians still cannot ingest objects that enter late. This is a real threat on sequences where everything is present from the start, which is why the pipeline is our headline baseline rather than a straw man.
Why not just use Split4D, which already decomposes static and moving content natively? Split4D decomposes the scene, but it represents motion with linear velocities and propagated features. It optimizes those by photometric gradients, which TrackerSplat identifies as brittle under fast motion. It also does not report dense surface tracking error. We will measure 3D trajectory error for Split4D directly on PanopticSports to settle this.
Proposed Experiments
The first check is background recovery under lighting change and stationary distractors. It runs on RobustNeRF and NeRF On-the-go, reports PSNR, SSIM, and LPIPS against clean views, and includes the photometric-label ablation. The prediction is that the gap between our method and residual-based labeling widens on scenes with illumination shifts. If the gap is flat, the claim that geometric labeling is more robust is wrong.
The second check is tracking under sparse, moving views. It uses PanopticSports with its 3D track ground truth. At each timestep we render from a rotating subset of the 31 cameras, which turns a static rig into a sparse, moving capture. We report the 2D and 3D trajectory errors used by Dynamic 3D Gaussians and compare against it, TrackerSplat, the pipeline, and the flow-threshold baseline.
The third check is late entry, in a controlled setting. We composite animated D-NeRF and DG-Mesh objects, which have ground-truth meshes, into a static multi-view scene at times $t>0$. We measure Chamfer distance and trajectory error of the inserted object and the PSNR of the background. For reference, DynaSurfGS reports a Chamfer distance of 0.910 on DG-Mesh without any entry or background.
Decomposition quality is additionally reported as mIoU on Multi-Human against Split4D's 0.893. N3DV rendering PSNR is reported as a sanity check that the separation does not cost view synthesis quality.
The decisive outcome is simultaneous. Our method must equal the best distractor filter on the background and beat the mask-then-track pipeline on trajectory error in both the moving-camera and late-entry protocols. Anything less shows which half of the claim fails.
References
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- DeGauss: Dynamic-Static Decomposition with Gaussian Splatting for Distractor-free 3D Reconstruction
- Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis
- TrackerSplat: Exploiting Point Tracking for Fast and Robust Dynamic 3D Gaussians Reconstruction
- Efficient Dynamic Scene Editing via 4D Gaussian-based Static-Dynamic Separation
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- DynaSurfGS: Dynamic Surface Reconstruction with Planar-based Gaussian Splatting