Suppose you want every moving person, tool and car in a multi-camera video labelled in 3D, and you are willing to click once per object. Today you face a forced choice. Fast 3D segmentation methods work only on frozen, static scenes. Dynamic-scene methods need a full 2D mask for every frame. This proposal removes that choice.
The approach starts from a 3D Gaussian splatting reconstruction of the dynamic scene. Such a model describes a scene as a cloud of small coloured ellipsoids that are projected and blended onto the screen in real time. Two observations drive the idea. First, if each ellipsoid keeps its identity while it moves, then a pixel's rendered label at any time is still a weighted sum of fixed per-ellipsoid labels. Second, ellipsoids that move together almost always belong to the same object. So a few clicks can be spread across the whole space-time scene by solving one sparse linear system on a graph whose edges reward shared motion. This takes seconds, needs no video tracker, and rendering labels afterwards costs nothing beyond rendering colour.
Core Claim
FlashSplat showed that, with geometry frozen, rendering a 2D mask is linear in the per-Gaussian labels. That makes 3D segmentation a closed-form weighted vote: it finishes in under 30 seconds and assigns an object in 0.4 ms. The authors and the research file both treat this as a strictly static result.
We claim the static requirement is stronger than needed. In a deformation-based model such as 4D Gaussian Splatting, canonical Gaussians are warped by a learned deformation field. The blending weight $w_i(p,v,t)$ of Gaussian $i$ at pixel $p$, camera $v$ and time $t$ is fully determined once geometry is fit. Rendered labels therefore stay linear in time-invariant labels $f_i$. The vote can simply accumulate over all cameras and times.
Linearity alone does not help with clicks, however. A handful of clicked pixels touches only a few Gaussians. This is the sparse-to-dense gap. We close it with a second ingredient: each Gaussian's trajectory, $x_i(t)$ sampled over the sequence, defines an affinity. We then solve
$$ \big(L_{\text{traj}} + \lambda\, D_u\big)\, F = \lambda\, U Y, $$
where:
- $L_{\text{traj}}$ is the graph Laplacian of a k-nearest-neighbour graph whose edge weight $\exp(-\lVert x_i(\cdot)-x_j(\cdot)\rVert^2/\sigma^2)$ compares whole trajectories rather than positions at one instant;
- $U$ holds the click evidence, $u_{ic}=\sum_{(p,v,t)\in\text{clicks of class }c} w_i(p,v,t)$;
- $D_u$ is its row-sum diagonal;
- $Y$ maps clicks to classes;
- $\lambda$ trades click fidelity against smoothness.
This is the harmonic-function form of graph label propagation. It is solved by conjugate gradient, and its output is soft, so early mistakes do not harden into pseudo-labels.
Trajectory affinity matters because two objects can be spatially adjacent at every instant, such as a hand on a cup or a pedestrian against a wall, yet still separate by how they move. That separation is exactly what spatial-proximity propagation in static methods cannot provide.
Research Program
Reconstruction. The first stage fits an RGB-only 4D Gaussian model with persistent identities. We use the HexPlane-deformed canonical Gaussians of 4D-GS, which render at 82 FPS on D-NeRF scenes. We cache per-pixel blending weights for a subset of views and times, much as FlashSplat caches them for static views. Densification creates Gaussians that serve transient photometric error. RobustSplat identified this as the source of floaters. Such Gaussians tend to have erratic, short-support trajectories, so we down-weight edges by a trajectory-coherence score, and we test whether this alone keeps floaters from absorbing labels.
Propagation. The second stage builds the trajectory graph and solves the system above for whatever clicks are given, from one click per object in a single view and frame up to roughly ten. Because labels live on Gaussians, segmentation at any novel view and time is one extra rasterized channel.
Interaction loop. The third stage is a correction loop. A user adds a click where the rendered mask is wrong, and the system re-solves with the previous solution as a warm start. We will measure whether this stays at interactive latency.
Identity drift diagnostic. Our linearity argument rests on Gaussians tracking material. If the deformation field "reuses" a Gaussian for different surfaces over time, a single time-invariant label is wrong. We test this directly. Using ground-truth masks, we measure what fraction of Gaussians switch which instance they dominate across time. We report results both for the full method and for a variant that lets labels change slowly across time windows.
Baselines
Naive reference. Each click labels only the Gaussians it directly touches, and every other Gaussian takes the label of its nearest labelled neighbour in canonical space. This shows how much of the problem is solved by geometry alone.
Per-frame static voting. We run FlashSplat's static vote independently at each time step on that step's clicks. This isolates the benefit of pooling evidence across time.
Temporal vote ablation. The Gaussian-level vote is pooled over all views and times, but with no graph. This isolates linearity over time from propagation.
Spatial-graph ablation. The same linear solve is used, but on a graph built from instantaneous positions instead of trajectories. This isolates the key ingredient. The proposal's central claim stands or falls on the trajectory graph beating it, especially in contact zones.
Strongest existing method. Split4D trains Gaussians with velocities and identity features by streaming contrastive learning over dense per-frame masks. On the Multi-Human dataset it reports 0.893 mIoU, versus 0.592 for the tracking-based SA4D. Split4D cannot consume clicks, so we run it two ways:
- with masks produced from our same clicks by a promptable 2D segmenter plus a video tracker (the realistic sparse setting);
- with ground-truth dense masks, as an upper reference that uses far more annotation.
Fairness rules apply to all methods. Everyone uses the same reconstructed geometry where architecturally possible, the same click sets (simulated from ground-truth masks with fixed seeds), the same GPU, and an equal hyperparameter search budget on a held-out scene.
What counts as success. Beating Split4D means exceeding its click-derived variant in mIoU at equal clicks while using at least an order of magnitude less segmentation compute. Approaching its dense-mask 0.893 would be a strong result rather than a requirement.
Static sanity check. On NVOS we compare against FlashSplat's reported 91.8 mIoU. SAGA reports 90.9 in FlashSplat's table and 92.6 in its own paper. Note that these use masks, not clicks.
Why not just…?
Why not just turn clicks into dense masks with a 2D segmenter and video tracker, then run the temporal vote or Split4D? This is the real threat. Split4D's own comparison shows that tracking-reliant 4D segmentation degrades badly under occlusion and fast motion (0.592 vs 0.893 mIoU), and CGC reports that a large SAM takes about 5.1 s per mask. Still, a modern tracker might be good enough on easy scenes. We run exactly this pipeline at equal clicks and stratify results by occlusion and motion speed. If it matches us everywhere, the trajectory graph is not worth it, and we would report that. Tracker masks can also enter our solver as extra, down-weighted unary evidence, so the two approaches are complementary rather than exclusive.
Why not just learn contrastive identity features from the clicks by gradient descent, as Split4D or CGC do? Feature learning roughly doubles training cost; CGC reports +100% overhead. With only a few positive pixels per object, there is very little contrastive signal to learn from. Our click-fed Split4D variant measures this directly. We expect it to fall below the closed-form solve at one to three clicks and to cost minutes rather than seconds.
Why not just reconstruct a static scene with transients removed, as RobustSplat does? That method discards the moving objects, which are precisely what we want to label. It remains useful as the background prior that our coherence weighting borrows from.
Proposed Experiments
Main experiment. We use Multi-Human and Neu3D multi-view sequences, with clicks simulated from ground-truth instance masks at budgets of 1, 3 and 10 clicks per object. We report:
- mIoU over held-out views and times;
- boundary F-score restricted to contact regions, where Split4D notes identity bleed;
- solve time and memory, against FlashSplat's ~8 GB static footprint;
- label-render frame rate, which should match the underlying 4D-GS colour rendering.
Decisive checks. Three results determine whether the claims hold:
- the trajectory graph must beat the spatial graph in contact zones;
- the approach must beat click-plus-tracker on high-occlusion sequences;
- identity switches must stay rare enough that time-invariant labels remain valid. If switches are frequent, the windowed-label variant becomes the method.
Monocular stress test. HyperNeRF scenes probe the regime where 4D-GS's deformation is least reliable. No reported segmentation result was found on Lacuna for HyperNeRF or Neu3D, so these are new measurements rather than comparisons to published numbers.
References
- FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimally
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Segment Any 3D Gaussians
- Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation