One Vote Across Space and Time: Closed-Form Instance Lifting for Dynamic Gaussian Scenes

3D Gaussian splatting represents a scene as a cloud of small, colored, semi-transparent blobs that can be drawn to the screen very fast. People increasingly want these scenes to also know what each object is, so that a robot can query "the mug" or an editor can delete a person. The usual approach takes object masks from a 2D model such as the Segment Anything Model and copies them into the 3D blobs. This works on static scenes, but it breaks on scenes where things move. Current dynamic methods either depend on a 2D video tracker, which drifts during fast motion and occlusion, or they train a feature field for hours. Every mistake the tracker makes then becomes a permanent 3D defect.

This note proposes a different route. In an explicit dynamic Gaussian model, each blob keeps its identity for the whole video while it moves. Because of that, a 2D mask in any frame is really a vote about a set of blob indices, and those indices are the same at every timestamp. Two consequences follow. First, masks from different frames can be matched by how much their blob sets overlap, with no pixel tracker at all. Second, the label of every blob for all time can be solved in a single closed-form weighted vote pooled over every view and every timestamp. That makes lifting training-free and takes minutes rather than hours. It also lets a 2D error made in one frame be outvoted by all the others. The labeled model then renders a single integer label channel, at nearly the speed of its color rendering.

Core Claim

FlashSplat showed that for a static 3D Gaussian model with fixed geometry, rendering a mask is linear in the per-Gaussian labels. Label assignment therefore reduces to a weighted majority vote, solved in under 30 seconds with a 0.4 ms query time and 91.8% mIoU on NVOS. The research file lists extending this to deforming geometry as unexplored. Its concern is that re-solving per frame would destroy the latency benefit.

Our claim is that no per-frame re-solve is needed. Instance identity is time-invariant even though geometry is not. The linearity therefore holds jointly over all (view, time) pairs. Let $w_i(p,v,t)$ be the alpha-blending weight (opacity times accumulated transmittance) of Gaussian $i$ at pixel $p$ in view $v$ at time $t$, computed from the already-trained dynamic model. Let $g(p,v,t)$ be the group ID assigned to that pixel after association. The accumulated vote is

$$A_i^k=\sum_{v,t,p} w_i(p,v,t)\,\mathbb{1}[g(p,v,t)=k],$$

and the label is $\ell_i=\arg\max_k A_i^k$. A background bias $\gamma$ is added to the background score, as in FlashSplat, to reject weak foreground noise.

Computing every $A_i^k$ takes one rasterization pass over the training frames, about the cost of one training epoch. It needs no gradient steps. Pooling over time helps with 2D errors in a specific way. A mask that briefly bleeds onto the background during motion blur contributes a small, isolated share of the vote. It loses to the many frames in which the same Gaussian was labeled correctly.

Research Program

The base representation is Ex4DGS. It stores Gaussian attributes at keyframes and interpolates them with splines, with no neural deformation network. It reports 32.11 dB PSNR on N3V at 62.1 FPS. We chose it for two reasons: its Gaussians are persistent, explicitly moving entities, and its per-frame rendering weights are cheap to compute. For comparison we also run the method on canonical-plus-deformation 4D-GS, to test whether the idea depends on explicit motion.

The work has three components.

Tracker-free association. We run a class-agnostic 2D segmenter independently on every training frame. For each mask $m$, we compute its Gaussian footprint: the Gaussians whose rendering weight falls mostly inside $m$ in that frame. Two masks from different times refer to the same object if their footprints overlap strongly under a weighted Jaccard score. Moving the object does not change which Gaussians make it up. We cluster masks on this overlap graph. Following Gaga and Unified-Lift, we give large, stable segments priority. Gaga uses a 3D memory bank on static scenes and reaches 92.3% mIoU on LERF-Mask. Unified-Lift uses area-aware ID mapping. Neither handles motion. Our change is to put the memory in Gaussian-index space, so that it follows moving objects.

Solve and self-correct. We solve the vote above once. We then render the labels back into every training frame and score each 2D mask by how much it disagrees with its rendered group. We down-weight masks that disagree strongly and re-solve once. This gives the 2D-to-3D direction a limited path back: the 3D model can correct 2D beliefs, not only inherit them, which addresses the one-way-flow limitation noted in the file. A secondary diagnostic comes free. The per-Gaussian vote entropy across time flags Gaussians whose motion is wrong, because such Gaussians collect votes from different objects at different times.

Queryable, fast output. Following Segment then Splat, open-vocabulary retrieval attaches one pooled CLIP embedding per group rather than a per-Gaussian language field. That work pools one CLIP vector per object set and reaches 88.53 mIoU on 3DOVS. The renderer therefore adds only an integer channel, avoiding the bandwidth cost of high-dimensional feature rendering.

Baselines

The naive reference renders RGB from the trained dynamic model and runs the same 2D segmenter on every rendered novel-view frame. Its IDs come from the segmenter alone. It shows what lifting buys at all, both in cross-frame ID consistency and in per-query latency. We run it with the same segmenter and checkpoint as our method.

The strongest existing method on monocular dynamic segmentation is Segment then Splat. It tracks instances with SAM2, splits the point cloud into object sets, and optimizes each set separately. On HyperNeRF it reports 69.48 mIoU with 218 minutes of optimization, against 7.83 mIoU and 1,564 minutes for DGD. We will rerun it from released code on the same scenes and GPU, using the same 2D mask model where its pipeline allows. We will report its wall-clock time against ours, where ours includes Ex4DGS reconstruction; the file gives no reconstruction time for Ex4DGS, so we will measure it. Beating it means matching or exceeding 69.48 mIoU at a small fraction of its total time. Speed alone at a mIoU loss of more than a few points would count as a partial result, not a win.

On multi-entity decomposition, the strongest competitor is Split4D. It learns per-Gaussian velocity and features chronologically, with no tracker, and reports 0.893 mIoU on Multi-Human against 0.592 for tracker-based SA4D. We will run it with its published schedule. Beating it means comparable mIoU without its feature training, together with better separation in contact zones, a weakness it acknowledges.

Two ablations isolate the key ingredients. The first is per-frame FlashSplat: an independent closed-form solve at each timestamp, with IDs linked by our association. It isolates the value of pooling votes over time. It will also show whether temporal pooling costs anything when the motion is wrong. The second keeps our solver but takes group IDs from SAM2 tracking instead of Gaussian-index overlap. It isolates the association step from the solver. Finally, we remove the reweighting pass to measure how much the self-correction contributes.

Why not just…?

Why not just track masks with SAM2 and splat them, as Segment then Splat does? This is the real threat. On slow HyperNeRF captures the tracker may simply be good enough, in which case our gain is speed and training-free operation rather than accuracy. Split4D's gap over tracker-based SA4D on Multi-Human (0.893 vs 0.592) is evidence that trackers break under interaction. Gaga's sparse-view result (tracking falls to 10.00% mIoU against 29.67% for 3D-guided association) points the same way. The tracked-ID ablation above settles the question directly. We also add a stress split with subsampled frames, where fast apparent motion is exactly where drift should appear.

Why not just solve FlashSplat independently at every frame? Doing so costs one solve per timestamp. More importantly, each solve sees only that frame's few views, so a mask error in that frame goes uncorrected. The per-frame ablation measures both the extra time and the accuracy loss. If temporal pooling helps only on multi-view N3V-style captures and not on monocular video, we will report that honestly.

Why not just render RGB fast and segment the rendered frames? A 2D forward pass per frame undoes the speed of a renderer running at 1,000 FPS such as 4DGS-1K. The resulting IDs also switch across views and time. The naive baseline reports both effects: per-frame latency, and ID consistency measured as mIoU under a single global matching for the whole sequence.

Why not just remove the dynamics with RobustSplat and label the static remainder? RobustSplat delays densification and bootstraps transient masks so that moving content can be discarded. That throws away the people and tools we want to segment. It remains a useful comparison only for background classes.

Proposed Experiments

Accuracy is measured on HyperNeRF and on Multi-Human, using the numbers quoted above. Rendering fidelity and speed are measured on N3V, starting from Ex4DGS's 32.11 dB and 62.1 FPS. We report RGB FPS alongside label-channel FPS. For segmentation on Neu3D, no reported result was found on Lacuna; we will therefore annotate a small held-out set of frames.

Three checks are decisive. The first is whether tracker-free association with a single joint solve reaches Segment-then-Splat-level mIoU on HyperNeRF at a much lower total time. The second is whether vote entropy correlates with regions where the 4D geometry is visibly wrong; if it does, the vote doubles as a free diagnostic of motion errors. The third concerns pruning. We apply 4DGS-1K's spatio-temporal pruning after labeling and measure per-object recall by object size. We then test a simple guard that exempts Gaussians carrying a large share of a small group's vote mass. The aim is to see whether kilo-FPS rendering and small-object recall can coexist, a conflict the file identifies but no paper tests.

The main failure mode is monocular geometry error. Because every label is pooled through the Gaussians, bad deformation spreads labels across objects. We will stratify results by reconstruction PSNR to show where the method stops working.