Systems that let you type "the teapot's handle" and get back a 3D region usually work by copying the outlines drawn by 2D segmentation models onto a 3D scene representation. A common choice for that representation is 3D Gaussian Splatting (3D-GS), which builds a scene from many small, semi-transparent ellipsoids. The 2D outlines jitter from frame to frame, and the 3D model has no way to tell a real edge from a jittery one. Recent methods sharpen boundaries by splitting any ellipsoid that receives conflicting "inside" and "outside" signals. That treats every conflict as evidence of a real edge, so teacher noise produces extra ellipsoids and locks errors into the geometry. Our proposal is simple. Before splitting an ellipsoid, check whether the 2D boundary that caused the conflict stays put when you move it into neighbouring video frames using the scene's own depth and camera motion, and then move it back. Real edges survive this round trip. Jitter does not. We split only on boundaries that survive and stop supervising from those that fail.
The opportunity matters because boundary quality is now the weak point of open-vocabulary 3D segmentation. On LERF, the strongest boundary-oriented distillation method lifts boundary IoU from 55.45% to 67.66% while overall mIoU moves only from 76.07% to 82.07% (Rh-3DGS). Boundary-splitting methods, meanwhile, are known to inflate primitive counts and to fail when the 2D teacher is noisy (COB-GS). A filter that is cheap, needs no labels and works for any category would address both problems at once, provided the filter actually separates signal from noise. That is what we test.
Core Claim
We claim that in boundary-adaptive splitting, a large share of split decisions are triggered by teacher mask jitter rather than by true surface discontinuities. We further claim that a forward-backward consistency check between nearby video frames, computed through the reconstructed geometry, can tell the two apart. If so, gating splits on that check should give equal or better boundary IoU with substantially fewer primitives. It should also degrade much more gracefully as 2D mask noise increases.
The check is category-agnostic. It only asks whether a mask label changes at the same 3D location across views, never what the label means. It therefore works unchanged under open-vocabulary querying. In this proposal the closed-vocabulary constraint acts as a design requirement and an evaluation setting, not as the mechanism.
This differs from the closest prior work in specific ways. Segment then Splat uses video, but only as forward 2D tracking with SAM2. It partitions points into objects before optimization and never splits primitives or checks masks against 3D geometry. COB-GS splits on any label-gradient conflict and has no notion of whether the conflict is trustworthy. Rh-3DGS down-weights unreliable boundary pixels using single-view rasterizer statistics (accumulated opacity and depth moments), and it never modifies geometry. Our signal comes from agreement across time and views, and it is used both to gate geometric splitting and to mask supervision.
Research Program
The pipeline starts from a standard 3D-GS reconstruction of a video capture, with per-frame class-agnostic masks from SAM or SAM2.
For each frame $t$ and each mask-boundary pixel $p$, we render depth and use the known camera poses to warp $p$ into frame $t+k$ for small $k$. We record whether the mask identity there agrees, and then warp back. We define a reliability
$$r_t(p) = \mathbb{1}\big[\,\text{label agrees at } W_{t\to t+k}(p)\,\big]\cdot \exp\!\big(-\|W_{t+k\to t}(W_{t\to t+k}(p)) - p\|^2/\sigma^2\big),$$
where $W_{a\to b}$ is the depth-and-pose warp from frame $a$ to frame $b$, and $\sigma$ is a pixel tolerance. A depth test excludes pixels that become occluded. The return-to-start structure is borrowed from temporal cycle-consistency learning (Temporal Cycle-Consistency Learning), which used such loops to supervise feature learning. Here the loop instead audits a teacher.
Nearby frames matter because they are where SAM's granularity should be stable and occlusion is small. A disagreement between them is therefore diagnostic rather than expected. For static scenes this is a multi-view reprojection check restricted to short temporal baselines. For dynamic scenes (HyperNeRF), we replace the depth warp with dense point tracks. Point tracks are known to drift under fast rotation and occlusion (Densemarks), so we treat the dynamic case as a stress test rather than the main claim.
Reliability is used in two places. First, we recompute COB-GS's per-Gaussian mask-consistency strength using only gradient contributions weighted by $r_t(p)$. A Gaussian is split only if the conflict it receives from reliable pixels exceeds the threshold. Second, supervision from pixels with low $r_t(p)$ is dropped, so that unsplit Gaussians are not pulled toward jitter.
Language identity is attached afterwards, and that step is interchangeable. We test two versions. One pools a CLIP embedding per segment, as Segment then Splat does. The other uses per-Gaussian features averaged on the hypersphere, as Rh-3DGS does, with split children inheriting their parent's feature. This lets us show the boundary gain is independent of how language is attached.
We will also measure, for ungated COB-GS, what fraction of splits are driven mainly by low-reliability pixels. This is the descriptive finding that motivates the method, and it is publishable even if the method's gains are modest.
Baselines
The naive reference is plain 3D-GS with lifted masks and no boundary handling. It shows how much any boundary method buys.
The strongest geometric competitor is COB-GS, which splits ambiguous Gaussians and alternates mask and texture refinement. It reports 92.1% mIoU and 0.682 CLIP-IQA boundary clarity on NVOS, above FlashSplat's 91.8% and 0.626. It is the strongest competitor because our method is literally COB-GS plus a gate. If the gate does not help, nothing else in the proposal matters. We run it on identical reconstructions and identical masks with its released split threshold. We also run a threshold sweep, so that we can compare both systems at matched primitive counts.
The strongest open-vocabulary boundary competitors on LERF are two methods. ILGS adds identity embeddings, outlier filtering of tracker masks and progressive mask expansion; it reports 80.5 mIoU and 76.0 mBIoU. Rh-3DGS reports 82.07% mIoU and 67.66% mBIoU. The two may use different evaluation protocols, so we rerun both under one protocol before comparing. On 3D-OVS, ILGS reports 94.4 mIoU and Segment then Splat reports 88.53. On dynamic HyperNeRF, Segment then Splat reports 69.48 mIoU, and it is the baseline there.
No primitive counts are reported for any of these methods in the material available to us. We found no reported result on Lacuna for primitive count or memory under COB-GS splitting, so we measure it ourselves.
All methods share the same masks, the same CLIP model and the same training iterations. Each gets an equal hyperparameter-tuning budget on a held-out scene.
The key ablation replaces our temporal reliability with Rh-3DGS's single-view opacity and depth-moment gate inside the same splitting code. This isolates whether cross-frame verification is the ingredient that matters.
Beating the baselines means one of two outcomes. Either mBIoU is higher than COB-GS-plus-language at equal primitive count, or mBIoU is equal with at least a clearly smaller primitive count. In either case the mIoU must not fall below the Rh-3DGS and ILGS reruns.
Why not just…?
Why not just use SAM2's video memory, as Segment then Splat does? SAM2 enforces consistent object identity over time. It does not check that a boundary sits at a geometrically consistent 3D location, and Segment then Splat cannot split a primitive that straddles an interface after partitioning. ILGS needed explicit outlier filtering precisely because tracker pseudo-labels drift (ILGS). We will test this directly by feeding SAM2-tracked masks into both ungated and gated splitting. If gating adds nothing on SAM2 masks, the honest conclusion is that a good video teacher already removes the jitter we target.
Why not just raise COB-GS's split threshold? This is the simplest threat. Fewer splits also means fewer noise-driven splits. The matched-primitive-count sweep settles it. A global threshold suppresses real and spurious splits alike, so if gating is no better than the best threshold at equal primitive count, the gate is unnecessary and we will report that.
Why not just use Rh-3DGS's visibility-calibrated down-weighting? This is a real threat for mBIoU on static scenes. Opacity and depth statistics catch occlusion edges, which are a major source of bad supervision. They cannot detect a mask that is wrong in the interior of a visible surface, which is where SAM granularity flips occur (CAGS). The swap-in ablation decides between the two signals, and we also report their combination.
Why not just solve labels in closed form on frozen geometry, as FlashSplat does? FlashSplat is fast and its background bias cleans noisy edges. With frozen primitives, however, every straddling ellipsoid must go wholly to one side, and COB-GS already beats it on boundary clarity. We include it as a fast reference on NVOS only, since it is binary.
Proposed Experiments
The decisive check is a controlled noise-injection study on scenes with ground-truth masks, using Messy Rooms and Replica. We perturb teacher boundaries with increasing dilation and erosion jitter and with granularity flips. We then plot boundary IoU and primitive count against noise level for ungated splitting, threshold-raised splitting, the opacity-gated variant and our gated method. The prediction is specific. Ungated primitive count should grow with noise while ours stays roughly flat, and ours should lose less boundary IoU. If both curves move together, the core claim is false.
The second check is the diagnostic itself. On real LERF captures, we test whether low-reliability pixels coincide with ground-truth mask errors more often than high-reliability pixels do, reported as precision and recall of the reliability score as an error detector.
We then report LERF mIoU and mBIoU, 3D-OVS mIoU, NVOS mIoU and CLIP-IQA, primitive count, peak memory and rendering frame rate against the baselines above. A HyperNeRF run with track-based warping tests whether the idea survives motion. We expect that run to expose track drift, and will report it either way. Finally, a sensitivity sweep over the frame gap $k$ shows where the temporal baseline stops helping, because larger gaps bring occlusion and granularity changes that make disagreement less informative.
References
- Rh-3DGS: Robust Open-Vocabulary Scene Understanding via Riemannian Huber Distillation and Manifold-Aware Sampling
- COB-GS: Clear Object Boundaries in 3DGS Segmentation Based on Boundary-Adaptive Gaussian Splitting
- Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting
- Temporal Cycle-Consistency Learning
- Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks
- Identity-aware Language Gaussian Splatting for Open-vocabulary 3D Semantic Segmentation
- CAGS: Open-Vocabulary 3D Scene Understanding with Context-Aware Gaussian Splatting
- FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimally