TL;DR
Existing 3D Gaussian segmentation methods often produce accurate 2D masks while failing to capture the full 3D volume of an object, leaving behind "ghost" remnants when an object is removed. This paper introduces VODA, a plug-and-play supervision objective that uses local depth clustering to ensure all Gaussians belonging to an object—even occluded or low-contribution ones—receive strong supervision.
What Problem Does This Paper Address?
3D Gaussian Splatting (3DGS) has revolutionized real-time 3D reconstruction, but segmenting these scenes into distinct objects remains tricky. Most current methods rely on rendering-based supervision: they compare a rendered 2D feature map against a 2D target (like a SAM mask or CLIP features).
The authors identify a fundamental flaw in this approach: the "contribution-weighted gradient" problem. Because 3DGS uses alpha-blending to render pixels, the training gradients are weighted by each Gaussian's contribution to that pixel. Dominant, visible Gaussians get plenty of supervision, but occluded or low-opacity Gaussians receive almost none. Consequently, a model might learn to render a perfect 2D mask while only selecting a thin "shell" of the 3D object.
Methods
The authors propose VODA (Volumetric Object-level Direct Association) to bridge this gap. It consists of three main components:
- Depth-Aware Rasterizer: Instead of just looking at the final rendered pixel, this rasterizer groups all Gaussians contributing to a pixel into clusters based on depth gaps. The idea is that Gaussians belonging to the same object will naturally cluster together in depth.
- Target Cluster Selection: For each pixel, VODA identifies the "target cluster" with the highest total rendering contribution. This cluster is assumed to represent the primary object at that pixel.
- Direct Gaussian-Level Supervision: This is the "secret sauce." Instead of letting gradients trickle down through the alpha-blending chain rule, VODA creates pseudo-targets from the rendered features and applies them directly to every Gaussian in the target cluster. This ensures that low-contribution Gaussians receive the same supervision signal as the dominant ones.
Additionally, a pixel-level push loss is used to penalize feature similarity between different object masks, helping to sharpen boundaries.
Datasets
The paper evaluates VODA on standard benchmarks and introduces a new one specifically for volumetric completeness:
- LERF & ScanNet: Standard benchmarks for open-vocabulary and semantic segmentation.
- Removal-3D (New): A synthetic indoor-scene benchmark. It provides paired ground-truth depth maps of scenes before and after an object is removed.
- Removal Depth Error (RDE) (New Metric): A metric that quantifies how well a selection captures the 3D volume by comparing the depth change after removing predicted Gaussians vs. removing the ground-truth object.
Key Insights and Ideas
- The Mask is Not the Object: A high mIoU on 2D masks is a deceptive metric for 3D editing. True volumetric completeness requires supervising the "dark matter" of the scene—the Gaussians that don't contribute much to the final image but are part of the object's structure.
- Bypassing the Chain Rule: By detaching targets and applying them directly to primitives, VODA avoids the vanishing gradient problem inherent in alpha-blended rendering losses.
- Depth as a Prior: Local depth structure is a powerful, "free" signal for object grouping that is often overlooked in feature-distillation methods.
Key Results
VODA was tested as a "plug-and-play" addition to four major baselines: OpenGaussian, LaGa, Gaussian Grouping, and ILGS.
- Volumetric Accuracy: On the new Removal-3D benchmark, VODA consistently reduced the Removal Depth Error (RDE) across all baselines, often by significant margins (e.g., reducing RDE from 39.31 to 27.34 for OpenGaussian).
- Standard Benchmarks: VODA maintained or improved performance on LERF and ScanNet, showing that better 3D selection also leads to cleaner 2D boundaries.
- Qualitative Improvement: When objects are "removed" from the scene, VODA-trained models leave significantly fewer artifacts and better reveal the background surfaces.
Limitations and Future Work
- Contacting Objects: Depth clustering can struggle when two different objects are physically touching or very close in depth, potentially merging them into one cluster.
- Sparse Interiors: If an object has a very sparse interior (common in some 3DGS reconstructions), there may not be enough Gaussians to cluster effectively.
- Evaluation Constraints: The RDE metric requires ground-truth "empty" scenes, which are currently only available in synthetic datasets like Removal-3D. Extending this rigorous volumetric evaluation to real-world data remains a challenge.
Summary updated