The Mask Is Not the Object: Volumetric Supervision for 3D Gaussian Segmentation

Jeonghwan Cho, Minsu Kim, Junyoung Hong, Seon Joo KimarXiv 2026 · arXiv

TL;DR

Existing 3D Gaussian segmentation methods often produce accurate 2D masks while failing to capture the full 3D volume of an object, leaving behind "ghost" remnants when an object is removed. This paper introduces VODA, a plug-and-play supervision objective that uses local depth clustering to ensure all Gaussians belonging to an object—even occluded or low-contribution ones—receive strong supervision.

What Problem Does This Paper Address?

3D Gaussian Splatting (3DGS) has revolutionized real-time 3D reconstruction, but segmenting these scenes into distinct objects remains tricky. Most current methods rely on rendering-based supervision: they compare a rendered 2D feature map against a 2D target (like a SAM mask or CLIP features).

The authors identify a fundamental flaw in this approach: the "contribution-weighted gradient" problem. Because 3DGS uses alpha-blending to render pixels, the training gradients are weighted by each Gaussian's contribution to that pixel. Dominant, visible Gaussians get plenty of supervision, but occluded or low-opacity Gaussians receive almost none. Consequently, a model might learn to render a perfect 2D mask while only selecting a thin "shell" of the 3D object.

Figure 1: A plausible rendered mask does not imply complete 3D selection. For the query “sheep,” (a) the selected Gaussians render a plausible object mask, but (b) the depth change after removing the same Gaussian set indicates incomplete object removal in the reconstructed scene. This exposes a gap between rendered-mask quality and Gaussian-level selection completeness.

Methods

The authors propose VODA (Volumetric Object-level Direct Association) to bridge this gap. It consists of three main components:

  1. Depth-Aware Rasterizer: Instead of just looking at the final rendered pixel, this rasterizer groups all Gaussians contributing to a pixel into clusters based on depth gaps. The idea is that Gaussians belonging to the same object will naturally cluster together in depth.
  2. Target Cluster Selection: For each pixel, VODA identifies the "target cluster" with the highest total rendering contribution. This cluster is assumed to represent the primary object at that pixel.
  3. Direct Gaussian-Level Supervision: This is the "secret sauce." Instead of letting gradients trickle down through the alpha-blending chain rule, VODA creates pseudo-targets from the rendered features and applies them directly to every Gaussian in the target cluster. This ensures that low-contribution Gaussians receive the same supervision signal as the dominant ones.
Figure 2: Comparison of supervision and association strategies for per-pixel Gaussian contrib- utors. (a) Rendering-based feature learning delivers contribution-weighted gradients, giving weak supervision to low-contribution Gaussians. (b) Direct association links each pixel to a sparse subset of contributors selected by contribution-based rules. (c) VODA directly supervises every Gaussian in the selected depth cluster, including low-contribution Gaussians associated with the same object.

Additionally, a pixel-level push loss is used to penalize feature similarity between different object masks, helping to sharpen boundaries.

Datasets

The paper evaluates VODA on standard benchmarks and introduces a new one specifically for volumetric completeness:

  • LERF & ScanNet: Standard benchmarks for open-vocabulary and semantic segmentation.
  • Removal-3D (New): A synthetic indoor-scene benchmark. It provides paired ground-truth depth maps of scenes before and after an object is removed.
  • Removal Depth Error (RDE) (New Metric): A metric that quantifies how well a selection captures the 3D volume by comparing the depth change after removing predicted Gaussians vs. removing the ground-truth object.

Key Insights and Ideas

  • The Mask is Not the Object: A high mIoU on 2D masks is a deceptive metric for 3D editing. True volumetric completeness requires supervising the "dark matter" of the scene—the Gaussians that don't contribute much to the final image but are part of the object's structure.
  • Bypassing the Chain Rule: By detaching targets and applying them directly to primitives, VODA avoids the vanishing gradient problem inherent in alpha-blended rendering losses.
  • Depth as a Prior: Local depth structure is a powerful, "free" signal for object grouping that is often overlooked in feature-distillation methods.

Key Results

VODA was tested as a "plug-and-play" addition to four major baselines: OpenGaussian, LaGa, Gaussian Grouping, and ILGS.

  • Volumetric Accuracy: On the new Removal-3D benchmark, VODA consistently reduced the Removal Depth Error (RDE) across all baselines, often by significant margins (e.g., reducing RDE from 39.31 to 27.34 for OpenGaussian).
  • Standard Benchmarks: VODA maintained or improved performance on LERF and ScanNet, showing that better 3D selection also leads to cleaner 2D boundaries.
  • Qualitative Improvement: When objects are "removed" from the scene, VODA-trained models leave significantly fewer artifacts and better reveal the background surfaces.

Limitations and Future Work

  • Contacting Objects: Depth clustering can struggle when two different objects are physically touching or very close in depth, potentially merging them into one cluster.
  • Sparse Interiors: If an object has a very sparse interior (common in some 3DGS reconstructions), there may not be enough Gaussians to cluster effectively.
  • Evaluation Constraints: The RDE metric requires ground-truth "empty" scenes, which are currently only available in synthetic datasets like Removal-3D. Extending this rigorous volumetric evaluation to real-world data remains a challenge.

Summary updated

Abstract

3D Gaussian Splatting enables object-level operations on Gaussian primitives, yet existing segmentation methods are largely trained and evaluated through rendered views. This mismatch can produce accurate rendered masks from incomplete 3D selections since alpha-blended feature supervision concentrates learning on dominant contributors while leaving occluded or low-contribution Gaussians weakly supervised. We propose VODA, a plug-and-play supervision objective that exploits local depth structure to group per-pixel contributors and directly supervises all Gaussians in selected depth clusters, bypassing contribution-weighted gradient delivery. To evaluate volumetric completeness, we introduce Removal-3D, a synthetic object-removal benchmark, and the Removal Depth Error (RDE), a metric comparing predicted and ground-truth depth changes. Across four rendering-based baselines, VODA consistently improves volumetric segmentation on Removal-3D while maintaining or improving performance on LERF and ScanNet. Further analysis shows that VODA recovers lower-contribution Gaussians, while simple gradient reweighting and spatial propagation fail to reproduce comparable gains.
BibTeX
@misc{cho2026mask,
  author = {Jeonghwan Cho and Minsu Kim and Junyoung Hong and Seon Joo Kim},
  title = {{The Mask Is Not the Object: Volumetric Supervision for 3D Gaussian Segmentation}},
  year = {2026},
  eprint = {2611.21383},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21383},
}
Figure