TL;DR
This paper introduces Lifting by Gaussians (LBG), a novel method for open-world 3D instance segmentation of 3D Gaussian Splatted Radiance Fields (3DGS). LBG achieves this by lifting 2D segmentation masks from foundation models directly onto 3DGS, enabling fast and efficient segmentation without per-scene training, and significantly outperforming existing methods in both speed and quality.
What Problem Does This Paper Address?
Semantic scene understanding – the ability to identify and delineate individual objects within a 3D scene – is crucial for applications in AR/VR, robotics, and autonomous systems. While 2D image segmentation has seen remarkable progress thanks to powerful foundation models like SAM, extending this capability to 3D has been challenging. The primary hurdles include the scarcity of annotated 3D data and the computational expense of existing 3D segmentation methods, which often require extensive per-scene training or complex inverse rendering techniques. This paper aims to bridge this gap by providing a fast, flexible, and training-free method for segmenting existing 3DGS reconstructions into objects, parts, and subparts.
Methods
LBG's core innovation lies in its "lifting" strategy. It takes a pre-trained 3DGS field and a sequence of posed 2D RGB images as input.
- 2D Mask and Feature Extraction: For each input image, LBG utilizes a 2D foundation model (like SAM) to generate class-agnostic segmentation masks for objects, parts, and subparts. Simultaneously, it extracts semantic features for these masked regions using models like CLIP and DINOv2.
- 2D-to-3D Lifting (Pixel-to-Gaussian Assignment): Instead of traditional methods that might assign object IDs to multiple Gaussians based on thresholds (leading to semantic bleeding), LBG employs a more precise pixel-to-Gaussian mapping. Each pixel in a 2D mask is associated with the Gaussian that has the maximum alpha-blending weight for that pixel. This ensures that the 2D mask's semantic information is directly and accurately transferred to the corresponding 3D Gaussians.
- Incremental Merging of 3D Object Fragments: Object fragments generated from each frame are incrementally merged into a coherent scene-level object map. This merging process uses a combination of geometric overlap (counting shared Gaussians) and semantic similarity (cosine similarity of feature vectors) to decide whether to merge a new fragment with an existing object or instantiate a new one. A running average is used to update the semantic features of merged objects.
- Hierarchical Decomposition: The lifting and merging process is applied hierarchically. It starts with object-level masks, then recursively breaks down these objects into parts and subparts, creating a scene graph.
- Post-Processing: An optional step includes statistical outlier removal and a split-and-merge operation to refine segmentations and clean up any remaining noise or under-segmented regions.
A key advantage is LBG's independence from gradient-based optimization, allowing it to work seamlessly with any 3DGS representation without modification.
Datasets
The paper evaluates LBG on two datasets:
- LERF dataset: Features "in-the-wild" scenarios captured with a standard iPhone camera. The authors re-annotated three scenes (figurines, ramen, teatime) with denser 2D instance labels.
- 3D-OVS dataset: Includes a collection of long-tail object categories.
For evaluation, they also introduce a novel protocol for assessing the quality of individually segmented 3D assets, which involves rendering multiple views of extracted objects and comparing them to ground truth using metrics like PSNR, SSIM, and LPIPS.
Key Insights and Ideas
- Direct 2D-to-3D Lifting is Sufficient: The paper demonstrates that a direct, training-free lifting of 2D segmentation masks and features onto 3D Gaussians, combined with a smart merging strategy, can achieve state-of-the-art results without complex inverse rendering or per-scene optimization.
- Max-Contributor Assignment Prevents Semantic Bleeding: The pixel-to-Gaussian mapping strategy is crucial for accurately transferring semantic information and avoiding the common issue of semantic bleeding seen in other methods.
- Hierarchical Segmentation for Richer Representations: Applying the segmentation process hierarchically (object, part, subpart) provides a more detailed and structured understanding of the 3D scene.
- 3D Asset Quality is the True Benchmark: The authors argue that evaluating 3D segmentation solely on 2D mask rendering is insufficient. Their proposed 3D asset extraction evaluation protocol offers a more robust measure of segmentation quality.
- Efficiency Gains are Substantial: LBG is an order of magnitude faster than existing methods, making it practical for real-time applications and large-scale 3D scene processing.
Key Results
- Speed: LBG is up to 10x faster than state-of-the-art methods, achieving segmentation in minutes rather than hours.
- 3D Asset Quality: LBG produces significantly cleaner and more complete 3D object segments compared to Gaussian Grouping and SAGA, as evidenced by qualitative comparisons (Figure 3) and quantitative metrics (Table 2, showing superior PSNR, SSIM, and lower LPIPS). For instance, on the figurines dataset, LBG achieves a PSNR of 28.689, SSIM of 0.934, and LPIPS of 0.065, outperforming baselines.
- 2D Mask Quality: Despite not being specifically optimized for it, LBG performs competitively on 2D novel view synthesis of instance masks, often providing more complete segmentations than SAGA (Table 3, Figure 4).
Limitations and Future Work
- Foundation Model Size: The reliance on large visual foundation models can hinder fast start-up times. Future work could explore model compression techniques.
- Small Object Segmentation: LBG occasionally struggles with segmenting very small objects, potentially missing them during the merging process. The authors suggest a refinement step to break up composite segments as a potential solution.
- Hierarchical Granularity: While LBG provides hierarchical segmentation, achieving the desired granularity for all cases might require further tuning or more sophisticated part/subpart detection.
Summary updated