Abstract
Seg-Wild is a novel framework designed to segment 3D scenes reconstructed from unconstrained, "in-the-wild" photo collections. By integrating 3D Gaussian Splatting (3DGS) with a scale-adaptive Segment Anything Model (SAM) and a geometric "spiky" Gaussian pruning technique, Seg-Wild effectively handles transient occlusions and inconsistent lighting common in Internet datasets. The method achieves high-quality interactive 3D segmentation, significantly outperforming existing state-of-the-art approaches in both accuracy and visual coherence.
Introduction
Reconstructing 3D scenes from Internet photo collections—such as tourist photos of the Trevi Fountain—is a significant challenge in computer vision. These "in-the-wild" datasets are characterized by transient occlusions (e.g., tourists, vehicles), inconsistent lighting conditions, and varying viewpoints. Traditional 3D segmentation methods often struggle with these inconsistencies, leading to noisy results and "spiky" artifacts where elongated ellipsoids protrude from object boundaries.
Methodology
Seg-Wild builds upon 3D Gaussian Splatting (3DGS), specifically leveraging the "Gaussians in the Wild" (GS-W) approach to decouple static intrinsic features from dynamic appearance features like lighting or weather. The framework introduces three core innovations to address the complexities of unconstrained environments.
3D Feature Field Construction
The framework lifts 2D features from SAM into 3D space, embedding them directly into each Gaussian. A compactness loss ($L_{Com}$) is employed to ensure that Gaussians belonging to the same object exhibit similar feature signatures, thereby creating well-defined boundaries.
Scale-Adaptive Segmentation Module (SASM)
Standard SAM implementations use fixed scales for prompt points, which can be suboptimal for varied 3D scenes. Seg-Wild dynamically adjusts SAM’s granularity by calculating average camera distance and Gaussian density. Additionally, a sky-masking step using DeepLabV3+ prevents infinite background depth from distorting the sampling distribution.
Spiky 3D Gaussian Cutter (SGC)
To eliminate the jagged edges typical of 3DGS, Seg-Wild introduces a geometric post-processing step. The SGC projects Gaussians to 2D and evaluates their coverage by the SAM mask. Gaussians with excessive "tails" protruding from the mask are trimmed or adjusted, ensuring smooth and coherent object edges.
Evaluation and Results
The authors evaluated Seg-Wild on challenging datasets including Photo Tourism (PT) and NeRF-On-the-go. A new benchmark was also introduced, providing manually refined ground-truth masks for landmark scenes.
Seg-Wild demonstrates superior performance over previous state-of-the-art methods like SAGA and Feature 3DGS. For instance, on the Brandenburg Gate "Right Pavilion" task, Seg-Wild achieved an Intersection over Union (IoU) of 0.958, compared to 0.840 for SAGA. Qualitatively, the method produces cleaner boundaries and successfully ignores transient occluders that typically cause "floating" artifacts in other models. Furthermore, the decoupled representation allows for advanced applications such as appearance tuning and novel view synthesis.
Conclusion
Seg-Wild provides a robust solution for interactive 3D segmentation in unconstrained environments. While it remains dependent on the quality of the underlying 2D foundation model, its ability to leverage 3D density and depth information significantly improves consistency across viewpoints. Future work may explore tighter integration with advanced models like SAM 2 to further mitigate 2D mask errors using 3D geometric priors.
Summary updated