Seg-Wild: Interactive Segmentation based on 3D Gaussian Splatting for Unconstrained Image Collections

Yongtang Bao, Chengjie Tang, Yuze Wang, Haojie Li

Abstract

Seg-Wild is a novel framework designed to segment 3D scenes reconstructed from unconstrained, "in-the-wild" photo collections. By integrating 3D Gaussian Splatting (3DGS) with a scale-adaptive Segment Anything Model (SAM) and a geometric "spiky" Gaussian pruning technique, Seg-Wild effectively handles transient occlusions and inconsistent lighting common in Internet datasets. The method achieves high-quality interactive 3D segmentation, significantly outperforming existing state-of-the-art approaches in both accuracy and visual coherence.

Introduction

Reconstructing 3D scenes from Internet photo collections—such as tourist photos of the Trevi Fountain—is a significant challenge in computer vision. These "in-the-wild" datasets are characterized by transient occlusions (e.g., tourists, vehicles), inconsistent lighting conditions, and varying viewpoints. Traditional 3D segmentation methods often struggle with these inconsistencies, leading to noisy results and "spiky" artifacts where elongated ellipsoids protrude from object boundaries.

Overview of Seg-Wild: (a) Unconstrained input, (b) Prompt-based segmentation, (c) Appearance tuning and novel view synthesis.

Methodology

Seg-Wild builds upon 3D Gaussian Splatting (3DGS), specifically leveraging the "Gaussians in the Wild" (GS-W) approach to decouple static intrinsic features from dynamic appearance features like lighting or weather. The framework introduces three core innovations to address the complexities of unconstrained environments.

3D Feature Field Construction

The framework lifts 2D features from SAM into 3D space, embedding them directly into each Gaussian. A compactness loss ($L_{Com}$) is employed to ensure that Gaussians belonging to the same object exhibit similar feature signatures, thereby creating well-defined boundaries.

Scale-Adaptive Segmentation Module (SASM)

Standard SAM implementations use fixed scales for prompt points, which can be suboptimal for varied 3D scenes. Seg-Wild dynamically adjusts SAM’s granularity by calculating average camera distance and Gaussian density. Additionally, a sky-masking step using DeepLabV3+ prevents infinite background depth from distorting the sampling distribution.

Spiky 3D Gaussian Cutter (SGC)

To eliminate the jagged edges typical of 3DGS, Seg-Wild introduces a geometric post-processing step. The SGC projects Gaussians to 2D and evaluates their coverage by the SAM mask. Gaussians with excessive "tails" protruding from the mask are trimmed or adjusted, ensuring smooth and coherent object edges.

The Seg-Wild pipeline, illustrating the integration of 3D Gaussians, SAM-based feature extraction, and the segmentation refinement process.

Evaluation and Results

The authors evaluated Seg-Wild on challenging datasets including Photo Tourism (PT) and NeRF-On-the-go. A new benchmark was also introduced, providing manually refined ground-truth masks for landmark scenes.

Seg-Wild demonstrates superior performance over previous state-of-the-art methods like SAGA and Feature 3DGS. For instance, on the Brandenburg Gate "Right Pavilion" task, Seg-Wild achieved an Intersection over Union (IoU) of 0.958, compared to 0.840 for SAGA. Qualitatively, the method produces cleaner boundaries and successfully ignores transient occluders that typically cause "floating" artifacts in other models. Furthermore, the decoupled representation allows for advanced applications such as appearance tuning and novel view synthesis.

Conclusion

Seg-Wild provides a robust solution for interactive 3D segmentation in unconstrained environments. While it remains dependent on the quality of the underlying 2D foundation model, its ability to leverage 3D density and depth information significantly improves consistency across viewpoints. Future work may explore tighter integration with advanced models like SAM 2 to further mitigate 2D mask errors using 3D geometric priors.

Summary updated

BibTeX
@inproceedings{bao2025seg,
  author = {Yongtang Bao and Chengjie Tang and Yuze Wang and Haojie Li},
  title = {{Seg-Wild: Interactive Segmentation based on 3D Gaussian Splatting for Unconstrained Image Collections}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3755567},
}
Figure