WildSeg3D: Segment Any 3D Objects in the Wild from 2D Images

Yansong Guo, Jie Hu, Yansong Qu, Liujuan Cao

TL;DR

This paper introduces WildSeg3D, a novel feed-forward approach for intuitive 3D object segmentation directly from 2D images, eliminating the need for scene-specific training. It achieves this by introducing Dynamic Global Alignment (DGA) to improve multi-view pointmap alignment and Multi-view Group Mapping (MGM) for real-time interactive segmentation, demonstrating significant speedups over existing methods while maintaining high accuracy.

What Problem Does This Paper Address?

Current methods for 3D object segmentation from 2D images, often relying on powerful techniques like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), typically require extensive scene-specific training. This makes them impractical for real-time applications and limits their adaptability to new, unseen environments. A key challenge in feed-forward approaches is the accumulation of 3D alignment errors across multiple views, leading to inaccurate segmentation. This paper aims to bridge this gap by enabling accurate, scene-agnostic, and real-time 3D segmentation from arbitrary 2D images.

Methods

WildSeg3D operates in three stages:

  1. 2D Mask Pre-processing: This stage leverages the Segment Anything Model 2 (SAM2) to generate precise 2D object masks from multi-view images. These masks are stored offline in a "mask cache" to avoid real-time computation during segmentation. SAM2's video tracking capabilities ensure consistent masks across frames.
  2. 3D Point Dynamic Global Aligning (DGA): This is the core novel component for improving alignment accuracy. It addresses the issue of accumulated alignment errors caused by background clutter and difficult-to-match points. DGA dynamically adjusts attention on view-specific points during matching by introducing soft masks and confidence weighting. This focuses the alignment process on challenging sample points, leading to more accurate 3D reconstruction.
  3. Multi-view Group Mapping (MGM): This stage enables real-time intuitive segmentation. It retrieves pre-generated masks from the mask cache based on user prompts. The transform matrices learned by DGA are then used to map these multi-view masks into an aligned 3D space in real-time, allowing for rapid user interaction and segmentation.

Datasets

The paper evaluates WildSeg3D on several benchmark datasets:

  • NVOS dataset: Provides reference views with scribble annotations and target views with segmentation masks, captured from frontal perspectives.
  • SPIn-NeRF dataset: A 3D scene dataset annotated using NeRF-style data, used for assessing intuitive 3D segmentation quality in complex environments.
  • T&T dataset
  • Mip-NeRF360 dataset

Key Insights and Ideas

  • Scene-agnostic 3D Segmentation is Achievable: The paper demonstrates that it's possible to achieve high-quality 3D segmentation without scene-specific training, a significant step towards more generalizable AI systems.
  • Dynamic Alignment for Robustness: The DGA technique is crucial. By intelligently focusing on difficult-to-align points and dynamically adjusting confidence, it effectively mitigates the error accumulation that plagues simpler feed-forward alignment methods.
  • Offline Caching for Real-time Performance: Pre-generating and caching 2D masks significantly reduces the computational burden during inference, enabling the real-time interactive capabilities of MGM.
  • Leveraging 2D Foundation Models for 3D: The successful integration of SAM2 for robust 2D mask generation highlights the power of leveraging advanced 2D vision models to boost 3D understanding tasks.

Key Results

WildSeg3D achieves state-of-the-art (SOTA) accuracy on the NVOS dataset, matching existing methods while offering a remarkable 40x speedup. Specifically, it completes scene reconstruction in under 30 seconds, compared to SA3D's 780 seconds. Real-time interaction results are delivered in just 5-20 milliseconds. Ablation studies show that DGA significantly improves performance over traditional global alignment methods, and their proposed confidence adjustment function yields the best results.

Limitations and Future Work

  • Limitations: While the paper highlights impressive speedups, the "real-time" aspect is primarily for the interactive segmentation phase after initial pre-processing. The initial 2D mask pre-processing stage is performed offline. The paper also notes that the performance of DGA is evaluated by integrating it into existing DUSt3R and MASt3R models, suggesting that further improvements might be possible by end-to-end training the entire WildSeg3D pipeline.
  • Future Work: The authors suggest that their method's ability to generalize across diverse scenes and its efficiency make it suitable for practical applications, implying future work could focus on further refining the real-time performance and expanding its applicability to even more complex and dynamic environments.

Summary updated

Abstract

Recent advances in interactive 3D segmentation from 2D images have demonstrated impressive performance. However, current models typically require extensive scene-specific training to accurately reconstruct and segment objects, which limits their applicability in real-time scenarios. In this paper, we introduce WildSeg3D, an efficient approach that enables the segmentation of arbitrary 3D objects across diverse environments using a feed-forward mechanism. A key challenge of this feed-forward approach lies in the accumulation of 3D alignment errors across multiple 2D views, which can lead to inaccurate 3D segmentation results. To address this issue, we propose Dynamic Global Aligning (DGA), a technique that improves the accuracy of global multi-view alignment by focusing on difficult-to-match 3D points across images, using a dynamic adjustment function. Additionally, for real-time interactive segmentation, we introduce Multi-view Group Mapping (MGM), a method that utilizes an object mask cache to integrate multi-view segmentations and respond rapidly to user prompts. WildSeg3D demonstrates robust generalization across arbitrary scenes, thereby eliminating the need for scene-specific training. Specifically, WildSeg3D not only attains the accuracy of state-of-the-art (SOTA) methods but also achieves a $40\times$ speedup compared to existing SOTA models. Our code will be publicly available.
BibTeX
@inproceedings{guo2025wildseg,
  author = {Yansong Guo and Jie Hu and Yansong Qu and Liujuan Cao},
  title = {{WildSeg3D: Segment Any 3D Objects in the Wild from 2D Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure