DORec: Decomposed Object Reconstruction Utilizing 2D Self-Supervised Features

Jun Wu, Sicheng Li, Sihui Ji, Yue Wang, Rong Xiong, Yiyi Liao

TL;DR

DORec is a new framework that reconstructs high-quality 3D objects from multi-view images while automatically separating them from complex backgrounds—all without requiring any manual 2D labels. By leveraging self-supervised 2D features at two different levels of "granularity," the system achieves cleaner object decomposition and more accurate geometry than previous label-free methods.

What Problem Does This Paper Address?

In computer vision and robotics, we often need a 3D model of a specific object, not the entire room it's sitting in. While Neural Radiance Fields (NeRF) and similar implicit representations have revolutionized 3D reconstruction, they typically reconstruct the whole scene.

To "decompose" a single object from its background, researchers usually rely on one of two things:

  1. Costly manual annotations: Human-drawn masks for every training image.
  2. Category-specific priors: Models that only know how to find "cars" or "chairs."

Recent attempts have tried using self-supervised 2D features (like those from the DINO model) to avoid labels. However, these features are often "noisy" and inconsistent across camera angles. This leads to "floaters" in the 3D model or parts of the background being fused into the object's surface.

Methods

DORec (Decomposed Object Reconstruction) tackles this by using a compositional neural implicit network. It splits the world into two models: a foreground model (representing the object as a Signed Distance Field or SDF for sharp surfaces) and a background model (using a standard NeRF density field).

The core innovation is how the model is supervised using 2D features of varying granularity:

  • Coarse-grained Masks: The system generates rough binary masks to provide "early guidance," ensuring the training doesn't converge on a physically wrong solution.
  • Median-grained Masks (K-clusters): Instead of just "object vs. background," the authors use K-means clustering on DINO features to group pixels into semantically similar regions. These clusters are more stable across different views than raw pixel features.
Median-grained cluster masks generated from self-supervised 2D features.
  • Two-Stage Training: The model is first trained with coarse masks to establish a baseline decomposition. It is then fine-tuned using a cluster mask loss, which forces pixels belonging to the same semantic cluster to be classified consistently as either foreground or background.
  • Point-wise Max Composition: To merge the two models during rendering, they use a simple "max" operation on the densities. This encourages a hard "either-or" boundary, which helps in cleanly separating the object from the scene.

Datasets

The authors evaluated DORec on five diverse real-world datasets to prove its versatility:

  • BlendedMVS & DTU: Standard benchmarks for multi-view 3D reconstruction.
  • Tanks and Temples: Used to test performance in large, unbounded outdoor scenes.
  • OnePose & YCB Video: Challenging "cluttered" datasets featuring common household objects on various surfaces.

While the authors manually labeled test sets to calculate accuracy metrics, no manual labels were used during the training process.

Key Insights and Ideas

  • The "Median" Sweet Spot: Raw 2D features are too noisy for direct 3D supervision, while binary masks are too simplistic. "Median-grained" clusters provide enough semantic information to resolve ambiguities (like a brown box on a brown table) without being overwhelmed by pixel-level noise.
  • Robustness to View Inconsistency: By using a per-cluster loss rather than a per-pixel loss, DORec allows the network to "vote" on the classification of a region, making it much more resilient to occasional errors in the 2D features.
  • Implicit Surface Advantage: Using an SDF for the foreground allows for the extraction of clean, watertight 3D meshes ready for use in physics simulators or CAD software.

Key Results

DORec significantly outperformed existing state-of-the-art label-free methods:

DORec results showing high-quality foreground reconstructions from multi-view images.
  • Segmentation Accuracy: On the YCB Video dataset, DORec achieved a Mean Intersection-over-Union (mIoU) of 0.942, compared to 0.855 for the next best self-supervised method.
  • Reconstruction Quality: Quantitative metrics (Chamfer Distance) showed that DORec produces more accurate geometries than baselines like SNeRF+D, which often struggle with noisy boundaries.
  • Downstream Utility: The reconstructed models are clean enough to be used for zero-shot pose estimation. They were successfully plugged into FoundationPose to track objects in new scenes, proving the models are "robot-ready."

Limitations and Future Work

  • Training Time: It takes about 4 hours to reconstruct a single object on an NVIDIA 3090Ti. Integrating faster backbones like Instant-NGP is a logical next step.
  • Complex Topology: The system struggles when the background is "encased" by the foreground, such as the space inside a mug handle.
  • Scale: The current method is optimized for single foreground objects. Extending this to scenes with multiple interacting objects remains an open challenge.

Summary updated

BibTeX
@article{wu2025dorec,
  author = {Jun Wu and Sicheng Li and Sihui Ji and Yue Wang and Rong Xiong and Yiyi Liao},
  title = {{DORec: Decomposed Object Reconstruction Utilizing 2D Self-Supervised Features}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2025},
  url = {https://openreview.net/forum?id=HIXyU9fJVn},
}
Figure