DreamUp3D: Object-Centric Generative Models for Single-View 3D Scene Understanding and Real-to-Sim Transfer

Yizhe Wu, Haitz Sáez de Ocáriz Borde, Jack Collins, Oiwi Parker Jones, Ingmar Posner

Abstract

Effective robotic manipulation in unstructured environments requires a robust understanding of 3D scenes from limited perspectives. Current approaches, such as Neural Radiance Fields (NeRFs), often struggle with real-time constraints, while standard Object-Centric Generative Models (OCGMs) frequently fail under occlusion. DreamUp3D introduces a self-supervised, object-centric generative framework that performs real-time 3D scene understanding from a single RGB-D image. By integrating generative radiance fields (GRAFs) with a novel shape-completion module, the system reconstructs occluded geometries and estimates 6D poses with high accuracy, enabling efficient real-to-sim transfer for robotic tasks.

Introduction

Robots operating in real-world settings typically encounter scenes characterized by partial observability. When viewing a scene from a single perspective, significant portions of objects are hidden from the sensor. To interact with these objects, a robot must move beyond simple point-cloud representations and develop a conceptual understanding of the environment as a collection of distinct, three-dimensional entities with specific orientations and volumes.

Existing methodologies face a trade-off between fidelity and speed. NeRF-based systems produce high-quality reconstructions but require extensive multi-view data and prohibitive per-scene optimization times. Conversely, traditional OCGMs often conflate an object's appearance with its spatial pose and struggle to maintain accuracy when objects are partially blocked. DreamUp3D addresses these limitations by providing a fast, single-view system capable of "imagining" missing geometries to facilitate precise manipulation.

Methodology

DreamUp3D utilizes a multi-stage pipeline designed for both accuracy and inference speed:

1. Feature Extraction and Segmentation

The process begins with a single RGB-D image converted into a point cloud. A U-Net backbone utilizing Kernel Point Convolution (KPConv) layers extracts local and global features while preserving critical spatial relationships. To decompose the scene into individual objects without human-annotated labels, the model employs the Instance Colouring Stick-Breaking Process (IC-SBP) algorithm, which clusters the point cloud into distinct object masks.

2. Shape Completion and Pose Estimation

The core innovation of DreamUp3D is its Shape Completion Module. By learning a latent space of object shapes, the model predicts the full 3D geometry of an object based on its visible segments. Once the shape is completed, the system calculates a 6D pose by fitting a minimum volume bounding box to the predicted geometry. This decoupling of appearance and pose is essential for robotic grippers that require precise orientation data.

3. Generative Radiance Fields (GRAF) and Distillation

The model uses a tri-plane based GRAF to decode latent embeddings into high-fidelity 3D shapes and colors. To ensure the system is fast enough for closed-loop control, the authors implement a self-distillation mechanism. The complex GRAF predictions serve as a training signal for the lighter shape-completion module, significantly reducing the computational overhead of ray marching during inference.

Experimental Setup

The model was evaluated using a custom dataset collected with a 7-DoF Franka Panda robot and an Intel RealSense camera. The dataset featured a subset of the YCB (Yale-CMU-Berkeley) object set, including common items like mustard bottles and cracker boxes. Training was conducted on 200 scenes (captured from four views, though inference remains single-view), and testing was performed on 12 scenes with randomized, unseen configurations.

DreamUp3D was benchmarked against several baselines, including ObSuRF and ObPose, as well as indirect comparisons to MultiShapeNet and CLEVR benchmarks.

Results and Discussion

Performance and Speed

DreamUp3D demonstrated a significant advantage in computational efficiency, achieving an inference speed of approximately 0.24 seconds. In comparison, NeRF-based baselines required over 127 seconds to reconstruct the same scene, rendering them impractical for real-time robotics.

Reconstruction and Pose Accuracy

In terms of reconstruction quality, DreamUp3D achieved a Directed Hausdorff Distance (DHD) of 0.0075, significantly outperforming both NeRF baselines (0.0351) and the previous state-of-the-art OCGM, ObSuRF (0.0130). Furthermore, the shape completion module allowed the system to maintain high pose accuracy (mIoU) even when up to 70% of an object's point cloud was missing or obscured by noise.

Object Matching

The model's ability to learn robust latent representations was tested through an object-matching task across different scenes. DreamUp3D achieved 57.6% accuracy, more than doubling the performance of a CLIP-based visual-semantic baseline (27.1%).

Limitations and Future Work

While DreamUp3D represents a significant step forward, several challenges remain:

  • Sensor Sensitivity: Performance is currently affected by depth sensor noise, particularly on reflective surfaces like metal or plastic.
  • Clutter Scaling: The current study focused on tabletop environments with 2–4 objects; scaling to highly cluttered scenes remains an open research question.
  • Background Complexity: The system relies on RANSAC for planar background identification, which may struggle with more complex, non-planar environments.
  • Generalization: Reconstructing entirely novel object categories from a single view remains an ill-posed problem that requires further exploration into broader generative priors.

Conclusion

DreamUp3D demonstrates that by combining generative radiance fields with dedicated shape-completion modules, robots can effectively overcome the challenges of partial observability. The system's ability to perform high-speed, accurate 3D reconstruction and pose estimation from a single view provides a viable path for real-to-sim transfer and autonomous manipulation in dynamic real-world environments.

Summary updated

BibTeX
@article{wu2024dreamup,
  author = {Yizhe Wu and Haitz Sáez de Ocáriz Borde and Jack Collins and Oiwi Parker Jones and Ingmar Posner},
  title = {{DreamUp3D: Object-Centric Generative Models for Single-View 3D Scene Understanding and Real-to-Sim Transfer}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  doi = {10.1109/lra.2024.3362678},
}
Figure