Abstract
Reconstructing high-fidelity, 360° 3D surfaces of deforming objects from a single monocular RGB-D video is a significant challenge in computer vision due to the inherently under-determined nature of unobserved regions. We present MorpheuS, a novel framework that addresses this by distilling knowledge from a view-dependent diffusion prior into a dynamic neural surface representation. By combining motion-aware deformation fields with generative priors, MorpheuS accurately captures observed dynamics while realistically completing the geometry and appearance of regions never seen by the camera.
1. Introduction
Reconstructing dynamic, non-rigid objects—such as moving toys, humans, or pets—from a single camera is a classic ill-posed problem. While modern neural representation methods like Neural Radiance Fields (NeRF) can reconstruct observed surfaces with high fidelity, they struggle to "hallucinate" unobserved regions. This often results in blurry artifacts or "spurious surfaces" because these models lack an inherent understanding of 3D object priors.
Existing model-based approaches attempt to solve this using parametric templates (e.g., SMPL for humans), but these are limited to specific categories. MorpheuS proposes a model-agnostic solution capable of reconstructing arbitrary moving objects in full 360° by leveraging the generative power of large-scale diffusion models.
2. Methodology
MorpheuS integrates dynamic neural rendering with generative diffusion priors through three primary components:
2.1 Dynamic Neural Surface Rendering
The framework utilizes a hyper-dimensional Canonical Field to store the object’s base shape and color. To account for temporal changes, a Deformation Field network warps points from individual video frames into this canonical space. This allows the model to maintain a consistent representation of the object even as it moves and deforms throughout the sequence.
2.2 Diffusion Priors for Completion
To reconstruct the "dark side" of the object, MorpheuS leverages Zero-1-to-3, a diffusion model pre-trained to predict novel views of an object from a single image. The system employs Score Distillation Sampling (SDS) to distill this knowledge into the 3D representation. In this setup, the diffusion model acts as a critic, providing feedback on the plausibility of unobserved regions and guiding the 3D model toward realistic geometric and textural completions.
2.3 Temporal View-Dependent SDS
Standard SDS is typically optimized for static scenes. To handle the complexities of video, MorpheuS introduces a temporal strategy that selects specific keyframes as conditioning references. This ensures that the hallucinated portions of the geometry remain temporally consistent with the parts of the object that are actually observed as it moves through the camera's field of view.
3. Implementation Details
The authors identified that standard "blob" initializations, common in text-to-3D tasks, often cause deformation fields to diverge. Instead, MorpheuS adopts a geometric initialization (SAL) to provide a coherent starting point. Furthermore, to prevent the model from collapsing into "flat" geometry with painted-on textures, the framework applies regularization directly to slices of the hyper-dimensional canonical space, forcing the learning of true 3D volume.
4. Experimental Results
4.1 Datasets
The framework was evaluated on both real-world and synthetic data:
- Real-world: 9 scenes from KillingFusion, DeepDeform, and the iPhone dataset, featuring casual captures of toys and pets.
- Synthetic: 4 scenes from the AMA and BANMo datasets, providing ground-truth meshes for precise quantitative analysis.
4.2 Performance Metrics
MorpheuS demonstrates significant improvements over existing state-of-the-art methods:
- Geometric Accuracy: On real-world datasets, MorpheuS achieved an average reconstruction accuracy of 0.88 cm, compared to 1.92 cm for Neural Dynamic Reconstruction (NDR).
- Visual Realism: The model achieved higher CLIP similarity scores, indicating that its hallucinated views are significantly more convincing and photo-realistic than those produced by previous baseline methods.
5. Discussion and Limitations
While MorpheuS represents a leap forward in monocular 360° reconstruction, it faces certain challenges. It is susceptible to the "Janus Effect" (the multi-face problem) when camera views are extremely limited. Additionally, very rapid motion or significant self-occlusion can lead to temporal incoherence. Finally, the computational cost remains high, with training requiring 2–3 hours on an NVIDIA RTX 4090.
6. Conclusion
MorpheuS successfully bridges the gap between neural interpolation and generative extrapolation. By anchoring dynamic neural surfaces with diffusion-based priors, it enables the creation of complete, high-fidelity digital twins from simple monocular recordings. Future work will focus on integrating depth-conditioned diffusion priors to further improve robustness against complex motions and occlusions.
Summary updated