Abstract
Monocular dense 3D reconstruction of deformable objects is a hard ill-posed problem in computer vision. Current techniques either require dense correspondences and rely on motion and deformation cues, or assume a highly accurate reconstruction (referred to as a template) of at least a single frame given in advance and operate in the manner of non-rigid tracking. Accurate computation of dense point tracks often requires multiple frames and might be computationally expensive. Availability of a template is a very strong prior which restricts system operation to a pre-defined environment and scenarios. In this work, we propose a new hybrid approach for monocular non-rigid reconstruction which we call Hybrid Deformation Model Network (HDM-Net). In our approach, deformation model is learned by a deep neural network, with a combination of domain-specific loss functions. We train the network with multiple states of a non-rigidly deforming structure with a known shape at rest. HDM-Net learns different reconstruction cues including texture-dependent surface deformations, shading and contours. We show generalisability of HDM-Net to states not presented in the training dataset, with unseen textures and under new illumination conditions. Experiments with noisy data and a comparison with other methods demonstrate robustness and accuracy of the proposed approach and suggest possible application scenarios of the new technique in interventional diagnostics and augmented reality.
Introduction
Monocular Non-Rigid 3D Reconstruction (MNR) is a fundamental challenge in computer vision, aiming to recover the 3D shape of a deforming object from a single 2D camera view. This problem is inherently ill-posed as a single 2D image can correspond to an infinite number of 3D configurations. Traditional approaches like Non-Rigid Structure from Motion (NRSfM) require tracking points over multiple frames, making them slow and sensitive to sudden motion. Conversely, Template-Based Reconstruction (TBR) requires a perfect 3D "rest" model (template) beforehand, which is often unavailable in real-world scenarios.
HDM-Net addresses this gap by replacing rigid templates with a learned deformation model. It achieves state-of-the-art accuracy with inference times of approximately 5ms, enabling real-time applications in fields like medical imaging and augmented reality.
Methods
The Hybrid Deformation Model Network (HDM-Net) utilizes an encoder-decoder architecture to regress 3D point clouds directly from single images. Unlike methods that output intermediate representations like depth maps, HDM-Net outputs a dense $73 \times 73$ grid of 3D coordinates.
The training process is guided by a combination of domain-specific loss functions:
- 3D Error Loss: A standard supervised loss comparing predicted coordinates to ground truth.
- Isometry Prior: A smoothing loss ensuring neighboring points maintain consistent distances, mimicking the physical resistance to stretching in real-world materials.
- Differentiable Contour Loss: A novel contribution that projects the predicted 3D shape back to 2D to compare its silhouette with the input image. This is achieved through a differentiable pipeline using bilinear interpolation and soft thresholding.
Evaluation and Results
The model was trained on a synthetic dataset generated in Blender, containing over 4,600 unique deformation states under various lighting and texture conditions. Evaluation focused on generalization to unseen textures and illumination settings.
HDM-Net demonstrates significant improvements in both speed and accuracy. It operates at ~200 FPS (5ms/frame), whereas traditional template-based methods like Yu et al. run at ~0.3 FPS. In terms of accuracy, HDM-Net achieved a 3D error ($e_{3D}$) of 0.0251 on endoscopically textured surfaces, vastly outperforming baselines like AMP (1.6189). Furthermore, the model proved robust to noise and generalized well to real-world imagery, such as surgical footage and musical notes, despite being trained solely on synthetic data.
Conclusion
HDM-Net provides a robust and efficient solution for monocular non-rigid reconstruction by learning a class-specific deformation model. While the model is currently limited by the variety of deformations present in its training data, its speed and accuracy make it a promising tool for clinical diagnostics and AR. Future work will explore the integration of GANs and RNNs to improve temporal consistency and handle more complex biological tissue deformations.
Summary updated