HDM-Net: Monocular Non-Rigid 3D Reconstruction with Learned Deformation Model

Vladislav Golyanik, Soshi Shimada, Kiran Varanasi, Didier Stricker

Abstract

Monocular dense 3D reconstruction of deformable objects is a hard ill-posed problem in computer vision. Current techniques either require dense correspondences and rely on motion and deformation cues, or assume a highly accurate reconstruction (referred to as a template) of at least a single frame given in advance and operate in the manner of non-rigid tracking. Accurate computation of dense point tracks often requires multiple frames and might be computationally expensive. Availability of a template is a very strong prior which restricts system operation to a pre-defined environment and scenarios. In this work, we propose a new hybrid approach for monocular non-rigid reconstruction which we call Hybrid Deformation Model Network (HDM-Net). In our approach, deformation model is learned by a deep neural network, with a combination of domain-specific loss functions. We train the network with multiple states of a non-rigidly deforming structure with a known shape at rest. HDM-Net learns different reconstruction cues including texture-dependent surface deformations, shading and contours. We show generalisability of HDM-Net to states not presented in the training dataset, with unseen textures and under new illumination conditions. Experiments with noisy data and a comparison with other methods demonstrate robustness and accuracy of the proposed approach and suggest possible application scenarios of the new technique in interventional diagnostics and augmented reality.

Introduction

Monocular Non-Rigid 3D Reconstruction (MNR) is a fundamental challenge in computer vision, aiming to recover the 3D shape of a deforming object from a single 2D camera view. This problem is inherently ill-posed as a single 2D image can correspond to an infinite number of 3D configurations. Traditional approaches like Non-Rigid Structure from Motion (NRSfM) require tracking points over multiple frames, making them slow and sensitive to sudden motion. Conversely, Template-Based Reconstruction (TBR) requires a perfect 3D "rest" model (template) beforehand, which is often unavailable in real-world scenarios.

HDM-Net addresses this gap by replacing rigid templates with a learned deformation model. It achieves state-of-the-art accuracy with inference times of approximately 5ms, enabling real-time applications in fields like medical imaging and augmented reality.

Methods

The Hybrid Deformation Model Network (HDM-Net) utilizes an encoder-decoder architecture to regress 3D point clouds directly from single images. Unlike methods that output intermediate representations like depth maps, HDM-Net outputs a dense $73 \times 73$ grid of 3D coordinates.

HDM-Net architecture featuring an encoder-decoder structure for direct 3D point cloud regression from a single input image.

The training process is guided by a combination of domain-specific loss functions:

  • 3D Error Loss: A standard supervised loss comparing predicted coordinates to ground truth.
  • Isometry Prior: A smoothing loss ensuring neighboring points maintain consistent distances, mimicking the physical resistance to stretching in real-world materials.
  • Differentiable Contour Loss: A novel contribution that projects the predicted 3D shape back to 2D to compare its silhouette with the input image. This is achieved through a differentiable pipeline using bilinear interpolation and soft thresholding.

Evaluation and Results

The model was trained on a synthetic dataset generated in Blender, containing over 4,600 unique deformation states under various lighting and texture conditions. Evaluation focused on generalization to unseen textures and illumination settings.

Qualitative comparison showing the input image, the HDM-Net reconstruction, and the ground truth 3D mesh.

HDM-Net demonstrates significant improvements in both speed and accuracy. It operates at ~200 FPS (5ms/frame), whereas traditional template-based methods like Yu et al. run at ~0.3 FPS. In terms of accuracy, HDM-Net achieved a 3D error ($e_{3D}$) of 0.0251 on endoscopically textured surfaces, vastly outperforming baselines like AMP (1.6189). Furthermore, the model proved robust to noise and generalized well to real-world imagery, such as surgical footage and musical notes, despite being trained solely on synthetic data.

Conclusion

HDM-Net provides a robust and efficient solution for monocular non-rigid reconstruction by learning a class-specific deformation model. While the model is currently limited by the variety of deformations present in its training data, its speed and accuracy make it a promising tool for clinical diagnostics and AR. Future work will explore the integration of GANs and RNNs to improve temporal consistency and handle more complex biological tissue deformations.

Summary updated

Abstract

Monocular dense 3D reconstruction of deformable objects is a hard ill-posed problem in computer vision. Current techniques either require dense correspondences and rely on motion and deformation cues, or assume a highly accurate reconstruction (referred to as a template) of at least a single frame given in advance and operate in the manner of non-rigid tracking. Accurate computation of dense point tracks often requires multiple frames and might be computationally expensive. Availability of a template is a very strong prior which restricts system operation to a pre-defined environment and scenarios. In this work, we propose a new hybrid approach for monocular non-rigid reconstruction which we call Hybrid Deformation Model Network (HDM-Net). In our approach, deformation model is learned by a deep neural network, with a combination of domain-specific loss functions. We train the network with multiple states of a non-rigidly deforming structure with a known shape at rest. HDM-Net learns different reconstruction cues including texture-dependent surface deformations, shading and contours. We show generalisability of HDM-Net to states not presented in the training dataset, with unseen textures and under new illumination conditions. Experiments with noisy data and a comparison with other methods demonstrate robustness and accuracy of the proposed approach and suggest possible application scenarios of the new technique in interventional diagnostics and augmented reality.
BibTeX
@inproceedings{golyanik2018hdm,
  author = {Vladislav Golyanik and Soshi Shimada and Kiran Varanasi and Didier Stricker},
  title = {{HDM-Net: Monocular Non-Rigid 3D Reconstruction with Learned Deformation Model}},
  booktitle = {EuroVR},
  year = {2018},
  url = {https://openreview.net/forum?id=Wc4Z3mfTnnK},
}
Figure