Shape Reconstruction Using Differentiable Projections and Deep Priors

Matheus Gadelha, Rui Wang, Subhransu Maji

Abstract

We present a learning-free method for 3D shape reconstruction that leverages the inherent geometric priors of neural network architectures. By combining differentiable rendering with the Deep Image Prior framework, our approach optimizes network weights to match noisy 2D projections without requiring large-scale training datasets. We demonstrate robust reconstruction across various modalities, including silhouettes, depth maps, and tomographic scans, while simultaneously correcting for viewpoint uncertainty.

Introduction

Reconstructing 3D geometry from 2D observations—such as silhouettes, depth maps, or X-ray projections—is a fundamental inverse problem in computer vision and medical imaging. Traditional approaches, including space carving and filtered back-projection, often struggle with high noise levels and imprecise camera calibration. While deep learning offers a solution through learned shape priors, these models typically necessitate extensive labeled 3D datasets.

Overview of the 3D shape reconstruction process from 2D silhouettes.

This work explores an alternative: using the architecture of a 3D convolutional neural network (CNN) as a self-regularizing prior. By making the projection process differentiable, we can backpropagate errors from 2D measurements directly to the network weights and camera parameters, enabling high-quality reconstruction from limited or noisy data.

Methodology

The proposed framework replaces explicit voxel grid optimization with a Deep Shape Prior. A 3D U-Net, initialized with fixed noise, generates a 3D volume. The core of the method lies in three Differentiable Projection Operators:

  1. Radon Projection ($T_R$): Facilitates tomographic reconstruction by simulating X-ray absorption.
  2. Silhouette Projection ($T_S$): Employs a smooth exponential falloff to create a differentiable mapping from 3D volumes to 2D outlines.
  3. Depth Image Projection ($T_D$): Enables reconstruction from noisy depth sensors by modeling visibility and occlusion.
Visualization of 2D shape projections including visibility and depth maps.

The optimization objective minimizes the discrepancy between the projected 3D shape and the observed 2D data. Because the projection operators are differentiable, gradient descent can simultaneously refine the 3D geometry and the camera viewpoint parameters.

Experiments and Results

The method was evaluated on benchmarks including the Shepp-Logan phantom, MRI slices from BrainWeb, and meshes from ModelNet40 (e.g., the Stanford Bunny and Utah Teapot).

  • Tomography: In sparse-view scenarios (e.g., 30 angles), the Deep Shape Prior significantly outperformed Filtered Back Projection (FBP) and iterative BM3D, achieving superior structural similarity (SSIM) scores.
  • Shape-from-Silhouette: Unlike traditional space carving, which produces jagged artifacts under viewpoint noise, our method generated smooth, plausible volumes by leveraging the CNN's architectural bias toward natural shapes.
  • Noisy Depth Maps: The system successfully recovered high-fidelity 3D structures from depth maps corrupted by significant Gaussian noise ($\sigma = 0.1$).

Conclusion

This research demonstrates that neural network architectures can serve as powerful, learning-free geometric priors for 3D reconstruction. By integrating differentiable rendering into the optimization loop, we achieve robustness to noise and viewpoint uncertainty without the need for massive training sets. While computationally intensive due to per-object optimization, this approach paves the way for more efficient representations beyond voxels, such as point clouds or implicit functions.

Summary updated

Abstract

We investigate the problem of reconstructing shapes from noisy and incomplete projections in the presence of viewpoint uncertainities. The problem is cast as an optimization over the shape given measurements obtained by a projection operator and a prior. We present differentiable projection operators for a number of reconstruction problems which when combined with the deep image prior or shape prior allows efficient inference through gradient descent. We apply our method on a variety of reconstruction problems, such as tomographic reconstruction from a few samples, visual hull reconstruction incorporating view uncertainties, and 3D shape reconstruction from noisy depth maps. Experimental results show that our approach is effective for such shape reconstruction problems, without requiring any task-specific training.
BibTeX
@inproceedings{gadelha2019shape,
  author = {Matheus Gadelha and Rui Wang and Subhransu Maji},
  title = {{Shape Reconstruction Using Differentiable Projections and Deep Priors}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00011},
}
Figure