SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images

Chen-Hsuan Lin, Chaoyang Wang, Simon Lucey

TL;DR

SDF-SRN is a novel framework for reconstructing dense 3D objects from a single 2D image without requiring 3D ground truth or multi-view training data. By leveraging 2D distance transforms and a new differentiable rendering formulation for Signed Distance Functions (SDFs), the model learns to infer complex 3D geometries and topologies from "static" image collections, outperforming existing mesh- and voxel-based methods.

What Problem Does This Paper Address?

Reconstructing 3D shapes from 2D images is a fundamental challenge in computer vision. While deep learning has made strides here, most methods rely on 3D supervision (paired image-to-shape datasets), which is incredibly difficult to scale for real-world objects.

To avoid this, researchers turned to 2D supervision, using object silhouettes (masks) to "carve out" 3D shapes. However, most existing 2D-supervised methods require multi-view annotations—multiple photos of the exact same object instance from different angles—during the training phase. This requirement limits these models to synthetic datasets like ShapeNet.

This paper tackles the "single-view supervision" problem: learning to reconstruct 3D shapes from a collection of images where each object is seen only once.

SDF-SRN learns 3D reconstruction from single-view static images during training and infers 3D shapes from a single image at test time.

Methods

The authors introduce SDF-SRN, which combines implicit shape representations with an advanced rendering pipeline.

  1. Implicit Representation (SDF): Instead of using voxels (which are low-res) or meshes (which struggle with changing topologies), they represent the 3D shape as a Signed Distance Function (SDF). A Multi-Layer Perceptron (MLP) takes a 3D coordinate and outputs its distance to the nearest surface.
  2. Learning from Distance Transforms: The core mathematical contribution is a proposition relating 2D distance transforms (the distance of every pixel to the nearest silhouette boundary) to 3D SDF values. This allows the model to receive geometric supervision from every pixel in an image, not just those on the silhouette edge.
  3. Differentiable Rendering via SRN: They build upon Scene Representation Networks (SRN), which use a learnable ray-marching algorithm (an LSTM) to find the 3D surface.
  4. Bilevel Optimization: To ensure the ray-marcher actually hits the SDF surface ($f(x)=0$), they implement a second-stage refinement using the bisection method. This makes the rendering process geometrically consistent and differentiable, allowing the model to be trained end-to-end using only RGB images and silhouettes.

Datasets

The model was evaluated on two primary benchmarks:

  • ShapeNet v2: A synthetic dataset of 3D CAD models. The authors used the airplane, car, and chair categories, specifically testing the "single-view" setting where the model never sees the same instance from two different angles during training.
  • PASCAL3D+: A challenging real-world dataset containing images of airplanes, cars, and chairs with varying lighting, textures, and backgrounds.

Key Insights and Ideas

  • Distance Transforms as "Projections": The authors realized that a 2D distance transform isn't just a mask; it's a deterministic map that provides a lower bound for the 3D SDF. This "disperses" the geometric information across the entire image plane, providing much richer supervision than a binary mask.
  • Category-Level Priors: Even though the model only sees one view per object, it learns a "mean shape" and variations for a category (e.g., "what does a typical chair look like?") by seeing thousands of different chairs from different random angles.
  • Topology Independence: Because the model uses an implicit SDF rather than a mesh template, it can naturally reconstruct objects with holes (like the back of a chair) or complex branching (like airplane engines) without manual adjustments.

Key Results

  • State-of-the-Art Performance: On ShapeNet, SDF-SRN achieved significantly lower Chamfer Distances (a measure of 3D error) compared to baselines like SoftRas (mesh-based) and DVR (occupancy-based). For airplanes, SDF-SRN scored 0.193 vs. SoftRas's 0.250 and DVR's 1.795 (lower is better).
  • Real-World Robustness: On the PASCAL3D+ dataset, SDF-SRN outperformed the Category-specific Mesh Reconstruction (CMR) method, proving it can handle the "domain gap" of real-world textures and lighting.
  • Ablation Success: The authors showed that including the rendering loss ($L_{RGB}$) was crucial for resolving depth ambiguities that silhouettes alone cannot fix (e.g., whether a car door is concave or convex).
Qualitative comparison on ShapeNet showing SDF-SRN producing more accurate and topologically complex shapes than baselines.

Limitations and Future Work

  • Camera Poses: The current framework assumes that camera poses (the angle from which the photo was taken) are known during training. In many real-world "in-the-wild" scenarios, these are unknown.
  • Concavity Limits: While the rendering loss helps, silhouettes are fundamentally limited in capturing deep internal concavities that never appear on an object's outline.
  • Future Directions: The authors suggest integrating pose estimation (perhaps from 2D keypoints) to make the system fully autonomous for unlabelled image collections. They also see potential in applying this to more complex, non-rigid categories like animals or humans.

Summary updated

Abstract

Dense 3D object reconstruction from a single image has recently witnessed remarkable advances, but supervising neural networks with ground-truth 3D shapes is impractical due to the laborious process of creating paired image-shape datasets. Recent efforts have turned to learning 3D reconstruction without 3D supervision from RGB images with annotated 2D silhouettes, dramatically reducing the cost and effort of annotation. These techniques, however, remain impractical as they still require multi-view annotations of the same object instance during training. As a result, most experimental efforts to date have been limited to synthetic datasets. In this paper, we address this issue and propose SDF-SRN, an approach that requires only a single view of objects at training time, offering greater utility for real-world scenarios. SDF-SRN learns implicit 3D shape representations to handle arbitrary shape topologies that may exist in the datasets. To this end, we derive a novel differentiable rendering formulation for learning signed distance functions (SDF) from 2D silhouettes. Our method outperforms the state of the art under challenging single-view supervision settings on both synthetic and real-world datasets.
BibTeX
@inproceedings{lin2020sdf,
  author = {Chen-Hsuan Lin and Chaoyang Wang and Simon Lucey},
  title = {{SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=4rbM5hULv_},
}
Figure