Abstract
Dense 3D object reconstruction from a single image has recently witnessed remarkable advances, but supervising neural networks with ground-truth 3D shapes is impractical due to the laborious process of creating paired image-shape datasets. Recent efforts have turned to learning 3D reconstruction without 3D supervision from RGB images with annotated 2D silhouettes, dramatically reducing the cost and effort of annotation. These techniques, however, remain impractical as they still require multi-view annotations of the same object instance during training. As a result, most experimental efforts to date have been limited to synthetic datasets. In this paper, we address this issue and propose SDF-SRN, an approach that requires only a single view of objects at training time, offering greater utility for real-world scenarios. SDF-SRN learns implicit 3D shape representations to handle arbitrary shape topologies that may exist in the datasets. To this end, we derive a novel differentiable rendering formulation for learning signed distance functions (SDF) from 2D silhouettes. Our method outperforms the state of the art under challenging single-view supervision settings on both synthetic and real-world datasets.
Introduction
Reconstructing 3D geometry from a single 2D image is a long-standing goal in computer vision. While deep learning has made significant strides, most successful methods rely on large-scale datasets of paired 2D images and 3D ground-truth shapes. Collecting such 3D data for real-world objects is prohibitively expensive and often impossible.
To circumvent this, researchers have explored unsupervised or weakly-supervised learning using 2D silhouettes. However, a major limitation of current state-of-the-art methods is the requirement for multi-view supervision during training—the model must observe the same object instance from multiple viewpoints. In practice, large-scale image collections typically consist of "unlinked" static images where each object is seen only once. Furthermore, traditional 3D representations like voxels or meshes often struggle with memory efficiency or complex topologies.
SDF-SRN addresses these challenges by learning implicit 3D shape representations from single-view image collections. By utilizing Signed Distance Functions (SDFs), the model can represent arbitrary topologies smoothly and efficiently.
Methodology
The core of SDF-SRN lies in its ability to bridge the gap between 2D silhouettes and 3D SDFs through two primary innovations.
SDF Supervision from 2D Distance Transforms
Standard silhouette-based supervision only provides information at the object boundaries. SDF-SRN leverages 2D Distance Transforms, which map every pixel to its distance from the nearest silhouette boundary. The authors demonstrate that this 2D distance serves as a lower bound for the 3D SDF. This mathematical insight allows the model to receive geometric signals from every pixel in the image, significantly improving the learning signal compared to binary masks.
Differentiable Rendering with Learned Ray-Marching
The framework extends Scene Representation Networks (SRN) by incorporating a learned ray-marching algorithm. An LSTM-based recurrent neural network predicts steps along a camera ray to find the surface intersection. To ensure high precision, SDF-SRN introduces a second-stage optimization that aligns the predicted depth with the "zero-level set" of the SDF, representing the true object surface.
The training objective combines an SDF loss derived from the distance transform, an RGB rendering loss for appearance consistency, and an Eikonal regularization term to maintain the properties of a valid distance field.
Experiments and Results
The performance of SDF-SRN was evaluated on the synthetic ShapeNet v2 dataset and the real-world PASCAL3D+ benchmark.
Quantitative and Qualitative Performance
SDF-SRN demonstrates a significant leap in accuracy over previous unsupervised methods. On ShapeNet, it achieved a Chamfer distance of 0.401 for the chair category, vastly outperforming DVR (1.367). This is particularly notable as chairs often feature complex, non-genus-zero topologies (e.g., holes in the backrest) that mesh-based methods struggle to capture.
On real-world data from PASCAL3D+, SDF-SRN produced cleaner and more semantically accurate reconstructions than existing mesh-based approaches like CMR, which often result in "blobby" geometries. The model successfully learns category-specific priors, allowing it to reconstruct plausible geometry for parts of the object not visible in the input image.
Discussion and Conclusion
Despite its success, SDF-SRN has limitations. It currently requires known camera poses during training and can struggle with heavy occlusions. Future research could focus on integrating pose-estimation modules to enable training on completely uncalibrated image collections. Additionally, extending the model from category-specific training to a more general-purpose 3D reconstructor remains an important future direction.
In summary, SDF-SRN provides a robust framework for learning high-quality 3D reconstruction from single-view static images. By combining implicit SDF representations with 2D distance transform supervision, it overcomes the need for multi-view training data and handles complex topologies with ease, marking a significant step toward scalable 3D vision in the real world.
Summary updated