Unsupervised 3D Reconstruction Networks

Geonho Cha, Minsik Lee, Songhwai Oh

Abstract

Recovering 3D structures from 2D observations is a fundamental but ill-posed challenge in computer vision. Traditional methods often require ground-truth 3D models or struggle to generalize across different object instances. This paper introduces 3D-URN (Unsupervised 3D Reconstruction Networks), a deep learning framework that learns category-level 3D priors without any 3D supervision. By utilizing a multi-basis shape reconstructor and a differentiable rotation refiner, 3D-URN achieves state-of-the-art performance in both rigid and non-rigid 3D reconstruction, enabling instantaneous inference for unseen objects.

Introduction

The task of 3D reconstruction from 2D feature points is inherently ambiguous, as multiple 3D configurations can produce the same 2D projection. Existing approaches typically fall into two categories: Non-Rigid Structure from Motion (NRSfM), which focuses on specific objects over time, and Structure from Category (SfC), which leverages commonalities within a class. However, NRSfM often fails to generalize, and SfC methods are frequently optimization-heavy, requiring significant computation for every new instance.

3D-URN bridges this gap by training a neural network to act as a fast inference engine. Once trained on a category (e.g., cars or faces), the network can reconstruct the 3D shape of a new, unseen instance in a single forward pass, effectively learning the "essence" of the category.

Methods

The 3D-URN architecture employs a dual-branch approach to solve for shape and rotation simultaneously in an unsupervised loop.

3D Shape Reconstructor

Instead of directly regressing a single 3D shape, the network utilizes $n_f$ parallel reconstructors to generate a set of basis shapes. A weight estimator then predicts the optimal blending coefficients ($w$) to combine these bases into the final 3D structure. This hierarchical approach allows the model to capture complex variations within a category.

Architecture of the 3D-URN shape reconstructor, illustrating the multi-basis blending and hierarchical modules.

Rotation Estimator & Refiner

To project the 3D shape back into the 2D plane for training, the network must estimate the camera's rotation. Because raw neural network outputs often fail to meet the mathematical requirements of a rotation matrix (orthogonality), the authors implemented a Rotation Refiner. This differentiable layer uses Singular Value Decomposition (SVD) to enforce orthogonality and resolve reflection ambiguities.

Unsupervised Learning Objectives

Training is driven by two primary loss functions:

  1. Projection Loss: Measures the discrepancy between the projected 3D shape and the original 2D input points.
  2. Low-Rank Prior Loss: Uses a nuclear-norm constraint to ensure that reconstructed shapes across a batch remain consistent with the category's underlying structure.

To prevent vanishing gradients in deep architectures, the authors applied "deep supervision," calculating losses at every intermediate module rather than just the final output.

Key Results and Insights

3D-URN was evaluated on several benchmarks, including PASCAL3D+ for rigid objects and the MUCT face database for non-rigid structures.

  • State-of-the-Art Performance: On PASCAL3D+, 3D-URN achieved a mean 3D error of 0.168, outperforming previous unsupervised methods like MUS (0.186) and CNR (0.192).
  • Generalization: A key finding was the model's ability to handle unseen instances. In tests, the network performed nearly as well on novel data as it did on the training set, proving it learns a robust category prior.
  • Non-Rigid Reconstruction: The model successfully recovered complex deformations in faces and moving cloth without requiring temporal smoothness assumptions.
Qualitative reconstruction results on the MUCT face dataset, showing the model's accuracy in capturing non-rigid 3D facial structures.

Conclusion

3D-URN represents a significant step forward in unsupervised 3D vision. By combining differentiable geometry with deep category-level learning, it provides a fast and robust solution for 3D reconstruction. Future work may focus on handling occluded data points and developing universal models capable of reconstructing multiple categories simultaneously.

Summary updated

Abstract

In this paper, we propose 3D unsupervised reconstruction networks (3D-URN), which reconstruct the 3D structures of instances in a given object category from their 2D feature points under an orthographic camera model. 3D-URN consists of a 3D shape reconstructor and a rotation estimator, which are trained in a fully-unsupervised manner incorporating the proposed unsupervised loss functions. The role of the 3D shape reconstructor is to reconstruct the 3D shape of an instance from its 2D feature points, and the rotation estimator infers the camera pose. After training, 3D-URN can infer the 3D structure of an unseen instance in the same category, which is not possible in the conventional schemes of non-rigid structure from motion and structure from category. The experimental result shows the state-of-the-art performance, which demonstrates the effectiveness of the proposed method.
BibTeX
@inproceedings{cha2019unsupervised,
  author = {Geonho Cha and Minsik Lee and Songhwai Oh},
  title = {{Unsupervised 3D Reconstruction Networks}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00395},
}
Figure