Self-Supervised 3D Mesh Reconstruction From Single Images

Tao Hu, Liwei Wang, Xiaogang Xu, Shu Liu, Jiaya Jia

Abstract

Recent single-view 3D reconstruction methods reconstruct object's shape and texture from a single image with only 2D image-level annotation. However, without explicit 3D attribute-level supervision, it is still difficult to achieve satisfying reconstruction accuracy. In this paper, we propose a Self-supervised Mesh Reconstruction (SMR) approach to enhance 3D mesh attribute learning process. Our approach is motivated by observations that (1) 3D attributes from interpolation and prediction should be consistent, and (2) feature representation of landmarks from all images should be consistent. By only requiring silhouette mask annotation, our SMR can be trained in an end-to-end manner and generalizes to reconstruct natural objects of birds, cows, motorbikes, etc. Experiments demonstrate that our approach improves both 2D supervised and unsupervised 3D mesh reconstruction on multiple datasets. We also show that our model can be adapted to other image synthesis tasks, e.g., novel view generation, shape transfer, and texture transfer, with promising results. Our code is publicly available at https://github.com/Jia-Research-Lab.

Introduction

Reconstructing a 3D object from a single 2D image is a fundamental challenge in computer vision. While deep learning has significantly advanced this field, high-performing models typically rely on extensive 3D supervision—expensive datasets of 3D CAD models or scans that are difficult to scale to diverse real-world objects.

Existing 2D-supervised methods attempt to bridge this gap using differentiable renderers to project predicted 3D shapes back into 2D space for comparison. However, these methods often suffer from low accuracy because multiple 3D shapes can appear identical from a single perspective. The proposed Self-supervised Mesh Reconstruction (SMR) framework addresses this by introducing novel consistency constraints that achieve 3D-level accuracy using only 2D silhouette annotations.

Methodology

The SMR architecture employs an attribute encoder ($E_\theta$) to disentangle an input image into four key components: Camera pose, Shape, Texture, and Light. These attributes are then processed by a differentiable renderer ($R$) to produce a reconstructed 2D image and silhouette.

The SMR architecture featuring the attribute encoder, differentiable renderer, and the two core consistency losses: Interpolated Consistency (IC) and Landmark Consistency (LC).

To enhance the learning process without 3D ground truth, the authors introduce two self-supervised losses:

  1. Interpolated Consistency (IC): This module is based on the insight that interpolating between the predicted 3D attributes of two different images (e.g., a red bird and a blue bird) should result in a valid "hybrid" 3D object. By rendering this hybrid object into a synthetic image, the model can use the known interpolated attributes as "pseudo-3D" labels to train the encoder. This effectively creates an infinite supply of training data from novel viewpoints.
  2. Landmark Consistency (LC): To ensure local geometric accuracy, mesh vertices are treated as semantic landmarks. The model projects these vertices onto the 2D image to extract local features. A classifier then ensures that specific landmarks (e.g., a bird's beak) maintain a consistent feature representation across all images in a category, preventing "texture drifting" and ensuring semantic alignment.

Experiments and Results

The model was evaluated across several diverse datasets, including ShapeNet (synthetic CAD models), BFM (3D faces), CUB-200-2011 (real-world birds), and LSUN (in-the-wild animals and vehicles).

  • Quantitative Performance: On ShapeNet, SMR achieved a mean 3D Intersection over Union (IoU) of 46.5%, significantly outperforming standard 2D-supervised methods (42.8%) and nearly matching models trained with full 3D supervision (47.2%). On the CUB-200-2011 dataset, SMR achieved a state-of-the-art Mask IoU of 80.6%.
  • Qualitative Results: The model demonstrates high fidelity in recovering complex shapes and textures from single images, as shown in the reconstruction of various bird species.
Qualitative reconstruction results on the CUB-200-2011 dataset, showing the input image, reconstructed object, and the underlying 3D shape from multiple angles.

Beyond reconstruction, the disentangled nature of the learned attributes allows for impressive downstream applications, such as shape transfer (applying one object's geometry to another's texture), re-lighting, and novel view synthesis.

Conclusion

SMR demonstrates that high-quality 3D mesh reconstruction is possible without explicit 3D supervision. By leveraging Interpolated and Landmark Consistency, the model creates a robust pseudo-supervision loop that generalizes well to real-world objects. While currently focused on rigid or semi-rigid categories, future work aims to extend this framework to highly articulated non-rigid objects like human bodies and move toward fully unsupervised reconstruction that eliminates the need for silhouette masks.

Summary updated

Abstract

Recent single-view 3D reconstruction methods reconstruct object's shape and texture from a single image with only 2D image-level annotation. However, without explicit 3D attribute-level supervision, it is still difficult to achieve satisfying reconstruction accuracy. In this paper, we propose a Self-supervised Mesh Reconstruction (SMR) approach to enhance 3D mesh attribute learning process. Our approach is motivated by observations that (1) 3D attributes from interpolation and prediction should be consistent, and (2) feature representation of landmarks from all images should be consistent. By only requiring silhouette mask annotation, our SMR can be trained in an end-to-end manner and generalizes to reconstruct natural objects of birds, cows, motorbikes, etc. Experiments demonstrate that our approach improves both 2D supervised and unsupervised 3D mesh reconstruction on multiple datasets. We also show that our model can be adapted to other image synthesis tasks, e.g., novel view generation, shape transfer, and texture transfer, with promising results. Our code is publicly available at https://github.com/Jia-Research-Lab.
BibTeX
@inproceedings{hu2021self,
  author = {Tao Hu and Liwei Wang and Xiaogang Xu and Shu Liu and Jiaya Jia},
  title = {{Self-Supervised 3D Mesh Reconstruction From Single Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=JQjUHHp46ZH},
}
Figure