Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projections Matching

Chao Chen, Zhizhong Han, Yu-Shen Liu, Matthias ZwickerICCV 2021

TL;DR

This paper introduces a novel unsupervised method for generating detailed 3D point clouds from 2D images. Instead of relying on pixel-wise comparisons, it uses a "2D projection matching" approach with irregularly sampled points from silhouette images, leading to significantly improved recovery of fine 3D structures.

What Problem Does This Paper Address?

Generating 3D shapes from 2D images is a fundamental problem in computer vision with applications ranging from 3D reconstruction to novel shape generation. While supervised methods achieve good results, they require expensive and time-consuming 3D ground truth data. Unsupervised methods aim to overcome this by learning from 2D images, often using differentiable renderers to compare generated 3D shapes with 2D ground truth images. However, existing unsupervised methods struggle to accurately reconstruct fine 3D structures like thin tubes or planes, likely due to the limitations of per-pixel comparisons and the inherent sparseness of 3D point clouds in 2D projections.

Methods

The core innovation of this paper is the 2D projection matching approach. Instead of using entire 2D silhouette images for supervision, the authors propose to:

  1. Irregular Point Supervision: Discretize the 2D ground truth silhouette images by randomly sampling 2D points within the silhouette. These sampled points form an "irregular point supervision" set.
  2. Projection Matching Loss: Train a neural network to generate a 3D point cloud such that its perspective projections onto the 2D image plane closely match the irregular point supervision set. This is achieved by minimizing a loss function based on the Chamfer Distance (CD) between the projected points and the sampled supervision points.
  3. Structure Adaptive Sampling (SAS): To improve consistency across different views, they introduce SAS, a method that samples points within each silhouette based on a ratio of area coverage, aiming to provide a more uniform distribution of supervision points.

This method avoids the complexities of traditional differentiable rendering pipelines (like visibility handling and shading) and directly focuses on matching projected 3D points to a sparse, yet informative, 2D representation.

Datasets

The authors evaluate their method using datasets derived from ShapeNet, a widely used 3D shape dataset. Specifically, they use rendered images and corresponding ground truth point clouds from three categories: chairs, cars, and airplanes. The benchmark provides images at different resolutions (32x32, 64x64, 128x128) and ground truth point clouds with varying numbers of points.

Key Insights and Ideas

  • Irregularity is Key: The paper argues that using irregularly sampled 2D points as supervision is more effective for capturing fine 3D structures than dense, pixel-wise comparisons. This is because fine structures are often represented by only a few pixels, making per-pixel losses less informative.
  • Beyond Per-Pixel Loss: By moving away from per-pixel difference losses, the method simplifies the pipeline and potentially avoids issues related to rendering artifacts or inconsistencies.
  • Robustness to Resolution: The irregular point supervision is less sensitive to the resolution of the input 2D images compared to methods that rely on pixel-level comparisons.
  • Structure Adaptive Sampling (SAS): The SAS method is crucial for ensuring that the irregular point supervision is consistent and informative across different viewpoints, which is vital for learning accurate 3D structures.

Key Results

The proposed method achieves state-of-the-art results in unsupervised 3D point cloud generation across various benchmarks.

  • Single Image Reconstruction: The method significantly outperforms existing differentiable renderers (like DSS, CAP, DPC, DRWR) in terms of Chamfer Distance (CD) across all categories and resolutions. For example, the mean CD is reported as 3.68 for their method compared to 4.16 for DRWR at 32x32 resolution.
  • Comparison with Supervised Methods: Their unsupervised method achieves competitive results compared to some supervised methods, notably outperforming the supervised method NOX in CD for all categories.
  • Qualitative Improvements: Visual comparisons demonstrate that their method can recover more detailed 3D structures, especially for complex shapes like chairs and airplanes, compared to previous methods.
  • Robustness: The method shows robustness to 2D supervision resolution, with less degradation in performance when trained on lower-resolution images compared to DRWR.

Limitations and Future Work

  • Computational Cost: While simpler than some differentiable renderers, the method still involves sampling points and calculating losses, which can be computationally intensive. The paper mentions that using too many sampled points can lead to redundancy.
  • Sampling Strategy: While SAS is introduced, the paper notes that other sampling methods can also work, and the performance difference between them is not drastic, suggesting room for further optimization in sampling strategies.
  • Future Work: The paper implicitly suggests that further exploration of sampling strategies and loss functions could lead to even better results. The robustness to resolution also opens avenues for using lower-quality 2D inputs. The method's success in novel shape generation also points towards its potential in generative modeling tasks.

Summary updated

BibTeX
@inproceedings{chen2021unsupervised,
  author = {Chao Chen and Zhizhong Han and Yu-Shen Liu and Matthias Zwicker},
  title = {{Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projections Matching}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
}
Figure