Flow-based GAN for 3D Point Cloud Generation from a Single Image

Wei Yao, George Vosselman, Michael Ying YangarXiv 2022 · DOI · arXiv

Abstract

Generating a 3D point cloud from a single 2D image is a critical task for 3D scene understanding. Existing deep learning-based approaches typically rely on either explicit or implicit generative modeling, both of which often suffer from limited reconstruction quality. In this work, we introduce a hybrid explicit-implicit generative modeling scheme. Our approach leverages flow-based explicit generative models to sample point clouds at arbitrary resolutions while improving detailed 3D structures through implicit generative adversarial networks (GANs). Experimental results on the ShapeNet dataset demonstrate superior performance over existing methods. Furthermore, we demonstrate the model's generalization capabilities on cross-category synthetic images and real-world data from the PASCAL3D+ dataset.

Introduction

Reconstructing a 3D object from a single 2D image is a classic "ill-posed" problem in computer vision, as infinitely many 3D shapes can result in the same 2D projection. Current deep learning approaches generally fall into two categories:

  1. Point-based models (VAEs/GANs): These treat point clouds as fixed-size matrices. They are often restricted to a pre-defined resolution (e.g., 2,048 points), which limits their ability to represent complex, fine-grained geometry.
  2. Distribution-based models (Normalizing Flows): These treat a point cloud as a probability distribution. While they allow for sampling an arbitrary number of points at inference time, they often produce noisy results and struggle to capture sharp, intricate details.

This research bridges this gap by creating a model that is both flexible in resolution and precise in detail.

Methodology: Hybrid Explicit-Implicit Modeling

The proposed architecture, "FlowGAN," combines the strengths of Normalizing Flows and Generative Adversarial Networks through two primary components.

The Flow-based Generator

Built on a Variational Autoencoder (VAE) framework, the generator uses a series of 63 affine coupling layers (Normalizing Flows). During training, it learns to map a simple Gaussian distribution to the complex distribution of 3D points belonging to an object. The generator is conditioned on a latent vector derived from the input image via a ResNet18 encoder. Because it models a distribution, the system can sample any number of points (e.g., 1,000 or 10,000) at inference time without retraining.

The Cross-modal Discriminator

To ensure the generated points are not merely a blurry approximation, a discriminator regularizes the flow-based generator. This "GAN" component evaluates both the input 2D image and the generated 3D point cloud. It uses a PointNet-based encoder for 3D features and a ResNet-based encoder for 2D features, concatenating them to determine if the 3D shape faithfully matches the structure and style of the object in the image. This cross-modal supervision ensures the reconstructed 3D shape is faithful to the specific object shown in the picture.

Key Results

The hybrid approach yields significant improvements in both quality and efficiency. On the ShapeNet benchmark, the model outperformed previous state-of-the-art methods, including DPF-Nets and MixNFs, across Chamfer Distance (CD), Earth Mover’s Distance (EMD), and F1-Score.

Qualitative comparison of 3D point cloud reconstruction against state-of-the-art methods.

Efficiency and Flexibility

A major advantage of this method is its computational efficiency. Unlike MixNFs, which uses multiple flow models for different object parts, this method utilizes a single-flow-based generator. This design allows for an inference speed of 273 samples per second, compared to just 5 samples per second for MixNFs. Additionally, the model's performance remains stable or improves as the number of sampled points increases, demonstrating the robustness of the flow-based approach.

Real-world Generalization

While trained on synthetic ShapeNet data, the model demonstrates strong generalization to real-world images from the PASCAL3D+ dataset. Despite the domain gap between synthetic renders and real-world photographs, the model successfully reconstructs recognizable and structurally sound 3D shapes.

Conclusion and Future Work

By hybridizing explicit flow-based modeling with implicit GAN-based refinement, this work provides a powerful framework for single-image 3D reconstruction. While the model handles standard views effectively, future work will focus on bridging the remaining domain gap for real-world images and improving performance on highly occluded objects or extreme perspectives.

Summary updated

Abstract

Generating a 3D point cloud from a single 2D image is of great importance for 3D scene understanding applications. To reconstruct the whole 3D shape of the object shown in the image, the existing deep learning based approaches use either explicit or implicit generative modeling of point clouds, which, however, suffer from limited quality. In this work, we aim to alleviate this issue by introducing a hybrid explicit-implicit generative modeling scheme, which inherits the flow-based explicit generative models for sampling point clouds with arbitrary resolutions while improving the detailed 3D structures of point clouds by leveraging the implicit generative adversarial networks (GANs). We evaluate on the large-scale synthetic dataset ShapeNet, with the experimental results demonstrating the superior performance of the proposed method. In addition, the generalization ability of our method is demonstrated by performing on cross-category synthetic images as well as by testing on real images from PASCAL3D+ dataset.
BibTeX
@misc{yao2022flow,
  author = {Wei Yao and George Vosselman and Michael Ying Yang},
  title = {{Flow-based GAN for 3D Point Cloud Generation from a Single Image}},
  year = {2022},
  eprint = {2210.04072},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.04072},
}
Figure