3D Semantic Scene Completion from a Single Depth Image Using Adversarial Training

Yueh-Tung Chen, Martin Garbade, Juergen Gall

Introduction

3D Semantic Scene Completion (SSC) is the task of predicting the full 3D geometry and semantic labels of a scene from a single 2.5D depth image. While humans naturally infer occluded structures—such as the floor continuing behind a piece of furniture—computational models face significant challenges due to the ambiguity of hidden spaces. Traditional 3D Convolutional Neural Networks (CNNs) typically minimize voxel-wise error, which often results in "blurry" or cluttered reconstructions that lack structural coherence. This paper investigates the use of Generative Adversarial Networks (GANs) to enforce more realistic and sharp 3D scene completions.

Proposed Architecture

The authors propose a conditional GAN (cGAN) framework consisting of a generator and a discriminator designed specifically for 3D volumetric data.

The proposed GAN architecture featuring a generator based on SSCNet and a discriminator for global or local adversarial loss.

The Generator is based on the SSCNet architecture, utilizing dilated convolutions to capture broad spatial context while maintaining high resolution. It transforms a 3D Truncated Signed Distance Function (TSDF) into a voxel grid of semantic labels.

The Discriminator is conditioned on the input depth image to ensure that generated geometry remains consistent with observed data. The authors also introduced a Local Adversarial Loss, where the discriminator evaluates smaller local cubes within the volume rather than the entire room at once. This approach encourages the model to capture fine-grained details and prevents "mode collapse" on large, simple structures like walls.

Experimental Setup and Datasets

The model was evaluated across three benchmarks:

  1. NYU Depth v2 (Kinect): Real-world data with inherent noise and potential misalignment between depth maps and human-annotated 3D labels.
  2. NYU Depth v2 (CAD): A cleaner version where depth maps are rendered directly from 3D CAD models, ensuring perfect alignment.
  3. SUNCG: A large-scale synthetic dataset providing perfect ground truth for training.

Results and Discussion

The study revealed that the effectiveness of adversarial training is highly dependent on data quality. On the "noisy" NYU Kinect dataset, the GAN models actually performed worse than the baseline, as they attempted to learn the distribution of misaligned labels. However, on clean data, the benefits were clear.

Qualitative comparison of scene completion results. The GAN-based models produce sharper boundaries and fewer cluttered voxels compared to the SSCNet baseline.

On the NYU CAD dataset, the conditional GAN with local loss (SSC-cGAN-LL) achieved a semantic scene completion accuracy of 42.3%, compared to the 37.6% of the SSCNet baseline. Visually, the GAN-generated scenes are significantly cleaner, effectively learning that unoccupied space should remain empty, which results in sharper object boundaries and more realistic room structures.

Conclusion

Adversarial training offers a promising path toward more realistic 3D scene completion by forcing models to learn the underlying distribution of real-world geometry. While highly sensitive to data alignment, the use of conditional and local adversarial losses significantly improves performance on high-quality datasets, producing structurally coherent and visually plausible 3D environments.

Summary updated

BibTeX
@inproceedings{chen2019d,
  author = {Yueh-Tung Chen and Martin Garbade and Juergen Gall},
  title = {{3D Semantic Scene Completion from a Single Depth Image Using Adversarial Training}},
  booktitle = {ICIP},
  year = {2019},
  doi = {10.1109/icip.2019.8803174},
}
Figure