Learning a Hierarchical Latent-Variable Model of Voxelized 3D Shapes

Shikun Liu, Alexander G. Ororbia Ii, C. Lee GilesarXiv 2017 · arXiv

Abstract

Generating and recognizing 3D shapes is a cornerstone of computer vision. However, most deep learning models rely on a single, "flat" latent vector to represent an entire object, which often fails to capture the inherent hierarchical structure of real-world geometry. We present the Variational Shape Learner (VSL), a hierarchical latent-variable model that employs synaptic skip-connections to learn a coarse-to-fine representation of voxelized objects. VSL achieves state-of-the-art performance in unsupervised classification and 3D reconstruction from single 2D images, demonstrating significant parameter efficiency compared to existing models.

Introduction

Traditional 3D representation models assume that a single block of numbers can efficiently capture everything from fine-grained edges to global geometry. This "flat" approach is restrictive because objects are naturally hierarchical—composed of parts, surfaces, and edges. Previous state-of-the-art models often required extra supervision, such as viewpoint labels or keypoints, to learn these nuances. The VSL aims to learn these hierarchical structures in a fully unsupervised way, making the model more expressive and its internal representations more meaningful.

Methods

The VSL is built on the Variational Autoencoder (VAE) framework with a novel hierarchical architecture designed to capture structural depth.

Hierarchical Latent Structure

Instead of a single latent vector, the model utilizes a global latent code ($z_0$) as a "command unit" and multiple local latent codes ($z_{1:n}$) representing different levels of feature abstraction. Synaptic skip-connections tie these layers together in a top-down fashion, forcing the model to learn a coarse-to-fine process where higher levels guide lower levels in placing geometric features.

Pipeline and Training

The architecture consists of:

  • Encoder (3D-ConvNet): Maps 3D voxels to the hierarchical latent space.
  • Decoder (3D-DeConvNet): Reconstructs the 3D shape from the hierarchical codes.
  • Image Regressor (2D-ConvNet): Maps 2D RGB images into the 3D latent space for retrieval tasks.

The model optimizes a combination of reconstruction loss, KL divergence for regularization, and a specific "latent loss" ($L_{lat}$) to align 2D image features with 3D shape features. A "warming-up" schedule for the latent loss was found essential to avoid poor local minima during 2D-to-3D training.

Results and Discussion

The VSL was evaluated on ModelNet10, ModelNet40, and PASCAL 3D datasets, showing strong results in both classification and generative tasks.

Efficiency and Classification

A key finding is the model's efficiency; VSL achieves state-of-the-art unsupervised classification results using a feature vector of fewer than 100 dimensions. In contrast, models like 3D-GAN require millions of dimensions for similar accuracy. VSL achieved 91.0% accuracy on ModelNet10 and 84.5% on ModelNet40.

Single Image 3D Retrieval

On the PASCAL 3D dataset, VSL outperformed existing methods like 3D-R2N2 and NRSfM in 8 out of 10 categories based on Intersection-over-Union (IoU) metrics.

Generative Quality and Shape Arithmetic

Qualitatively, the model produces sharp, recognizable 3D voxels across diverse categories. The well-structured latent space allows for "shape arithmetic," such as adding "armrest" features to a standard chair latent code to generate a new armchair model.

Voxelized reconstructions across various categories including vehicles, furniture, and household items.

Conclusion

The Variational Shape Learner demonstrates that hierarchical latent structures are a highly efficient way to encode "object-ness" in 3D space. While currently limited to voxel grids (typically 30x30x30), which can be computationally expensive at higher resolutions, the model provides a robust framework for unsupervised 3D shape understanding. Future work will explore mesh or point cloud representations and stabilizing end-to-end joint training of the 2D and 3D components.

Summary updated

BibTeX
@misc{liu2017learning,
  author = {Shikun Liu and Alexander G. Ororbia II and C. Lee Giles},
  title = {{Learning a Hierarchical Latent-Variable Model of Voxelized 3D Shapes}},
  year = {2017},
  eprint = {1705.05994},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.05994},
}
Figure