PointAE: Point Auto-Encoder for 3D Statistical Shape and Texture Modelling

Hang Dai, Ling ShaoICCV 2019 · DOI

Abstract

Statistical Shape Modelling (SSM) is fundamental to various computer vision and medical imaging applications. Traditional methods, primarily based on Principal Component Analysis (PCA), often fail to capture high-frequency geometric details due to their linear nature. We introduce PointAE, a deep learning framework designed to perform 3D statistical shape and texture modelling directly on raw 3D point clouds. By utilizing a point-based auto-encoder architecture combined with a novel correspondence refinement block, PointAE achieves state-of-the-art reconstruction accuracy across diverse datasets, including faces, heads, and full bodies.

Introduction

The development of 3D Morphable Models (3DMMs) has traditionally relied on PCA to represent variations in human anatomy. While effective for global shape capture, PCA's linear constraints often result in the loss of fine-grained features such as wrinkles or specific facial contours. Recent deep learning approaches have attempted to address these limitations but frequently require converting 3D data into intermediate representations like 2D UV maps or voxels. Such preprocessing can introduce artifacts and information loss.

PointAE addresses these challenges by operating directly on 3D point sets. This approach avoids the "information tax" associated with data conversion and allows for the joint modelling of shape (geometry) and texture (color) within a unified latent space.

Methodology

The PointAE architecture is built upon a PointNet-inspired encoder-decoder structure. The encoder processes input point sets using 1D convolutional layers and batch normalization.

The PointAE architecture featuring an encoder with attention blocks and a decoder for joint shape and texture reconstruction.

A critical component of the encoder is the attention block, based on Squeeze-and-Excitation networks, which enables the model to prioritize the most informative features within the point cloud. To preserve spatial hierarchies, skip-connections are integrated throughout the network. PointAE treats input data as a 6-channel point set $(x, y, z, r, g, b)$, allowing the model to learn the inherent correlations between an object's physical structure and its visual appearance.

Correspondence Refinement

Establishing dense correspondence—ensuring specific vertices represent the same anatomical landmarks across different scans—is a significant hurdle in SSM. PointAE introduces a Correspondence Refinement Block to address this iteratively.

Visualization of the mesh refinement process and the resulting error distribution.

The refinement process involves a bootstrapping mechanism where the model generates a denoised version of the input. A Laplace-Beltrami (LB) regularized refinement block then compares this output to the raw scan, adjusting vertex positions to maximize fidelity while ensuring a smooth mesh structure. This refined data is subsequently used to further train the PointAE, creating a feedback loop that improves model accuracy without requiring perfect initial ground-truth correspondence.

Experimental Results

The efficacy of PointAE was evaluated using three diverse datasets: BU3DFE (facial expressions), Headspace (human head scans), and Caesar (full-body registered scans). Performance was quantified using Normalized Mean Error (NME).

PointAE consistently outperformed traditional PCA and existing deep learning models like MeshVAE. On the BU3DFE dataset, PointAE achieved an NME of 0.0074 with a 1024-dimensional latent space, significantly lower than PCA's 0.0187. Qualitative analysis confirms that PointAE successfully reconstructs intricate details, such as the folds of the mouth and ear contours, which are typically blurred in linear models.

Conclusion and Future Work

PointAE demonstrates the power of non-linear auto-encoders in capturing multi-modal distributions and fine-grained 3D details. By working directly on point clouds, it offers a more robust and simpler pipeline for 3D modelling. While the model currently requires an initial coarse correspondence, future work aims to develop a fully end-to-end system for unorganized point clouds and extend the framework to dynamic 4D data sequences.

Summary updated

BibTeX
@inproceedings{dai2019pointae,
  author = {Hang Dai and Ling Shao},
  title = {{PointAE: Point Auto-Encoder for 3D Statistical Shape and Texture Modelling}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00551},
}
Figure