Abstract
Statistical Shape Modelling (SSM) is fundamental to various computer vision and medical imaging applications. Traditional methods, primarily based on Principal Component Analysis (PCA), often fail to capture high-frequency geometric details due to their linear nature. We introduce PointAE, a deep learning framework designed to perform 3D statistical shape and texture modelling directly on raw 3D point clouds. By utilizing a point-based auto-encoder architecture combined with a novel correspondence refinement block, PointAE achieves state-of-the-art reconstruction accuracy across diverse datasets, including faces, heads, and full bodies.
Introduction
The development of 3D Morphable Models (3DMMs) has traditionally relied on PCA to represent variations in human anatomy. While effective for global shape capture, PCA's linear constraints often result in the loss of fine-grained features such as wrinkles or specific facial contours. Recent deep learning approaches have attempted to address these limitations but frequently require converting 3D data into intermediate representations like 2D UV maps or voxels. Such preprocessing can introduce artifacts and information loss.
PointAE addresses these challenges by operating directly on 3D point sets. This approach avoids the "information tax" associated with data conversion and allows for the joint modelling of shape (geometry) and texture (color) within a unified latent space.
Methodology
The PointAE architecture is built upon a PointNet-inspired encoder-decoder structure. The encoder processes input point sets using 1D convolutional layers and batch normalization.
A critical component of the encoder is the attention block, based on Squeeze-and-Excitation networks, which enables the model to prioritize the most informative features within the point cloud. To preserve spatial hierarchies, skip-connections are integrated throughout the network. PointAE treats input data as a 6-channel point set $(x, y, z, r, g, b)$, allowing the model to learn the inherent correlations between an object's physical structure and its visual appearance.
Correspondence Refinement
Establishing dense correspondence—ensuring specific vertices represent the same anatomical landmarks across different scans—is a significant hurdle in SSM. PointAE introduces a Correspondence Refinement Block to address this iteratively.
The refinement process involves a bootstrapping mechanism where the model generates a denoised version of the input. A Laplace-Beltrami (LB) regularized refinement block then compares this output to the raw scan, adjusting vertex positions to maximize fidelity while ensuring a smooth mesh structure. This refined data is subsequently used to further train the PointAE, creating a feedback loop that improves model accuracy without requiring perfect initial ground-truth correspondence.
Experimental Results
The efficacy of PointAE was evaluated using three diverse datasets: BU3DFE (facial expressions), Headspace (human head scans), and Caesar (full-body registered scans). Performance was quantified using Normalized Mean Error (NME).
PointAE consistently outperformed traditional PCA and existing deep learning models like MeshVAE. On the BU3DFE dataset, PointAE achieved an NME of 0.0074 with a 1024-dimensional latent space, significantly lower than PCA's 0.0187. Qualitative analysis confirms that PointAE successfully reconstructs intricate details, such as the folds of the mouth and ear contours, which are typically blurred in linear models.
Conclusion and Future Work
PointAE demonstrates the power of non-linear auto-encoders in capturing multi-modal distributions and fine-grained 3D details. By working directly on point clouds, it offers a more robust and simpler pipeline for 3D modelling. While the model currently requires an initial coarse correspondence, future work aims to develop a fully end-to-end system for unorganized point clouds and extend the framework to dynamic 4D data sequences.
Summary updated