Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes

Sara Hahner, Jochen GarckeWACV 2022

TL;DR

This paper introduces a novel mesh convolutional autoencoder capable of handling 3D surface meshes of varying sizes and topologies. By transforming irregular meshes into semi-regular ones, the authors enable the application of consistent convolutional filters and pooling operations, leading to significantly improved reconstruction accuracy and the ability to analyze unseen mesh classes.

What Problem Does This Paper Address?

Analyzing deforming 3D surface meshes is crucial in fields like computer graphics, robotics, and biomechanics. Autoencoders are powerful tools for this, learning low-dimensional representations that capture underlying dynamics. However, existing mesh convolutional autoencoders typically require input meshes to have a fixed connectivity and size. This limitation stems from the reliance on spectral convolutional layers or mesh-dependent pooling operations, which are sensitive to the mesh's structure. Consequently, models trained on one dataset or mesh type cannot be directly applied to others, hindering generalization and requiring extensive retraining. This paper addresses this gap by proposing a method that can handle meshes of different sizes and topologies, enabling more versatile and transferable analysis of deforming 3D shapes.

Methods

The core of the proposed method lies in a two-step process: remeshing and a semi-regular mesh convolutional autoencoder.

  1. Remeshing to Semi-Regular Meshes: Irregular input meshes are first coarsened into a low-resolution base mesh. This base mesh is then iteratively subdivided to create a "semi-regular" mesh. A key characteristic of these semi-regular meshes is that most vertices have a locally regular connectivity (specifically, six neighbors), forming "regional patches." This structure is hierarchical, allowing for multi-resolution analysis. The remeshing process ensures that the resulting meshes, while potentially having different overall sizes and topologies, share a consistent local structure.
  2. Semi-Regular Mesh Convolutional Autoencoder:
    • Patch-based Processing: Instead of processing the entire mesh at once, the autoencoder operates on these regional patches. This patch-based approach is what allows the network to handle meshes of different sizes.
  3. Hexagonal Convolution: The regular local structure of the semi-regular meshes allows for the application of 2D-convolutional kernels, similar to image processing. The authors specifically leverage hexagonal convolution, which has shown good performance on such grids. These kernels are designed to be anisotropic and preserve orientation, enhancing expressivity.
  4. Padding: To account for the context of a patch within the larger mesh, padding is applied. This padding incorporates features from neighboring patches, ensuring that local deformations are considered in a broader context.
  5. Pooling and Unpooling: A novel average pooling operator is introduced, which leverages the multi-scale structure inherent in semi-regular meshes. This operator effectively undoes the subdivision process. The decoder uses a corresponding unpooling operation to reconstruct the mesh at higher resolutions.
  6. Autoencoder Architecture: The autoencoder consists of an encoder with hexagonal convolutional layers and average pooling, followed by a fully connected layer to produce a latent representation. The decoder mirrors this with fully connected layers, hexagonal convolutional layers, and average unpooling.

Datasets

The authors evaluate their approach on several datasets:

  • GALLOP: Contains motion sequences of galloping horses, elephants, and camels. Notably, the meshes for these animals have significantly different connectivities and numbers of vertices.
  • FAUST: A dataset of 100 corresponding meshes representing different poses of 10 human bodies.
  • TRUCK and YARIS: Datasets derived from car crash simulations. These datasets consist of meshes representing car components that deform. The TRUCK dataset was used for training, while the YARIS dataset served as an unseen benchmark to test transfer learning capabilities.

Key Insights and Ideas

  • Semi-regularity as a Bridge: The key insight is that transforming irregular meshes into a semi-regular representation creates a consistent local structure that bridges the gap between general 3D meshes and the structured grids required by CNNs. This allows for the application of powerful convolutional techniques.
  • Patch-based Processing for Size Invariance: By processing individual patches, the network becomes inherently capable of handling meshes of varying sizes and vertex counts, a significant limitation of prior work.
  • Contextual Information via Padding: The use of padding to incorporate information from neighboring patches is crucial for maintaining a coherent reconstruction and understanding the global context of local deformations.
  • Transfer Learning Potential: The success on the YARIS dataset, which was unseen during training, demonstrates the potential for transfer learning and applying models trained on one type of deforming object to another.
  • Improved Reconstruction Quality: The method consistently outperforms existing state-of-the-art models, particularly in reconstructing fine details and smooth transitions between patches.

Key Results

  • Reconstruction Error Reduction: The proposed autoencoder achieves reconstruction errors more than 50% lower than state-of-the-art models (COMA and Neural3DMM) on the GALLOP and FAUST datasets. For FAUST, the reduction is over 80%.
  • Handling Unseen Data: The model successfully reconstructs unseen poses and even unseen animal classes (elephant) from the GALLOP dataset, demonstrating generalization.
  • Parameter Efficiency: The autoencoder requires at least 30% fewer parameters compared to COMA and Neural3DMM.
  • Transfer Learning Success: On the TRUCK and YARIS datasets, the model shows promising results, with the error on the unseen YARIS dataset being less than 3 times higher than the error on the TRUCK testing set, relative to part length.
  • Qualitative Improvements: Reconstructions are noted to be smoother at patch boundaries, and fine details like legs are reconstructed more accurately.

Limitations and Future Work

  • Remeshing Dependency: The performance is dependent on the quality of the initial remeshing process. While the paper describes their adaptation of the Garland-Heckbert algorithm, the inherent limitations of remeshing (e.g., potential loss of fine details or introduction of artifacts) might still apply.
  • Computational Cost of Remeshing: The initial remeshing step adds a preprocessing overhead, which might be a consideration for real-time applications.
  • Future Work: The authors plan to extend their architecture to other tasks such as shape matching and segmentation. They also suggest that the ability to handle different mesh sizes and topologies opens doors for more diverse applications in computer vision and graphics.

Summary updated

BibTeX
@inproceedings{hahner2022mesh,
  author = {Sara Hahner and Jochen Garcke},
  title = {{Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure