Deep Learning on Irregular 3D Meshes

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2200 papers219 conceptsfrozen 2026-10-04

In the world of 2D computer vision, the "grid" is king. Images are composed of pixels arranged in a perfectly predictable, rectangular lattice. This regularity allows Convolutional Neural Networks (CNNs) to use a single sliding window to extract features across an entire image.

However, 3D geometry—the kind used in industrial design, medical imaging, and gaming—rarely lives on a grid. Instead, it is represented as meshes: irregular collections of vertices, edges, and faces. As Deok-Soo Kim and colleagues noted in The Visual Computer, this lack of a consistent neighborhood definition prevents the direct application of standard signal processing. On a mesh, one vertex might have three neighbors while another has ten, and there is no inherent "up" or "left" to guide a convolutional kernel.

Researchers have developed several distinct strategies to bridge this gap, effectively teaching neural networks to "see" on curved, irregular surfaces.

The Projective Shortcut

The most intuitive way to handle an irregular 3D mesh is to avoid processing it as a mesh entirely. Instead, we can project the 3D shape into a domain where standard 2D CNNs already excel.

One approach, pioneered by Evangelos Kalogerakis in Projective Convolutional Networks, involves "looking" at the 3D object from multiple 2D viewpoints. By aggregating features from these images, the model can perform semantic segmentation without ever needing to define a 3D convolution. Similarly, Jinglu Wang and Bo Sun developed MVPNet, which embeds 3D points into regular 2D grids to leverage efficient 2D convolutions for high-resolution surface reconstruction.

While effective, these methods can lose global context or struggle with internal structures. To address this, Xiangwen Zhao proposed slice-sampling, which treats the object like a CT scan, capturing internal geometry that surface-level projections might miss.

Synchronizing the Spectral Domain

If the spatial domain is too messy, we can move to the spectral domain. Just as a 1D signal can be decomposed into frequencies using a Fourier Transform, a 3D mesh can be decomposed using the eigenvectors of its Laplacian matrix.

The challenge is that every mesh has a different "basis"—the mathematical equivalent of a different alphabet. Li Yi, Hao Su, and Leonidas Guibas solved this in SyncSpecCNN by introducing a "Spectral Transformer Network." This synchronizes the underlying mathematical bases of different shapes, allowing the network to share weights across objects with completely different connectivity. This allows for effective part segmentation even when the mesh topology is highly irregular.

Intrinsic Graph Convolutions

Rather than transforming the mesh, some researchers aim to redefine convolution to work directly on the graph of the mesh. In these spatial methods, the "kernel" must adapt to the local neighborhood dynamically.

Nitika Verma and Edmond Boyer introduced FeaStNet, which uses "feature-steered" graph convolutions. Instead of using fixed spatial offsets (like "one pixel to the right"), the network learns to map filter weights to neighbors based on the features of those neighbors.

Other models, like MeshNet by Yutong Feng and Yue Gao, shift the focus from vertices to the triangular faces themselves. By treating faces as the fundamental unit and separating spatial location from local geometry, they overcome the structural inconsistency of varying vertex counts.

Canonical Parameterization

A third path involves mathematically "unwrapping" the 3D surface onto a regular 2D domain—a process called parameterization. If you can map a complex mesh to a flat surface without too much distortion, you can use standard CNNs again.

Haggai Maron and colleagues demonstrated this in Convolutional neural networks on surfaces via seamless toric covers, where they mapped sphere-like shapes onto a flat torus. This enables translation-invariant convolutions that preserve the global context of the shape. In the realm of physics simulation, Hong Qin used conformal cube maps to solve fluid dynamics equations on GPUs, which are natively optimized for such regular grid structures.

Generative Modeling and Self-Supervision

The most recent frontier is moving beyond classification toward generation and self-supervised learning. Because labeling 3D meshes is labor-intensive, researchers are adapting techniques like Masked Autoencoders (MAE) to 3D data.

Shanshan Zhao and Jing Zhang introduced MeshMAE, which learns by reconstructing missing geometric patches. By forcing the model to "fill in the blanks" of a masked mesh, it develops a deep understanding of 3D structure without needing human labels. For generation, Antonio Alliegro developed PolyDiff, a diffusion-based model that treats meshes as "quantized triangle soups," allowing for the direct generation of ready-to-use 3D assets.

These diverse approaches—from spectral synchronization to projective mapping—all share a common goal: taming the inherent irregularity of 3D surfaces to unlock the same "deep learning revolution" that transformed 2D images.

Papers in this version

Showing 60 of 200 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 335 papers
  2. Version 1 · 2026-10-04 · 200 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_non_euclidean_data_representation_a912e074,
  title = {non-Euclidean data representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-euclidean-data-representation/rdr_a912e07496b24d7eb342040e5d46f120},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_deep_learning_on_irregular_3d_meshes_a912e074_v1,
  title = {Deep Learning on Irregular 3D Meshes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deep-learning-on-irregular-3d-meshes/rdr_a912e07496b24d7eb342040e5d46f120/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}