Convolutional Neural Networks on non-uniform geometrical signals using Euclidean spectral transformation

Chiyu Max Jiang, Dequan Wang, Jingwei Huang, Philip Marcus, Matthias Nießner

Abstract

Convolutional Neural Networks (CNN) have been successful in processing data signals that are uniformly sampled in the spatial domain (e.g., images). However, most data signals do not natively exist on a grid, and in the process of being sampled onto a uniform physical grid suffer significant aliasing error and information loss. Moreover, signals can exist in different topological structures as, for example, points, lines, surfaces and volumes. It has been challenging to analyze signals with mixed topologies (for example, point cloud with surface mesh). To this end, we develop mathematical formulations for Non-Uniform Fourier Transforms (NUFT) to directly, and optimally, sample nonuniform data signals of different topologies defined on a simplex mesh into the spectral domain with no spatial sampling error. The spectral transform is performed in the Euclidean space, which removes the translation ambiguity from works on the graph spectrum. Our representation has four distinct advantages: (1) the process causes no spatial sampling error during the initial sampling, (2) the generality of this approach provides a unified framework for using CNNs to analyze signals of mixed topologies, (3) it allows us to leverage state-of-the-art backbone CNN architectures for effective learning without having to design a particular architecture for a particular data structure in an ad-hoc fashion, and (4) the representation allows weighted meshes where each element has a different weight (i.e., texture) indicating local properties. We achieve results on par with the state-of-the-art for the 3D shape retrieval task, and a new state-of-the-art for the point cloud to surface reconstruction task.

Introduction

While CNNs excel at processing grid-based data like 2D images, real-world 3D data typically exists in non-uniform formats such as point clouds, line meshes, or polygonal surfaces. Traditional methods to bridge this gap—such as voxelization, multi-view projections, or graph-based CNNs—often introduce significant compromises. Voxelization leads to memory-intensive aliasing and loss of detail, while multi-view approaches lose depth information. Graph CNNs, though powerful, often struggle to generalize across varying topologies and cannot easily utilize established high-performance CNN backbones. This paper proposes a unified framework to represent complex, non-uniform geometry in a way that preserves maximum information for standard CNN architectures.

Methodology: Euclidean Spectral Transformation

The core innovation of this work is the use of an analytical Non-Uniform Fourier Transform (NUFT) to map non-uniform signals directly into the spectral domain. Unlike discrete sampling methods that count points within a grid cell, the authors developed mathematical formulations for $j$-simplices (points, lines, triangles, and tetrahedra) to find the exact frequency representation of the geometry.

Overview of the NUFT pipeline for various simplex topologies and its application to shape classification and surface reconstruction.

The NUFT Pipeline

  1. Analytical Transform: A non-uniform signal (e.g., a mesh) is transformed via NUFT into its spectral representation. Because this transform is analytical, it eliminates initial spatial sampling errors.
  2. Inverse Mapping: An Inverse Fast Fourier Transform (IFFT) maps the spectral data back to a uniform physical grid.
  3. CNN Processing: This uniform grid is then processed by standard, high-performance CNN backbones like U-Net or DLA.
  4. Auxiliary Node Method: For solid volumes, the authors utilize a boundary-traversal technique to efficiently calculate the transform for the entire volume without exhaustive internal sampling.

Experimental Evaluation

The framework was evaluated across three distinct tasks to test its versatility and accuracy.

MNIST-as-Polygons

To demonstrate information preservation, the classic MNIST dataset was converted into vector polygons. The results showed that the NUFT representation is significantly more efficient than standard pixels. The model achieved high accuracy at much lower resolutions compared to binary pixel or distance function representations.

MNIST-as-Polygons experiment comparing NUFT to standard pixel-based and distance function representations across resolutions.

3D Shape Retrieval and Reconstruction

On the SHREC17 benchmark, the method performed on par with state-of-the-art multi-view models, notably outperforming other volumetric models despite lacking pre-training on large 2D image datasets. For 3D surface reconstruction (using ShapeNet55), the model established a new state-of-the-art, significantly outperforming methods like Deep Marching Cubes (DMC) and Poisson Surface Reconstruction (PSR) in Chamfer distance and overall accuracy.

Key Insights and Discussion

  • Information Efficiency: NUFT allows for "subgrid" precision. The Fourier basis enables the network to localize surfaces with non-linear boundaries that traditional voxel grids would miss. In some tests, NUFT achieved comparable accuracy to voxel grids while requiring 58x less resolution.
  • Topology Agnosticism: The mathematical framework is indifferent to whether the input is a point cloud or a textured mesh, providing a unified approach for mixed-topology data.
  • Robustness: The spectral representation proved highly resilient to noise within input point clouds, maintaining performance where spatial methods might degrade.

Conclusion

By analytically transforming non-uniform geometric signals into the spectral domain, this work provides a robust, efficient, and unified framework for applying standard CNNs to complex 3D data. The approach eliminates aliasing errors inherent in voxelization and allows researchers to leverage powerful existing CNN architectures for tasks ranging from shape classification to high-fidelity surface reconstruction.

Summary updated

Abstract

Convolutional Neural Networks (CNN) have been successful in processing data signals that are uniformly sampled in the spatial domain (e.g., images). However, most data signals do not natively exist on a grid, and in the process of being sampled onto a uniform physical grid suffer significant aliasing error and information loss. Moreover, signals can exist in different topological structures as, for example, points, lines, surfaces and volumes. It has been challenging to analyze signals with mixed topologies (for example, point cloud with surface mesh). To this end, we develop mathematical formulations for Non-Uniform Fourier Transforms (NUFT) to directly, and optimally, sample nonuniform data signals of different topologies defined on a simplex mesh into the spectral domain with no spatial sampling error. The spectral transform is performed in the Euclidean space, which removes the translation ambiguity from works on the graph spectrum. Our representation has four distinct advantages: (1) the process causes no spatial sampling error during the initial sampling, (2) the generality of this approach provides a unified framework for using CNNs to analyze signals of mixed topologies, (3) it allows us to leverage state-of-the-art backbone CNN architectures for effective learning without having to design a particular architecture for a particular data structure in an ad-hoc fashion, and (4) the representation allows weighted meshes where each element has a different weight (i.e., texture) indicating local properties. We achieve results on par with the state-of-the-art for the 3D shape retrieval task, and a new state-of-the-art for the point cloud to surface reconstruction task.
BibTeX
@inproceedings{jiang2019convolutional,
  author = {Chiyu Max Jiang and Dequan Wang and Jingwei Huang and Philip Marcus and Matthias Nießner},
  title = {{Convolutional Neural Networks on non-uniform geometrical signals using Euclidean spectral transformation}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure