Abstract
Convolutional Neural Networks (CNN) have been successful in processing data signals that are uniformly sampled in the spatial domain (e.g., images). However, most data signals do not natively exist on a grid, and in the process of being sampled onto a uniform physical grid suffer significant aliasing error and information loss. Moreover, signals can exist in different topological structures as, for example, points, lines, surfaces and volumes. It has been challenging to analyze signals with mixed topologies (for example, point cloud with surface mesh). To this end, we develop mathematical formulations for Non-Uniform Fourier Transforms (NUFT) to directly, and optimally, sample nonuniform data signals of different topologies defined on a simplex mesh into the spectral domain with no spatial sampling error. The spectral transform is performed in the Euclidean space, which removes the translation ambiguity from works on the graph spectrum. Our representation has four distinct advantages: (1) the process causes no spatial sampling error during the initial sampling, (2) the generality of this approach provides a unified framework for using CNNs to analyze signals of mixed topologies, (3) it allows us to leverage state-of-the-art backbone CNN architectures for effective learning without having to design a particular architecture for a particular data structure in an ad-hoc fashion, and (4) the representation allows weighted meshes where each element has a different weight (i.e., texture) indicating local properties. We achieve results on par with the state-of-the-art for the 3D shape retrieval task, and a new state-of-the-art for the point cloud to surface reconstruction task.
Introduction
While CNNs excel at processing grid-based data like 2D images, real-world 3D data typically exists in non-uniform formats such as point clouds, line meshes, or polygonal surfaces. Traditional methods to bridge this gap—such as voxelization, multi-view projections, or graph-based CNNs—often introduce significant compromises. Voxelization leads to memory-intensive aliasing and loss of detail, while multi-view approaches lose depth information. Graph CNNs, though powerful, often struggle to generalize across varying topologies and cannot easily utilize established high-performance CNN backbones. This paper proposes a unified framework to represent complex, non-uniform geometry in a way that preserves maximum information for standard CNN architectures.
Methodology: Euclidean Spectral Transformation
The core innovation of this work is the use of an analytical Non-Uniform Fourier Transform (NUFT) to map non-uniform signals directly into the spectral domain. Unlike discrete sampling methods that count points within a grid cell, the authors developed mathematical formulations for $j$-simplices (points, lines, triangles, and tetrahedra) to find the exact frequency representation of the geometry.
The NUFT Pipeline
- Analytical Transform: A non-uniform signal (e.g., a mesh) is transformed via NUFT into its spectral representation. Because this transform is analytical, it eliminates initial spatial sampling errors.
- Inverse Mapping: An Inverse Fast Fourier Transform (IFFT) maps the spectral data back to a uniform physical grid.
- CNN Processing: This uniform grid is then processed by standard, high-performance CNN backbones like U-Net or DLA.
- Auxiliary Node Method: For solid volumes, the authors utilize a boundary-traversal technique to efficiently calculate the transform for the entire volume without exhaustive internal sampling.
Experimental Evaluation
The framework was evaluated across three distinct tasks to test its versatility and accuracy.
MNIST-as-Polygons
To demonstrate information preservation, the classic MNIST dataset was converted into vector polygons. The results showed that the NUFT representation is significantly more efficient than standard pixels. The model achieved high accuracy at much lower resolutions compared to binary pixel or distance function representations.
3D Shape Retrieval and Reconstruction
On the SHREC17 benchmark, the method performed on par with state-of-the-art multi-view models, notably outperforming other volumetric models despite lacking pre-training on large 2D image datasets. For 3D surface reconstruction (using ShapeNet55), the model established a new state-of-the-art, significantly outperforming methods like Deep Marching Cubes (DMC) and Poisson Surface Reconstruction (PSR) in Chamfer distance and overall accuracy.
Key Insights and Discussion
- Information Efficiency: NUFT allows for "subgrid" precision. The Fourier basis enables the network to localize surfaces with non-linear boundaries that traditional voxel grids would miss. In some tests, NUFT achieved comparable accuracy to voxel grids while requiring 58x less resolution.
- Topology Agnosticism: The mathematical framework is indifferent to whether the input is a point cloud or a textured mesh, providing a unified approach for mixed-topology data.
- Robustness: The spectral representation proved highly resilient to noise within input point clouds, maintaining performance where spatial methods might degrade.
Conclusion
By analytically transforming non-uniform geometric signals into the spectral domain, this work provides a robust, efficient, and unified framework for applying standard CNNs to complex 3D data. The approach eliminates aliasing errors inherent in voxelization and allows researchers to leverage powerful existing CNN architectures for tasks ranging from shape classification to high-fidelity surface reconstruction.
Summary updated