Abstract
Reconstructing high-resolution volumetric 3D shapes from 2D images is a significant challenge in computer vision due to the cubic growth of computational costs. We propose a Fourier-based method that reconstructs 3D shapes in a 2D space by predicting slices in the frequency domain. Central to our approach is the "thickness map"—a 2D spatial projection of the 3D shape that bridges the domain gap between images and frequency slices. By leveraging the Fourier slice projection theorem, our method transfers 3D reconstruction into 2D space, drastically reducing memory and time requirements. Experiments on the ShapeNet dataset show that our method achieves competitive accuracy and superior efficiency, scaling successfully to $512^3$ resolutions.
Introduction: The Curse of Dimensionality
The primary hurdle in 3D reconstruction is the "curse of dimensionality." Standard volumetric methods represent shapes as 3D voxel grids, which scale cubically ($N^3$). Doubling the resolution requires eight times the memory and computation, making high-resolution outputs (like $256^3$ or $512^3$) nearly impossible for standard consumer hardware. While sparse structures like Octrees or point clouds offer some relief, they often necessitate complex, custom network architectures. Furthermore, the "domain gap" between a 2D RGB image and a 3D spatial volume makes direct mapping difficult for neural networks to learn.
Methodology
Our proposed pipeline translates 3D reconstruction into a series of manageable 2D prediction tasks.
The Thickness Map
Instead of predicting voxels directly, the network predicts "thickness maps." These are 2D projections where each pixel's intensity represents the total thickness of the 3D object along that ray. This acts as a structural bridge; it resembles a standard image, making it easier for Convolutional Neural Networks (CNNs) to learn, yet it encodes vital 3D structural data.
Fourier Slice Theorem and Pipeline
The core mathematical engine is the Fourier Slice Theorem, which states that a 2D projection of a 3D object corresponds exactly to a 2D slice passing through the origin of that object’s 3D Fourier transform.
The process follows these steps:
- Dual-Path Prediction: To capture both global shape and fine details, we use two sub-networks: one for the silhouette (global structure) and one for the edges (local details). These are fused to create the final thickness map.
- Frequency Assembly: Predicted thickness maps are transformed via 2D Fast Fourier Transform (FFT) into frequency slices.
- 3D Reconstruction: These slices are inserted into a 3D Fourier space. An inverse 3D FFT is then applied to the assembled volume to produce the final 3D voxel grid.
Key Insights: Sparsity and Symmetry
A critical observation in this work is that 3D shapes are inherently sparse in the frequency domain. We found that a full set of slices is not required for high-fidelity reconstruction.
As shown in Figure 2, using just three slices along each coordinate axis can achieve over 93% reconstruction accuracy (Intersection over Union, IoU). Additionally, by extending the Fourier slice theorem, the authors proved that high-frequency slices can be calculated from low-frequency ones, effectively halving the number of slices the network needs to predict.
Experimental Results
We evaluated our method on the ShapeNet dataset, using a subset of approximately 50,000 models across 13 categories, voxelized at resolutions up to $512^3$.
Computational Efficiency
The frequency-domain approach offers massive gains in efficiency. At $256^3$ resolution, our method uses only 1.93 GB of GPU memory, compared to 9.7 GB for a standard dense network. In terms of speed, the iteration time at $256^3$ is roughly 7x faster than dense networks (0.47s vs 3.22s) and even outperforms specialized Octree-based methods like OGN.
Accuracy and Scalability
On the ShapeNet benchmark at $32^3$ resolution, the method achieved a mean IoU of 0.605, outperforming R2N2 (0.560) and OGN (0.596). Crucially, the method successfully scales to $512^3$ resolution—a feat that typically causes standard dense networks to crash due to memory exhaustion.
Conclusion
By moving 3D reconstruction into the frequency domain, we bypass the cubic scaling laws that plague traditional voxel-based methods. The use of thickness maps provides an effective bridge between 2D images and 3D volumes. While current work relies on fixed projection views, future iterations could explore arbitrary angles or integration with implicit functions to further enhance geometric detail.
Summary updated