3D Shape Reconstruction From Images in the Frequency Domain

Weichao Shen, Yunde Jia, Yuwei WuCVPR 2019 · OpenReview

Abstract

Reconstructing high-resolution volumetric 3D shapes from 2D images is a significant challenge in computer vision due to the cubic growth of computational costs. We propose a Fourier-based method that reconstructs 3D shapes in a 2D space by predicting slices in the frequency domain. Central to our approach is the "thickness map"—a 2D spatial projection of the 3D shape that bridges the domain gap between images and frequency slices. By leveraging the Fourier slice projection theorem, our method transfers 3D reconstruction into 2D space, drastically reducing memory and time requirements. Experiments on the ShapeNet dataset show that our method achieves competitive accuracy and superior efficiency, scaling successfully to $512^3$ resolutions.

Introduction: The Curse of Dimensionality

The primary hurdle in 3D reconstruction is the "curse of dimensionality." Standard volumetric methods represent shapes as 3D voxel grids, which scale cubically ($N^3$). Doubling the resolution requires eight times the memory and computation, making high-resolution outputs (like $256^3$ or $512^3$) nearly impossible for standard consumer hardware. While sparse structures like Octrees or point clouds offer some relief, they often necessitate complex, custom network architectures. Furthermore, the "domain gap" between a 2D RGB image and a 3D spatial volume makes direct mapping difficult for neural networks to learn.

Methodology

Our proposed pipeline translates 3D reconstruction into a series of manageable 2D prediction tasks.

The Thickness Map

Instead of predicting voxels directly, the network predicts "thickness maps." These are 2D projections where each pixel's intensity represents the total thickness of the 3D object along that ray. This acts as a structural bridge; it resembles a standard image, making it easier for Convolutional Neural Networks (CNNs) to learn, yet it encodes vital 3D structural data.

Fourier Slice Theorem and Pipeline

The core mathematical engine is the Fourier Slice Theorem, which states that a 2D projection of a 3D object corresponds exactly to a 2D slice passing through the origin of that object’s 3D Fourier transform.

Overview of the proposed Fourier-based reconstruction framework, from input image to 3D voxel grid.

The process follows these steps:

  1. Dual-Path Prediction: To capture both global shape and fine details, we use two sub-networks: one for the silhouette (global structure) and one for the edges (local details). These are fused to create the final thickness map.
  2. Frequency Assembly: Predicted thickness maps are transformed via 2D Fast Fourier Transform (FFT) into frequency slices.
  3. 3D Reconstruction: These slices are inserted into a 3D Fourier space. An inverse 3D FFT is then applied to the assembled volume to produce the final 3D voxel grid.

Key Insights: Sparsity and Symmetry

A critical observation in this work is that 3D shapes are inherently sparse in the frequency domain. We found that a full set of slices is not required for high-fidelity reconstruction.

Visualizing the impact of slice count on reconstruction fidelity and IoU.

As shown in Figure 2, using just three slices along each coordinate axis can achieve over 93% reconstruction accuracy (Intersection over Union, IoU). Additionally, by extending the Fourier slice theorem, the authors proved that high-frequency slices can be calculated from low-frequency ones, effectively halving the number of slices the network needs to predict.

Experimental Results

We evaluated our method on the ShapeNet dataset, using a subset of approximately 50,000 models across 13 categories, voxelized at resolutions up to $512^3$.

Computational Efficiency

The frequency-domain approach offers massive gains in efficiency. At $256^3$ resolution, our method uses only 1.93 GB of GPU memory, compared to 9.7 GB for a standard dense network. In terms of speed, the iteration time at $256^3$ is roughly 7x faster than dense networks (0.47s vs 3.22s) and even outperforms specialized Octree-based methods like OGN.

Accuracy and Scalability

On the ShapeNet benchmark at $32^3$ resolution, the method achieved a mean IoU of 0.605, outperforming R2N2 (0.560) and OGN (0.596). Crucially, the method successfully scales to $512^3$ resolution—a feat that typically causes standard dense networks to crash due to memory exhaustion.

Conclusion

By moving 3D reconstruction into the frequency domain, we bypass the cubic scaling laws that plague traditional voxel-based methods. The use of thickness maps provides an effective bridge between 2D images and 3D volumes. While current work relies on fixed projection views, future iterations could explore arbitrary angles or integration with implicit functions to further enhance geometric detail.

Summary updated

Abstract

Reconstructing the high-resolution volumetric 3D shape from images is challenging due to the cubic growth of computational cost. In this paper, we propose a Fourier-based method that reconstructs a 3D shape from images in a 2D space by predicting slices in the frequency domain. According to the Fourier slice projection theorem, we introduce a thickness map to bridge the domain gap between images in the spatial domain and slices in the frequency domain. The thickness map is the 2D spatial projection of the 3D shape, which is easily predicted from the input image by a general convolutional neural network. Each slice in the frequency domain is the Fourier transform of the corresponding thickness map. All slices constitute a 3D descriptor and the 3D shape is the inverse Fourier transform of the descriptor. Using slices in the frequency domain, our method can transfer the 3D shape reconstruction from the 3D space into the 2D space, which significantly reduces the computational cost. The experiment results on the ShapeNet dataset demonstrate that our method achieves competitive reconstruction accuracy and computational efficiency compared with the state-of-the-art reconstruction methods.
BibTeX
@inproceedings{shen2019d,
  author = {Weichao Shen and Yunde Jia and Yuwei Wu},
  title = {{3D Shape Reconstruction From Images in the Frequency Domain}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=RkAaztQ2xAH},
}
Figure