Adaptive Surface Reconstruction with Multiscale Convolutional Kernels

Benjamin Ummenhofer, Vladlen Koltun

Introduction

3D surface reconstruction—the process of converting sparse point clouds into continuous digital meshes—is a fundamental challenge in computer vision. Traditional approaches often rely on volumetric fusion, which partitions space into a regular 3D grid. However, these methods suffer from "cubic growth": doubling the resolution increases the voxel count eightfold, making high-resolution reconstruction of large-scale scenes computationally prohibitive. While adaptive structures like octrees can mitigate memory usage, applying standard Convolutional Neural Networks (CNNs) to these irregular grids has historically been inefficient.

Methodology

This paper introduces a novel convolutional operator specifically designed for adaptive, face-balanced octrees. The core innovation lies in multiscale kernels, where convolutional elements possess both relative spatial locations and relative scales. This allows a single convolution to integrate features from neighboring cells that may reside at different resolutions within the octree structure.

The proposed reconstruction pipeline, illustrating the stages of octree generation, adaptive grid convolutions, and distance field decoding.

The system utilizes a U-Net-inspired architecture to predict a hybrid distance field. By combining Signed Distance Fields (SDF) for surface definition with Unsigned Distance Fields (UDF) to constrain reconstruction to regions with actual data, the method avoids common "ballooning" artifacts. Finally, an MLP decodes these features into precise distance values, and a dual contouring algorithm extracts the final triangle mesh.

Experimental Results

The model was trained on synthetic data from the Thingi10K and Scan the World datasets and evaluated on real-world benchmarks, including Tanks and Temples and the massive Citywall dataset containing 359 million points.

Reconstruction of the large-scale Citywall dataset, demonstrating the system's ability to capture fine geometric details within a massive, sparse environment.

Key findings include:

  • Superior Accuracy: On the Tanks and Temples benchmark, the method achieved a mean F-score of 45.15, significantly outperforming classical Poisson Surface Reconstruction (40.09) and previous learning-based approaches like Points2Surf (23.13).
  • Zero-Shot Generalization: Despite synthetic training, the model generalizes to complex real-world scenes without fine-tuning, suggesting the kernels learn fundamental geometric priors rather than memorizing specific shapes.
  • High Efficiency: The method reconstructed the Citywall dataset in 117 minutes, more than twice as fast as the leading GDMR baseline (274 minutes).

Conclusion

By leveraging multiscale convolutional kernels on adaptive grids, this approach effectively overcomes the scalability limitations of traditional 3D reconstruction. The results demonstrate a significant leap in both fidelity and speed for large-scale scene processing. Future research may extend these multiscale operators to other 3D tasks, such as semantic segmentation and object detection in expansive environments.

Summary updated

Abstract

We propose generalized convolutional kernels for 3D reconstruction with ConvNets from point clouds. Our method uses multiscale convolutional kernels that can be applied to adaptive grids as generated with octrees. In addition to standard kernels in which each element has a distinct spatial location relative to the center, our elements have a distinct relative location as well as a relative scale level. Making our kernels span multiple resolutions allows us to apply ConvNets to adaptive grids for large problem sizes where the input data is sparse but the entire domain needs to be processed. Our ConvNet architecture can predict the signed and unsigned distance fields for large data sets with millions of input points and is faster and more accurate than classic energy minimization or recent learning approaches. We demonstrate this in a zero-shot setting where we only train on synthetic data and evaluate on the Tanks and Temples dataset of real-world large-scale 3D scenes.
BibTeX
@inproceedings{ummenhofer2021adaptive,
  author = {Benjamin Ummenhofer and Vladlen Koltun},
  title = {{Adaptive Surface Reconstruction with Multiscale Convolutional Kernels}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00560},
}
Figure