Vis2Mesh: Efficient Mesh Reconstruction from Unstructured Point Clouds of Large Scenes with Learned Virtual View Visibility

Shuang Song, Zhaopeng Cui, Rongjun Qin

TL;DR

This paper introduces Vis2Mesh, a novel framework for reconstructing 3D surfaces from unstructured point clouds. It leverages learned virtual view visibility, combining deep learning for visibility prediction with traditional graph-cut methods for efficient and robust mesh generation, especially for large-scale scenes.

What Problem Does This Paper Address?

Accurate 3D surface reconstruction from point clouds is crucial for many applications like VR/AR, localization, and animation. While traditional methods like Delaunay triangulation and Poisson surface reconstruction exist, they often struggle with sparse or noisy point clouds and can be computationally expensive for large scenes. Existing learning-based methods, while powerful, can be scene-specific, computationally intensive, and require significant memory, limiting their scalability. The challenge is to create a reconstruction method that is robust, generalizable, scalable, and can recover fine geometric details from complex, unstructured point clouds.

Methods

Vis2Mesh tackles this by proposing a novel framework that integrates learned visibility with graph-cut based reconstruction. The core innovation lies in a three-step deep learning network (VDVNet) for predicting point visibility in virtual views. This network uses partial convolutions to handle sparse input data and incorporates a depth completion module to improve visibility prediction accuracy.

The process involves:

  1. Virtual View Sampling: Generating a large number of virtual camera poses around the point cloud to simulate diverse viewing conditions.
  2. Visibility Estimation Network (VDVNet):
    • Coarse VisNet: Predicts initial visibility from sparse depth maps.
  3. DepthCompNet: Fills in missing depth information to create a denser depth map.
  4. FineVisNet: Refines visibility prediction using both the coarse visibility and the completed dense depth map.
  5. Graph-Cut based Surface Reconstruction: The predicted visibility information from virtual views is used to guide a traditional graph-cut optimization process. A novel adaptive visibility weighting (AVW) term is introduced to suppress the influence of virtual rays with large incident angles, thereby preserving geometric details like creases and sharp features.

Datasets

The authors trained and evaluated their method on a variety of datasets, including:

  • BlendedMVS: Used for training the visibility estimation network.
  • COSEG, CONet, and BlendedMVS: Used for quantitative evaluation of reconstruction.
  • senseFly, KITTI, Mai City Dataset, Airborne LiDAR data of Columbus, Ohio, and Airborne LiDAR data of Toronto: Used for evaluating robustness and generalization on real-world data.

They also created synthetic data by sampling points from ground truth meshes and augmenting them with noise and outliers.

Key Insights and Ideas

  • Virtual View Visibility is Powerful: The paper demonstrates that leveraging visibility from a large number of virtual views, even if not physically captured, provides a richer and more robust signal for surface reconstruction than relying solely on actual sensor views, especially for complex geometries.
  • Depth Completion Enhances Visibility: Integrating a depth completion network within the visibility estimation pipeline significantly improves the accuracy of predicting which points are visible, leading to better surface reconstruction.
  • Adaptive Visibility Weighting Preserves Details: The proposed adaptive visibility weighting term is crucial for handling the unique characteristics of virtual views (e.g., large incident angles) and preventing the over-smoothing of fine geometric features like creases.
  • Generalization through Simple Tasks: By focusing the learning task on a simple binary classification (visibility prediction), the VDVNet achieves better generalization across different scene contexts and modalities compared to end-to-end reconstruction networks.

Key Results

  • Quantitative Performance: Vis2Mesh achieves state-of-the-art or competitive performance on various datasets, outperforming traditional methods like SPSR and learning-based methods like P2M, P2S, MIER, and CONet in terms of F-score and Chamfer distance, especially on large-scale outdoor scenes.
  • Robustness: The method shows robustness to noise and incomplete point clouds.
  • Scalability: The framework is designed to handle large scenes efficiently, with processing times comparable to SPSR for similar levels of detail.
  • Qualitative Results: Visualizations show that Vis2Mesh reconstructs smoother, more detailed, and complete surfaces, particularly for complex shapes and challenging areas, compared to other methods.

Limitations and Future Work

  • Limitations:
    • The performance can be affected by the density of the input point cloud, especially for very low-density data.
  • The method relies on the quality of the virtual view generators.
  • Future Work:
    • The authors suggest exploring more sophisticated virtual view sampling strategies.
  • Further investigation into handling extremely sparse point clouds could be beneficial.
  • Extending the framework to incorporate texture and color information for more complete scene reconstruction.

Summary updated

Abstract

We present a novel framework for mesh reconstruction from unstructured point clouds by taking advantage of the learned visibility of the 3D points in the virtual views and traditional graph-cut based mesh generation. Specifically, we first propose a three-step network that explicitly employs depth completion for visibility prediction. Then the visibility information of multiple views is aggregated to generate a 3D mesh model by solving an optimization problem considering visibility in which a novel adaptive visibility weighting in surface determination is also introduced to suppress line of sight with a large incident angle. Compared to other learning-based approaches, our pipeline only exercises the learning on a 2D binary classification task, i.e., points visible or not in a view, which is much more generalizable and practically more efficient and capable to deal with a large number of points. Experiments demonstrate that our method with favorable transferability and robustness, and achieve competing performances w.r.t. state-of-the-art learning-based approaches on small complex objects and outperforms on large indoor and outdoor scenes. Code is available at https://github.com/GDAOSU/vis2mesh.
BibTeX
@inproceedings{song2021vis,
  author = {Shuang Song and Zhaopeng Cui and Rongjun Qin},
  title = {{Vis2Mesh: Efficient Mesh Reconstruction from Unstructured Point Clouds of Large Scenes with Learned Virtual View Visibility}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00645},
}
Figure