Multi-view Stereo 3D Edge Reconstruction

Andrea Bignoli, Andrea Romanoni, Matteo Matteucci

TL;DR

The authors introduce EdgeGraph3D, a system that reconstructs both straight and curved 3D edges from unordered image sequences. By representing 2D edges as graphs and bootstrapping from sparse Structure-from-Motion (SfM) data, the method produces more accurate and complete 3D meshes, especially in scenes with challenging lighting or reflective surfaces.

What Problem Does This Paper Address?

Traditional Multi-View Stereo (MVS) algorithms typically rely on photometric consistency—matching the appearance of pixels across views—to build dense 3D models. This approach often fails on textureless surfaces, reflective materials, or under drastic lighting changes.

While 3D edges (the "skeletons" of objects) are much more robust to these conditions, previous research has been limited. Most existing methods only reconstruct straight line segments or require continuous video sequences to track edges. Reconstructing curved edges from an unordered set of photos is significantly harder because you cannot easily match a point on a curve across different perspectives without a temporal "trail."

Methods

The authors propose a pipeline that moves from 2D image gradients to a refined 3D wireframe:

  1. 2D Edge-Graph Generation: Instead of treating edges as a simple mask of pixels, they convert them into an undirected graph. Nodes represent 2D points, and connections represent the edge flow. They apply a "polyline smoothing" step (a variant of the Douglas-Peucker algorithm) to achieve sub-pixel accuracy and filter out "non-structural" noise by ranking edges based on their "regular length."
  2. Potential Edge Correspondence (PEC): To match edges across views, the system looks for 2D polylines that share a significant number of nearby 3D points (from an initial sparse SfM cloud). They use a community detection algorithm to group these related polylines across multiple images.
  3. Potential Edge-Point Correspondence (PEPC): Within these matched edges, the system identifies specific corresponding points. They use two strategies:
  4. From SfM points: Searching the neighborhood of known 3D points.
  5. From PECs: Intersecting matched polylines with epipolar lines.
  6. Validation and Expansion: A combinatorial selection process ensures that a 3D point is only generated if it can be triangulated across at least three views with low reprojection error. Once a "seed" 3D point is found, the algorithm "walks" along the 2D polylines in all views simultaneously to grow the 3D edge.

Datasets

The researchers evaluated their system on two standard computer vision benchmarks:

  • EPFL fountain-p11: A classic sequence used to test high-resolution MVS.
  • DTU Dataset: A collection of 6 sequences featuring various objects. This dataset is particularly useful for testing robustness against reflections (e.g., metallic cans) and complex geometries.

Key Insights and Ideas

  • Edges as Anchors: 3D edges provide a compact, salient representation of a scene. Because they correspond to physical boundaries or high-contrast features, they are less affected by the "shimmer" of reflections that confuses standard photometric methods.
  • Bootstrapping from SfM: The "aha moment" is using the existing (but sparse) SfM point cloud not just as a result, but as a spatial guide to solve the matching ambiguity of curved edges in unordered photos.
  • Graph-Based Filtering: By treating edges as a graph, the authors can calculate "structural significance." This allows them to ignore the "salt and pepper" edge noise often found in high-texture areas and focus on the primary geometry of the object.

Key Results

  • Increased Density: EdgeGraph3D significantly increased the number of reconstructed points compared to standard SfM (OpenMVG). For example, in the DTU-023 dataset, points increased from ~9,600 to over ~97,000.
  • Improved Mesh Accuracy: When these 3D edges were used to build a mesh (using Delaunay-based carving), the resulting models were more accurate than those built from points alone. The authors reported an average reduction in mesh error (RMSE) of roughly 30% compared to OpenMVG.
  • Superior to Line-Only Methods: Compared to Line3D++ (a state-of-the-art line segment reconstructor), EdgeGraph3D reduced mesh error by about 17%, largely because it could capture the curved surfaces that line-based methods ignore.
  • Robustness: The system successfully reconstructed structural elements on highly reflective metallic cans (DTU-098) where standard SfM pipelines failed completely.

Limitations and Future Work

  • Computational Cost: The process is relatively slow. The authors noted an average execution time of 13 minutes per dataset, which might be a bottleneck for real-time applications.
  • Dependency on SfM: The method relies on an initial set of SfM points to bootstrap the edge matching. If the initial SfM is extremely poor, the edge reconstruction will likely struggle.
  • Future Directions: The authors plan to integrate these 3D edges directly into the Bundle Adjustment process (the optimization step that refines camera positions) and explore "Constrained 3D Delaunay triangulation" to better embed these edges into the final surface meshes.

The source code for the project is available on GitHub.

Summary updated

Abstract

This paper presents a novel method for the reconstruction of 3D edges in multi-view stereo scenarios. Previous research in the field typically relied on video sequences and limited the reconstruction process to either straight linesegments, or edge-points, i.e., 3D points that correspond to image edges. We instead propose a system, denoted as EdgeGraph3D, able to recover both straight and curved 3D edges from an unordered image sequence. A second contribution of this work is a graph-based representation for 2D edges that allows the identification of the most structurally significant edges detected in an image. We integrate Edge-Graph3D in a multi-view stereo reconstruction pipeline and analyze the benefits provided by 3D edges to the accuracy of the recovered surfaces. We evaluate the effectiveness of our approach on multiple datasets from two different collections in the multi-view stereo literature. Experimental results demonstrate the ability of EdgeGraph3D to work in presence of strong illumination changes and reflections, which are usually detrimental to the effectiveness of classical photometric reconstruction systems.
BibTeX
@inproceedings{bignoli2018multi,
  author = {Andrea Bignoli and Andrea Romanoni and Matteo Matteucci},
  title = {{Multi-view Stereo 3D Edge Reconstruction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=Q9fNtv4sBF},
}

Loading related content...

Figure