TL;DR
The authors introduce EdgeGraph3D, a system that reconstructs both straight and curved 3D edges from unordered image sequences. By representing 2D edges as graphs and bootstrapping from sparse Structure-from-Motion (SfM) data, the method produces more accurate and complete 3D meshes, especially in scenes with challenging lighting or reflective surfaces.
What Problem Does This Paper Address?
Traditional Multi-View Stereo (MVS) algorithms typically rely on photometric consistency—matching the appearance of pixels across views—to build dense 3D models. This approach often fails on textureless surfaces, reflective materials, or under drastic lighting changes.
While 3D edges (the "skeletons" of objects) are much more robust to these conditions, previous research has been limited. Most existing methods only reconstruct straight line segments or require continuous video sequences to track edges. Reconstructing curved edges from an unordered set of photos is significantly harder because you cannot easily match a point on a curve across different perspectives without a temporal "trail."
Methods
The authors propose a pipeline that moves from 2D image gradients to a refined 3D wireframe:
- 2D Edge-Graph Generation: Instead of treating edges as a simple mask of pixels, they convert them into an undirected graph. Nodes represent 2D points, and connections represent the edge flow. They apply a "polyline smoothing" step (a variant of the Douglas-Peucker algorithm) to achieve sub-pixel accuracy and filter out "non-structural" noise by ranking edges based on their "regular length."
- Potential Edge Correspondence (PEC): To match edges across views, the system looks for 2D polylines that share a significant number of nearby 3D points (from an initial sparse SfM cloud). They use a community detection algorithm to group these related polylines across multiple images.
- Potential Edge-Point Correspondence (PEPC): Within these matched edges, the system identifies specific corresponding points. They use two strategies:
- From SfM points: Searching the neighborhood of known 3D points.
- From PECs: Intersecting matched polylines with epipolar lines.
- Validation and Expansion: A combinatorial selection process ensures that a 3D point is only generated if it can be triangulated across at least three views with low reprojection error. Once a "seed" 3D point is found, the algorithm "walks" along the 2D polylines in all views simultaneously to grow the 3D edge.
Datasets
The researchers evaluated their system on two standard computer vision benchmarks:
- EPFL fountain-p11: A classic sequence used to test high-resolution MVS.
- DTU Dataset: A collection of 6 sequences featuring various objects. This dataset is particularly useful for testing robustness against reflections (e.g., metallic cans) and complex geometries.
Key Insights and Ideas
- Edges as Anchors: 3D edges provide a compact, salient representation of a scene. Because they correspond to physical boundaries or high-contrast features, they are less affected by the "shimmer" of reflections that confuses standard photometric methods.
- Bootstrapping from SfM: The "aha moment" is using the existing (but sparse) SfM point cloud not just as a result, but as a spatial guide to solve the matching ambiguity of curved edges in unordered photos.
- Graph-Based Filtering: By treating edges as a graph, the authors can calculate "structural significance." This allows them to ignore the "salt and pepper" edge noise often found in high-texture areas and focus on the primary geometry of the object.
Key Results
- Increased Density: EdgeGraph3D significantly increased the number of reconstructed points compared to standard SfM (OpenMVG). For example, in the
DTU-023dataset, points increased from ~9,600 to over ~97,000. - Improved Mesh Accuracy: When these 3D edges were used to build a mesh (using Delaunay-based carving), the resulting models were more accurate than those built from points alone. The authors reported an average reduction in mesh error (RMSE) of roughly 30% compared to OpenMVG.
- Superior to Line-Only Methods: Compared to Line3D++ (a state-of-the-art line segment reconstructor), EdgeGraph3D reduced mesh error by about 17%, largely because it could capture the curved surfaces that line-based methods ignore.
- Robustness: The system successfully reconstructed structural elements on highly reflective metallic cans (DTU-098) where standard SfM pipelines failed completely.
Limitations and Future Work
- Computational Cost: The process is relatively slow. The authors noted an average execution time of 13 minutes per dataset, which might be a bottleneck for real-time applications.
- Dependency on SfM: The method relies on an initial set of SfM points to bootstrap the edge matching. If the initial SfM is extremely poor, the edge reconstruction will likely struggle.
- Future Directions: The authors plan to integrate these 3D edges directly into the Bundle Adjustment process (the optimization step that refines camera positions) and explore "Constrained 3D Delaunay triangulation" to better embed these edges into the final surface meshes.
The source code for the project is available on GitHub.
Summary updated