Introduction
Multi-View Stereo (MVS) aims to reconstruct 3D geometry from a set of 2D images. PatchMatch-based MVS has emerged as a leading paradigm due to its efficiency in searching depth hypotheses. However, traditional PatchMatch methods often suffer from "boundary bleeding"—where depth values from the foreground leak into the background—and "blind refinement," where local searches fail to escape incorrect local minima in complex scenes.
As shown in Figure 1, the authors identify that fixed, heuristic sampling is the primary bottleneck in current architectures. They propose DS-PMNet, which introduces a learnable DeformSampler to adaptively search the depth space based on scene geometry and matching uncertainty.
The DeformSampler
The core innovation of DS-PMNet is the DeformSampler, which replaces rigid sampling with two learnable modules integrated into an end-to-end network:
1. Plane Indicator ($P_\theta$)
Traditional methods sample from a fixed spatial neighborhood. The Plane Indicator instead learns a "plane flow field" that identifies which neighboring pixels belong to the same physical surface. By predicting offsets to these "reasonable" neighbors, the model ensures that depth propagation is structure-aware. This effectively prevents the noise and "bleeding" typically found at object boundaries.
2. Probabilistic Matcher ($M_\theta$)
To address the multi-modal nature of depth matching (where multiple depths may seem plausible due to reflections or occlusions), the Probabilistic Matcher models the depth probability along a ray using a Laplace Mixture Model (LMM). This allows the network to maintain multiple depth candidates and adjust its search range based on confidence—narrowing for precision and expanding when uncertain.
Figure 6 visualizes these components in action. The offset maps (a) show how the model learns to "look" along surfaces to find relevant neighbors, while the score map (c) reflects the refined matching probability across the scene.
Experimental Results
The authors evaluated DS-PMNet on several industry-standard benchmarks:
- DTU Dataset: Achieved a state-of-the-art overall score of 0.290mm, outperforming both traditional and recent learning-based leaders.
- Tanks & Temples: Ranked 1st among PatchMatch-based methods on the "Intermediate" set, demonstrating robust generalization to large-scale outdoor environments.
- BlendedMVS: Used for fine-tuning to handle diverse real-world and synthetic scenarios.
Qualitatively, the model excels at recovering thin structures (like chair legs) and sharp edges that often become blurry in other reconstructions, as seen in the comparison in Figure 1(d).
Conclusion
DS-PMNet shifts the focus of MVS research from evaluation and regularization back to the fundamental step of sampling. By making the sampling process deformable and distribution-sensitive, the model achieves superior accuracy and robustness. Future work may further explore the "perturbation stage" of depth refinement to enhance probabilistic modeling in even more challenging environments.
Summary updated