Enhancing Multi-view Stereo with Contrastive Matching and Weighted Focal Loss

Yikang Ding, Zhenyang Li, Dihe Huang, Zhiheng Li, Kai Zhang

Abstract

Learning-based multi-view stereo (MVS) methods have made impressive progress and surpassed traditional methods in recent years. However, their accuracy and completeness are still struggling. In this paper, we propose a new method to enhance the performance of existing networks inspired by contrastive learning and feature matching. First, we propose a Contrast Matching Loss (CML), which treats the correct matching points in depth-dimension as positive sample and other points as negative samples, and computes the contrastive loss based on the similarity of features. We further propose a Weighted Focal Loss (WFL) for better classification capability, which weakens the contribution of low-confidence pixels in unimportant areas to the loss according to predicted confidence. Extensive experiments performed on DTU, Tanks and Temples and BlendedMVS datasets show our method achieves state-of-the-art performance and significant improvement over baseline network.

Introduction

Multi-view Stereo (MVS) is a fundamental task in computer vision, aimed at reconstructing dense 3D models from a collection of 2D images. While deep learning has significantly advanced the field, two primary challenges persist: the difficulty of matching features in textureless regions and the high computational cost associated with architectural improvements like Transformers or deformable convolutions.

This research introduces a "plug-and-play" training strategy designed to improve reconstruction quality without increasing the model's complexity or inference time. By introducing two novel loss functions—Contrastive Matching Loss (CML) and Weighted Focal Loss (WFL)—the authors demonstrate that significant performance gains can be achieved by refining the learning objective rather than the network architecture itself.

Methodology

The proposed method is integrated into a cost-volume-based MVS framework, using CasMVSNet as the baseline architecture. The core innovation lies in how the network is incentivized to learn feature representations and depth classifications.

Contrastive Matching Loss (CML)

Traditional MVS networks often struggle with ambiguous matches in regions with uniform textures. The authors address this by reframing MVS as a feature-matching problem through the lens of contrastive learning. For any given point in a reference image, the model evaluates a "depth line" in the source images to identify the corresponding pixel.

In the CML framework, the feature at the ground-truth depth is designated as a positive sample, while features at all other depth hypotheses are treated as negative samples. By minimizing this loss, the network is forced to maximize the similarity between the reference feature and the correct depth feature while simultaneously minimizing similarity with incorrect hypotheses. This sharpens the matching capability of the network, leading to more precise depth estimations.

Weighted Focal Loss (WFL)

Depth estimation in MVS can also be conceptualized as a classification task, where the model must select the correct depth plane from a set of discrete hypotheses. While standard regression losses (like L1) are common, they often fail to distinguish between "hard" and "easy" training examples.

To address this, the authors utilize Focal Loss, which is specifically designed to handle class imbalance and difficult samples. However, they enhance this further by introducing a confidence weight. The insight here is that pixels in background or occluded areas often yield low-confidence predictions that act as noise during training. The WFL uses the model's own predicted confidence to down-weight these unreliable pixels. This allows the network to focus its learning capacity on high-importance foreground regions, preventing noisy gradients from corrupting the training process.

Experimental Results

The effectiveness of the proposed losses was evaluated on three major benchmarks: DTU, Tanks and Temples, and BlendedMVS.

Performance Benchmarks

  • DTU Dataset: The method achieved an "Overall" score of 0.335mm (lower is better), representing a notable improvement over the baseline CasMVSNet (0.355mm) and outperforming more complex architectures such as PatchmatchNet and UCS-Net.
  • Tanks and Temples: On the "Intermediate" benchmark, the model achieved a mean F-score of 61.75, a significant increase of +4.91 over the baseline.
  • BlendedMVS: The endpoint error (EPE) was reduced from 1.43 to 1.08, resulting in substantially cleaner and more accurate depth maps.

Efficiency and Generalization

A key advantage of this approach is its efficiency. Because CML and WFL are applied only during the training phase, the model's inference speed and memory footprint remain identical to the baseline. This makes the method highly practical for real-world applications where computational resources are limited.

Conclusion

This paper demonstrates that refining the training objective through Contrastive Matching and Weighted Focal Loss can yield state-of-the-art results in 3D reconstruction. By treating MVS as both a contrastive matching and a weighted classification problem, the authors successfully mitigate the issues of textureless regions and training noise. Future work may explore self-supervised adaptations of these losses to reduce the dependence on high-quality ground-truth depth data.

Summary updated

Abstract

Learning-based multi-view stereo (MVS) methods have made impressive progress and surpassed traditional methods in recent years. However, their accuracy and completeness are still struggling. In this paper, we propose a new method to enhance the performance of existing networks inspired by contrastive learning and feature matching. First, we propose a Contrast Matching Loss (CML), which treats the correct matching points in depth-dimension as positive sample and other points as negative samples, and computes the contrastive loss based on the similarity of features. We further propose a Weighted Focal Loss (WFL) for better classification capability, which weakens the contribution of low-confidence pixels in unimportant areas to the loss according to predicted confidence. Extensive experiments performed on DTU, Tanks and Temples and BlendedMVS datasets show our method achieves state-of-the-art performance and significant improvement over baseline network.
BibTeX
@inproceedings{ding2022enhancing,
  author = {Yikang Ding and Zhenyang Li and Dihe Huang and Zhiheng Li and Kai Zhang},
  title = {{Enhancing Multi-view Stereo with Contrastive Matching and Weighted Focal Loss}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=HjTJzx4_gU},
}
Figure