Enhancing Multi-View Stereo with Contrastive Matching and Weighted Focal Loss

Yikang Ding, Zhenyang Li, Dihe Huang, Zhiheng Li, Kai Zhang

TL;DR

This paper introduces two novel loss functions—Contrastive Matching Loss (CML) and Weighted Focal Loss (WFL)—to significantly boost the performance of Multi-view Stereo (MVS) networks. By refining how features are matched across views and intelligently weighting training samples based on confidence, the authors achieve state-of-the-art 3D reconstruction results on major benchmarks without increasing the computational cost during inference.

What Problem Does This Paper Address?

Multi-view Stereo (MVS) is the process of reconstructing a dense 3D model of a scene from a collection of 2D images. While deep learning has pushed MVS forward, two major hurdles remain:

  1. Textureless Regions: It is notoriously difficult to match pixels between images when surfaces are uniform (like a white wall), leading to "holes" or noise in 3D models.
  2. Computational Overhead: Many recent attempts to fix these issues involve adding complex modules like Transformers or deformable CNNs. While effective, these make the models slower and more memory-hungry.

The authors aim to improve reconstruction quality by changing how the models are trained, rather than just making the architectures bigger.

Methods

The researchers propose a "plug-and-play" training strategy that can be applied to existing cost-volume-based MVS networks (they use CasMVSNet as their baseline).

1. Contrastive Matching Loss (CML)

The authors treat MVS as a feature-matching problem. For a specific point in a reference image, the model looks along a "depth line" in a source image to find the matching pixel.

  • The Innovation: They apply contrastive learning here. The feature at the ground-truth depth is treated as a positive sample, while features at all other depth hypotheses are negative samples.
  • The Goal: The loss forces the network to make the reference feature more similar to the "correct" depth feature and less similar to the "incorrect" ones, sharpening the matching capability.

2. Weighted Focal Loss (WFL)

MVS can also be viewed as a classification task: for every pixel, the model must "classify" which depth plane is the correct one.

  • The Innovation: Standard models often use L1 loss (measuring distance). This paper uses Focal Loss but adds a confidence weight.
  • The Insight: Pixels in background or occluded areas often have low confidence and act as "noise" during training. WFL uses the model's own predicted confidence to down-weight these unreliable pixels, allowing the network to focus its "attention" on the more important foreground regions.

Datasets

The method was evaluated on three gold-standard MVS datasets:

  • DTU: An indoor dataset captured in controlled laboratory conditions.
  • Tanks and Temples: A public benchmark featuring large-scale outdoor scenes in realistic conditions.
  • BlendedMVS: A large-scale dataset containing a diverse mix of objects and scenes, used to demonstrate generalizability.

Key Insights and Ideas

  • Training vs. Architecture: The most powerful takeaway is that you don't always need a more complex model to get better results. By refining the loss functions (the "incentive structure" of the AI), you can extract more performance from existing, efficient architectures.
  • MVS as Classification: Shifting the perspective of MVS from simple regression (predicting a number) to classification (choosing the right bin) allows the use of Focal Loss, which is better at handling "hard" vs "easy" examples.
  • Uncertainty as a Signal: Instead of just trying to minimize uncertainty, the authors use it as a filter. By acknowledging that the model knows when it is guessing (low confidence), they prevent those guesses from corrupting the learning process for the rest of the scene.

Key Results

The authors integrated their losses into CasMVSNet and saw immediate improvements:

  • DTU Dataset: They achieved an "Overall" score of 0.335mm (lower is better), outperforming the baseline CasMVSNet (0.355mm) and several other complex models like PatchmatchNet and UCS-Net.
  • Tanks and Temples: On the "Intermediate" set, they achieved a mean F-score of 61.75, a significant jump of +4.91 over the baseline.
  • BlendedMVS: The endpoint error (EPE) dropped from 1.43 to 1.08, showing much cleaner depth maps.
  • Efficiency: Because these are loss functions used only during training, the model's speed and memory usage during actual use (inference) remain identical to the baseline.

Limitations and Future Work

The paper focuses primarily on improving the training of cost-volume-based architectures. While the results are impressive, the authors do not deeply explore how these losses would perform on non-volumetric MVS approaches (like point-based methods). Additionally, the reliance on ground-truth depth for the Contrastive Matching Loss means the quality of training is still tethered to the availability of high-quality labeled data. Future work might involve exploring self-supervised versions of these losses to remove the need for ground-truth depth.

Summary updated

Abstract

Learning-based multi-view stereo (MVS) methods have made impressive progress and surpassed traditional methods in recent years. However, their accuracy and completeness are still struggling. In this paper, we propose a new method to enhance the performance of existing networks inspired by contrastive learning and feature matching. First, we propose a Contrast Matching Loss (CML), which treats the correct matching points in depth-dimension as positive sample and other points as negative samples, and computes the contrastive loss based on the similarity of features. We further propose a Weighted Focal Loss (WFL) for better classification capability, which weakens the contribution of low-confidence pixels in unimportant areas to the loss according to predicted confidence. Extensive experiments performed on DTU, Tanks and Temples and BlendedMVS datasets show our method achieves state-of-the-art performance and significant improvement over baseline network.
BibTeX
@inproceedings{ding2022enhancing,
  author = {Yikang Ding and Zhenyang Li and Dihe Huang and Zhiheng Li and Kai Zhang},
  title = {{Enhancing Multi-View Stereo with Contrastive Matching and Weighted Focal Loss}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=HjTJzx4_gU},
}

Loading related content...

Figure