Abstract
Learning-based Multi-View Stereo (MVS) methods often struggle with generalization to unseen scenes due to fixed parameters at inference time. We present MVS-TTA, a novel framework that enables models to adapt to specific scenes at test time without requiring ground-truth labels. By combining meta-learning with a self-supervised cross-view consistency loss, MVS-TTA "learns how to adapt," effectively bridging the gap between fast, data-driven deep learning and accurate, scene-specific optimization. Our approach is model-agnostic and demonstrates consistent performance gains across various state-of-the-art architectures.
Introduction
Multi-View Stereo (MVS) is the process of reconstructing 3D geometry from multiple 2D images. Currently, the field is divided into two primary paradigms: learning-based methods and optimization-based methods. Learning-based models are fast and leverage large datasets to learn powerful geometric priors, but their parameters are frozen at test time. This leads to a "generalization gap" when a scene differs from the training data (e.g., unique lighting or textureless surfaces). Conversely, optimization-based methods (such as NeRF) are highly accurate as they optimize for every scene, but they are computationally expensive and lack the broad knowledge of deep networks.
MVS-TTA aims to give learning-based models the flexibility of optimization-based methods while maintaining the efficiency of deep learning. It allows a model to adjust its parameters to the specific characteristics of the input scene during inference.
Methodology: Meta-Auxiliary Learning
The core of the MVS-TTA framework is a two-stage approach: Meta-Auxiliary Learning during training and Test-Time Adaptation during inference.
The Auxiliary Task
To adapt at test time without ground-truth labels, the framework utilizes Cross-view Photometric Consistency. The logic is that if the predicted depth for a reference image is correct, warping a neighboring "source" image into the reference's perspective should result in an image that matches the reference perfectly. This self-supervised signal serves as a proxy for optimization.
Meta-Learning for Adaptation
Simply optimizing for photometric consistency at test time can sometimes lead to "negative transfer," where the model's performance on the primary task (depth estimation) actually degrades. To prevent this, MVS-TTA employs a Model-Agnostic Meta-Learning (MAML) strategy.
As illustrated in the architecture diagram, the training process involves two loops:
- Inner Loop: The model performs a few gradient steps using the unsupervised photometric consistency loss ($L_{photo}$).
- Outer Loop: The model is then evaluated on the actual supervised depth task ($L_{pri}$).
This process trains the model's initial weights so that a few steps of unsupervised optimization at test time are guaranteed to improve final depth accuracy. The framework is designed to be a "wrapper" that can be applied to almost any existing MVS architecture with minimal changes.
Experiments and Datasets
The authors evaluated MVS-TTA on two industry-standard benchmarks:
- DTU: A high-quality indoor dataset used to test high-precision reconstruction.
- BlendedMVS: A large-scale synthetic dataset covering diverse outdoor scenes.
To test the model's robustness, the researchers performed cross-dataset generalization tests, training on synthetic data (BlendedMVS) and testing directly on real-world scenes (DTU) to observe how well the adaptation handles massive shifts in data distribution.
Key Results
The framework consistently improved every backbone model it was tested on, including MVSFormer++, GeoMVSNet, TransMVSNet, and CasMVSNet.
On the DTU dataset, MVS-TTA reduced the absolute relative error of the state-of-the-art MVSFormer++ from 0.91 to 0.84. In cross-dataset scenarios, the relative error for CasMVSNet dropped from 4.31 to 4.10. Qualitatively, the adaptation shows significant improvements in textureless regions, such as smooth walls, where frozen models typically fail to provide accurate geometric constraints.
Efficiency and Limitations
By utilizing meta-learning, the model only requires two gradient steps at test time to achieve significant gains, adding approximately 1 second of overhead per sample. However, the authors noted that excessive adaptation (e.g., 10 steps) can lead to overfitting on noisy photometric cues, causing performance to drop. Additionally, the current consistency loss assumes a static scene, which may limit its application in dynamic environments.
Conclusion
MVS-TTA provides a scalable solution to the generalization challenges of deep MVS models. By teaching models how to optimize themselves for new environments using self-supervision, it bridges the gap between fast data-driven priors and accurate scene-specific optimization.
Summary updated