MVS-T: A Coarse-to-Fine Multi-View Stereo Network with Transformer for Low-Resolution Images 3D Reconstruction

Ruiming Jia, Xin Chen, Jiali Cui, Zhenghui Hu2022 · DOI

TL;DR

This paper introduces MVS-T, a novel coarse-to-fine multi-view stereo (MVS) network that leverages Transformers to improve 3D reconstruction accuracy for low-resolution images. By integrating Transformer's global context understanding with a multi-stage feature pyramid and cost volume refinement, MVS-T achieves state-of-the-art results on challenging datasets, offering a more complete and accurate 3D representation.

What Problem Does This Paper Address?

Reconstructing detailed 3D models from images is a fundamental task in computer vision with applications ranging from robotics to augmented reality. Traditional MVS methods often struggle with sparse point clouds and low accuracy, especially when dealing with low-resolution images that inherently lack fine-grained detail. While deep learning approaches have shown promise, they often require high-resolution inputs, which are computationally expensive and not always feasible for real-time applications like mobile robotics. This paper aims to bridge this gap by developing an effective and efficient MVS method specifically designed for low-resolution images, enabling better 3D reconstruction without the need for high-fidelity input data.

Methods

MVS-T employs a coarse-to-fine strategy to progressively refine depth estimation. The core innovation lies in the integration of Transformer architecture into the MVS pipeline.

  • Multi-stage Feature Pyramid (TFA): The network first constructs a feature pyramid using a three-stage aggregation module. This module is designed to capture both semantic and shallow spatial information across different scales, preserving crucial details that might be lost in standard downsampling.
  • Transformer for Global Context: At the coarsest resolution of the feature pyramid, a Transformer module is applied. This leverages the self-attention mechanism of Transformers to aggregate global context information from the image, enhancing the understanding of spatial relationships and internal feature correlations. This is a key departure from traditional CNN-based MVS methods that primarily focus on local features.
  • Image-like Feature Fusion: The output of the Transformer, which is in the form of patch embeddings, is then fused into an "image-like" feature representation. This module helps to reconstruct the spatial structure of the features, making them suitable for subsequent dense prediction tasks.
  • Coarse-to-Fine Cost Volume Regularization: Similar to established MVS methods, MVS-T builds a cost volume pyramid. This volume aggregates feature correspondences across different views and depths. The network then iteratively refines the depth map by regularizing this cost volume, starting from coarse estimates and progressively narrowing down the depth hypotheses.
  • Loss Function: The model is trained using an L1 loss function, which measures the absolute difference between the predicted depth map and the ground truth depth.

Datasets

  • DTU Dataset: This is the primary dataset used for training and evaluating the proposed MVS-T model. It is a well-known benchmark for MVS, containing 124 scenes captured under various lighting conditions, with structured light scans providing ground truth 3D geometry. The authors followed standard train/evaluation splits, using 79 scenes for training and 22 for evaluation.
  • BlendedMVS Dataset: This large-scale synthetic dataset was used to evaluate the generalization capabilities of MVS-T. While it contains a vast number of training samples, it lacks ground truth point clouds, making it suitable for qualitative visualization rather than quantitative evaluation.

Key Insights and Ideas

  • Transformers for MVS: The paper demonstrates that Transformers, with their inherent ability to capture global context, can significantly benefit MVS tasks, especially for low-resolution images where local information is scarce.
  • Coarse-to-Fine with Global Context: Combining a coarse-to-fine strategy with global context perception from Transformers is crucial for achieving accurate and complete 3D reconstructions from limited input data.
  • Feature Pyramid for Detail Preservation: The TFA module's design, which emphasizes preserving both semantic and shallow information, is vital for extracting meaningful features from low-resolution images.
  • Patch Size Optimization: The study highlights the importance of selecting an optimal patch size for Transformer's input, finding that a size of 4 yields the best results, balancing noise reduction and completeness.
  • Learnable Token's Role: The inclusion of a learnable classification token in the Transformer's fusion module was found to guide the model towards better overall dataset understanding and improved metrics.

Key Results

  • DTU Dataset Performance: MVS-T achieved the best overall metric on the DTU dataset among the compared methods, demonstrating a good balance between accuracy and completeness. Specifically, it achieved an accuracy of 0.9296 mm, completeness of 1.0120 mm, and an overall score of 0.9708 mm.
  • Comparison to State-of-the-Art: MVS-T outperformed several advanced MVS methods, including AA-RMVSNet, CasMVSNet, CVP-MVSNet, and AACVP-MVSNet, particularly in the overall metric, indicating its effectiveness for low-resolution inputs.
  • BlendedMVS Generalization: The model showed good generalization capabilities on the BlendedMVS dataset, producing reasonable 3D reconstructions for complex scenes even without fine-tuning.
  • Ablation Studies:
    • The complete model (TFA + Transformer) showed significant improvements over models with only TFA or Transformer.
  • Patch size = 4 was found to be optimal for Transformer input.
  • The learnable classification token improved performance.
  • Using 4 Transformer blocks yielded the best results.

Limitations and Future Work

  • Computational Overhead: The authors acknowledge that the computational complexity of Transformer blocks and cost volume construction is a limitation, especially for high-resolution 3D reconstruction.
  • High-Resolution Reconstruction: While MVS-T excels at low-resolution images, the paper did not explore 3D reconstruction at high resolutions due to computational constraints.
  • Future Work: The authors plan to design lightweight and compact networks to tackle MVS tasks on high-resolution images in the future. They also suggest further exploration of Transformer architectures and fusion strategies for improved performance.

Summary updated

BibTeX
@misc{jia2022mvs,
  author = {Ruiming Jia and Xin Chen and Jiali Cui and Zhenghui Hu},
  title = {{MVS-T: A Coarse-to-Fine Multi-View Stereo Network with Transformer for Low-Resolution Images 3D Reconstruction}},
  year = {2022},
  doi = {10.3390/s22197659},
}
Figure