TL;DR
This paper introduces WSCLoc, a novel system that significantly improves camera relocalization accuracy in sparse-view scenarios, a major challenge for existing weakly-supervised methods. WSCLoc achieves this by first generating accurate pose labels using a custom Neural Radiance Field (NeRF) model (WFT-NeRF) and then using these labels to train a relocalization model (WFT-Pose) with enhanced geometric constraints.
What Problem Does This Paper Address?
Accurate camera relocalization is crucial for applications like autonomous driving and augmented reality. Traditional deep learning methods for this task require extensive, manually annotated datasets with precise 6-DoF (six degrees of freedom) camera poses. Obtaining these labels is costly and time-consuming, often requiring specialized sensors like LiDAR or RGB-D cameras. While weakly-supervised methods have emerged to reduce this labeling burden, their performance drastically degrades when dealing with sparse camera views. This sparsity, common in real-world scenarios where data collection might be limited or uneven, leads to issues like scale drift and image distortions, making accurate pose estimation extremely difficult. WSCLoc aims to bridge this gap by enabling robust and accurate relocalization even with very few input images.
Methods
WSCLoc operates in two main stages:
- WFT-NeRF: Pose Label Generation: This stage tackles the core challenge of generating reliable pose labels from sparse views.
- Neural Radiance Fields (NeRFs): The system leverages NeRFs, which learn a scene's volumetric representation from images.
- Weakly-Supervised Free-Trajectory (WFT) NeRF: A custom NeRF model is developed to co-optimize image reconstruction and initial pose information.
- Explicit Scale Constraint (sim(3) optimization): Instead of standard SE(3) (rotation and translation), the system uses sim(3) optimization, which explicitly incorporates a scale factor. This is crucial for mitigating scale drift, a common problem with monocular cameras.
- Explicit Time Encoding: To handle image distortions like motion blur and rolling shutter effects inherent in sparse-view videos, time indices are explicitly encoded. This helps to smooth out the camera's impact on object appearance.
- Temporal Information: Incorporating temporal information from consecutive frames stabilizes the learning process.
- WFT-Pose: Relocalization Model Training: This stage uses the labels generated by WFT-NeRF to train a target relocalization model.
- Customizable Framework: WSCLoc is designed to be integrated with various existing deep learning relocalization models (e.g., DFNet, PoseNet).
- Time-Encoding Based Random View Synthesis (TE-based RVS): This data augmentation technique generates synthetic training data by perturbing poses and incorporating time encoding, further improving robustness and generalization.
- Inter-Frame Geometric Constraints: The system enforces constraints based on pose, depth, and RGB information between consecutive frames. This includes a point cloud loss and a KL divergence loss to ensure consistency.
Datasets
The paper evaluates WSCLoc on two publicly available datasets:
- 7-Scenes Dataset: An indoor dataset with complex trajectories, featuring RGB images, depth maps, and ground truth camera poses.
- Cambridge Landmarks Dataset: A large-scale outdoor dataset with six scenes, providing RGB images, visual reconstructions, and 6-DoF ground truth poses. (Experiments were conducted on five scenes due to issues with one scene's reconstruction).
Key Insights and Ideas
- sim(3) is Key for Sparse-View Scale: Explicitly optimizing for similarity transformations (sim(3)) rather than just rigid transformations (SE(3)) is critical for overcoming scale drift in monocular, sparse-view relocalization.
- Time is Not Just a Sequence Number: Encoding time information directly into the NeRF model is a powerful way to combat image distortions caused by camera motion, leading to cleaner renderings and more accurate pose labels.
- Two-Stage Approach for Robustness: Separating pose label generation (WFT-NeRF) from relocalization model training (WFT-Pose) allows each stage to specialize and leverage complementary techniques, leading to superior overall performance.
- WSCLoc as a General Enhancement Framework: The system's ability to be integrated with existing relocalization models highlights its versatility and potential to improve a wide range of existing methods.
Key Results
- Superior Sparse-View Performance: WSCLoc consistently outperforms baseline methods (like DFNet and PoseNet) in sparse-view scenarios (20% and 10% of images) on both indoor (7-Scenes) and outdoor (Cambridge) datasets.
- Comparable to Dense-View: In dense-view scenarios, WSCLoc achieves accuracy comparable to state-of-the-art methods, demonstrating its robustness.
- Significant Improvement: On the Cambridge dataset with 10% of images, WSCLoc achieved improvements of up to 51% in median translation error and 33% in median rotation error compared to the baseline.
- Accurate Pose Label Generation: The WFT-NeRF stage generates pose labels that are significantly closer to dense-view SfM ground truth than labels generated by SfM itself in sparse conditions.
Limitations and Future Work
- Feature Matching Sensitivity: The paper notes that in highly sparse-view scenarios, the inter-frame geometric constraints might be less effective due to the sparsity of matched feature points. Future work could explore deep learning-based feature matching to improve this.
- Computational Cost: While not explicitly detailed as a limitation, training NeRF-based models can be computationally intensive.
- Dataset Specifics: The study acknowledges potential issues with specific scenes in datasets (e.g., Cambridge's STREET scene), which is a common challenge in benchmarking.
The authors plan to make their code publicly available, which will facilitate further research and adoption of their WSCLoc system.
Summary updated