Abstract
Despite the advancements in deep learning for camera relocalization tasks, obtaining ground truth pose labels required for the training process remains a costly endeavor. While current weakly supervised methods excel in lightweight label generation, their performance notably declines in scenarios with sparse views. In response to this challenge, we introduce WSCLoc, a system capable of being customized to various deep learning-based relocalization models to enhance their performance under weakly-supervised and sparse view conditions. This is realized with two stages. In the initial stage, WSCLoc employs a multilayer perceptron-based structure called WFT-NeRF to co-optimize image reconstruction quality and initial pose information. To ensure a stable learning process, we incorporate temporal information as input. Furthermore, instead of optimizing $SE(3)$, we opt for $\mathfrak{sim}(3)$ optimization to explicitly enforce a scale constraint. In the second stage, we co-optimize the pre-trained WFT-NeRF and WFT-Pose. This optimization is enhanced by Time-Encoding based Random View Synthesis and supervised by inter-frame geometric constraints that consider pose, depth, and RGB information. We validate our approaches on two publicly available datasets, one outdoor and one indoor. Our experimental results demonstrate that our weakly-supervised relocalization solutions achieve superior pose estimation accuracy in sparse-view scenarios, comparable to state-of-the-art camera relocalization methods.
Introduction
Camera relocalization—the ability of a system to determine its precise position and orientation (6-DoF pose) from a single image—is a cornerstone of autonomous driving and augmented reality. Traditionally, training these models requires "ground truth" labels, which are notoriously difficult to obtain. You either need expensive sensors (like LiDAR) or dense-view image sequences to run Structure-from-Motion (SfM) algorithms.
However, in the real world, we often only have "sparse views" (few images with large gaps between them). In these scenarios, traditional weakly-supervised methods fall apart. They suffer from scale drift, motion blur, and rolling shutter artifacts, all of which lead to noisy labels and poor relocalization accuracy. WSCLoc aims to bridge this gap, providing a way to train accurate models even when data is scarce and noisy.
Methodology
WSCLoc operates in two distinct stages to bridge the gap between sparse data and accurate relocalization.
Stage 1: WFT-NeRF (Pose Label Generation)
The first stage introduces Weakly-supervised Free-Trajectory NeRF (WFT-NeRF) to generate initial pose labels.
- Explicit Scale Constraint: Instead of optimizing in standard Euclidean space ($SE(3)$), the model utilizes similarity transformations ($\mathfrak{sim}(3)$). This allows the model to explicitly account for and correct scale drift, a common issue in monocular camera setups.
- Temporal Encoding: To handle motion blur and rolling shutter effects, discrete time indices are encoded for each image. This helps the NeRF "understand" that blur is a function of time and motion, leading to sharper reconstructions and more accurate poses.
Stage 2: WFT-Pose (Relocalization Model Training)
Once labels are generated, they are used to train a target relocalization model (such as PoseNet or DFNet).
- TE-based RVS: The pre-trained WFT-NeRF synthesizes new, "clean" training views (Random View Synthesis) with temporal encoding to ensure image sharpness.
- Inter-Frame Geometric Constraints: During training, the system enforces consistency between adjacent frames. It uses depth maps from the NeRF to project points between frames, minimizing a "point cloud loss" and using KL-divergence to ensure the distributions of the two frames align.
Experiments and Results
The researchers evaluated WSCLoc on two benchmarks: the indoor 7-Scenes Dataset and the outdoor Cambridge Landmarks Dataset. To test robustness, models were trained using only 20% and 10% of the original training images.
WSCLoc demonstrated significant performance gains in sparse-view scenarios:
- Outdoor Performance: On the Cambridge dataset (10% view scenario), WSCLoc achieved a 51% reduction in translation error and a 33% reduction in rotation error compared to the baseline (DFNet).
- Indoor Robustness: In the 7-Scenes dataset, WSCLoc often matched or outperformed models trained on 100% of the data, despite only using 10%. For example, in the "Fire" scene, it achieved a translation error of 0.17m with 10% data, compared to the baseline's 0.52m.
Discussion and Conclusion
A key insight of this work is the use of NeRF as a geometric optimizer rather than just a rendering tool. By co-optimizing scene reconstruction and camera poses, the NeRF acts as a "denoiser" for initial noisy SfM labels. Furthermore, treating motion blur as a temporal artifact allows the network to maintain accuracy even with handheld camera trajectories.
While WSCLoc is highly effective, its performance remains somewhat dependent on the quality of initial feature matching. In extremely sparse scenarios, traditional feature extractors (like SIFT) can struggle to find matches. Future work will explore integrating deep learning-based feature matching (like SuperPoint or LoFTR) into the WFT-Pose stage to further improve robustness where visual overlap is minimal.
Summary updated