TL;DR
This paper introduces CoordiNet, a novel neural network architecture for visual camera localization that predicts both pose and its uncertainty. By integrating geometric cues and learning uncertainty jointly with pose, CoordiNet achieves state-of-the-art performance on challenging benchmarks and demonstrates robust real-time localization in large-scale urban environments, outperforming previous methods by a significant margin.
What Problem Does This Paper Address?
Accurate and reliable localization is crucial for autonomous vehicles and robotics. While visual-based localization using deep neural networks (DNNs) offers computational efficiency compared to traditional structure-based methods, existing DNNs often struggle with accuracy, producing outliers and failing to generalize to unseen environments. A key challenge is knowing when a predicted pose is unreliable. Current methods for estimating pose uncertainty are often computationally expensive or not well-correlated with actual errors. This paper addresses the need for a visual localization system that is not only accurate but also provides reliable uncertainty estimates to filter out failures and ensure smooth, consistent trajectories, especially in complex, dynamic urban settings.
Methods
The core of the proposed solution is CoordiNet, a fully convolutional neural network architecture. Key innovations include:
- CoordConv Layers: Standard convolutional layers are augmented with "CoordConv" layers, which explicitly incorporate pixel coordinates into the input tensor. This helps the network leverage geometric information and mitigate translation invariance issues inherent in standard CNNs.
- Confidence-Weighted Average Pooling (CWAP): Instead of standard Global Average Pooling (GAP), CoordiNet uses CWAP. This method computes a weighted mean of feature maps, where the weights are derived from a learned "confidence map." This is akin to a low-cost self-attention mechanism and helps focus on relevant scene features, even in large, diverse scenes.
- Joint Pose and Uncertainty Learning: CoordiNet predicts both the 6-DoF camera pose (3D translation and 3D rotation) and its uncertainty simultaneously. This is achieved through a unified loss function that minimizes the negative log-likelihood of the predicted pose and its associated uncertainty. Specifically, it learns heteroscedastic uncertainty, meaning the uncertainty can vary depending on the input image.
- EKF Fusion: At test time, the predicted poses and their learned uncertainties are fused with an Extended Kalman Filter (EKF). This allows for smoothing trajectories and effectively filtering out outliers, leading to more reliable localization.
Datasets
The paper evaluates CoordiNet on several datasets:
- Oxford RobotCar Dataset: Used for training and evaluation, including experiments with varying amounts of training data (2 sequences vs. 15 sequences).
- Cambridge Landmarks Dataset: A collection of smaller outdoor scenes.
- Paris and Shanghai Datasets: Newly collected large-scale datasets representing busy urban environments, used to demonstrate the method's scalability and real-time performance.
Key Insights and Ideas
- Geometric Cues Matter: Explicitly incorporating spatial coordinates via CoordConv layers significantly improves the network's ability to handle geometric tasks like pose regression, moving beyond purely semantic feature extraction.
- Uncertainty is Key to Reliability: Learning and utilizing pose uncertainty is crucial for filtering out erroneous predictions and achieving smooth, robust trajectories, especially in challenging scenarios.
- Joint Learning is Efficient: Predicting uncertainty alongside pose with a single loss function is computationally efficient and leads to uncertainties that are well-correlated with actual pose errors.
- Large Datasets Enable Sub-meter Accuracy: Training on significantly larger datasets (e.g., 15 sequences from Oxford RobotCar) dramatically improves localization accuracy, achieving sub-meter median errors.
- Calibrated Uncertainty is Beneficial: Fine-tuning the uncertainty estimation layer on a separate calibration set further improves the reliability of the uncertainty estimates, leading to better performance when fused with an EKF.
Key Results
- Oxford RobotCar: CoordiNet achieved a mean error of 8 meters, significantly outperforming previous best methods (19 meters). With an enlarged training set (15 sequences), the mean error dropped to 1.94m and median error to 1.25m.
- Cambridge Landmarks: CoordiNet reported the best results on all scenes, outperforming other monocular and sequential methods.
- Large-Scale Datasets (Paris/Shanghai): CoordiNet achieved sub-meter median errors (e.g., 0.29m in Paris, 0.69m in Shanghai) when trained on large datasets, outperforming AD-PoseNet by an order of magnitude.
- Real-time Performance: The method can operate in real-time (18 Hz) on an embedded GPU.
- EKF Fusion: When fused with an EKF, CoordiNet with calibrated uncertainty achieved the best performance in terms of smoothness and reduced maximum error on the Paris dataset.
Limitations and Future Work
- SE(3) Covariance Modeling: The paper acknowledges that the simplified diagonal covariance matrix used in the EKF is a limitation. Future work aims to improve the covariance matrix modeling in SE(3), potentially incorporating non-diagonal coefficients or more sophisticated formulations.
- Extrapolation: The method, like many others, may struggle to extrapolate to poses outside of its training distribution.
- Sequence-based Learning: While the current work focuses on single-image pose regression, the authors suggest that using sequences of images could further improve performance.
- More Sophisticated Uncertainty Representation: Exploring more advanced uncertainty representations beyond the current heteroscedastic model is a potential avenue for future research.
Summary updated