Abstract
We present a method of improving visual place recognition and metric localisation under very strong appearance change. We learn an invertable generator that can transform the conditions of images, e.g. from day to night, summer to winter etc. This image transforming filter is explicitly designed to aid and abet feature-matching using a new loss based on SURF detector and dense descriptor maps. A network is trained to output synthetic images optimised for feature matching given only an input RGB image, and these generated images are used to localize the robot against a previously built map using traditional sparse matching approaches. We benchmark our results using multiple traversals of the Oxford RobotCar Dataset over a year-long period, using one traversal as a map and the other to localise. We show that this method significantly improves place recognition and localisation under changing and adverse conditions, while reducing the number of mapping runs needed to successfully achieve reliable localisation.
Introduction
For autonomous systems, metric localization—the ability to determine a precise position within a known map—is a critical capability. Traditional localization pipelines typically rely on matching visual keypoints (like SURF or ORB) between a live camera feed and a pre-existing map. However, these methods are highly sensitive to appearance changes. A map generated during a clear day often becomes unusable at night or during heavy snow because the visual textures and illumination change too drastically for standard algorithms to recognize landmarks.
To address this, researchers often resort to "brute force" mapping, recording the same environment under every possible condition. This paper proposes a more efficient alternative: using Generative Adversarial Networks (GANs) to transform "adverse" images into a synthetic "clear day" domain that matches the existing map.
Methodology: Appearance Transfer
The core of the approach is a two-stage training pipeline designed to learn a robust image-to-image translation that preserves geometric integrity.
Stage 1: Unsupervised Cycle-Consistency
The first stage employs a CycleGAN-inspired architecture to learn the general style transformation between domains (e.g., Night to Day) without requiring paired images. To ensure the generator does not "hallucinate" incorrect geometry, the authors introduced a differentiable SURF detector and descriptor loss. By implementing a convolutional version of the SURF algorithm, the network receives mathematical feedback on whether the synthetic image preserves the edges and corners necessary for localization.
Stage 2: Supervised Fine-Tuning
Following the unsupervised phase, the generators are fine-tuned using a small set of pixel-aligned image pairs derived from GPS ground truth. This stage uses an L1 loss to ensure that the feature maps of the synthetic images perfectly align with the target "clear day" reference images, further refining the geometric accuracy of the output.
Experimental Evaluation
The system was evaluated using the Oxford RobotCar Dataset, which contains over a year of data captured in various weather and lighting conditions. The study focused on five challenging pairs: day-night, day-snow, day-dawn, day-sun, and day-rain. Localization accuracy was benchmarked against high-precision RTK-GPS ground truth.
Key Results
The proposed method demonstrated significant improvements in localization reliability, particularly in extreme scenarios. In the "Day vs. Night" test, the success rate of the baseline system (matching raw night images to a day map) was only 19.5%. By using synthetic images from the fine-tuned model, this success rate increased to 60%.
Beyond success rates, the precision of successful matches improved dramatically. The Root Mean Squared Error (RMSE) in translation dropped from 12.74 meters using raw images to 1.99 meters using the synthetic output. Qualitatively, the generator successfully reconstructed structural details like building outlines and window frames that were obscured by darkness or glare in the original frames.
Conclusion and Future Work
This research demonstrates that appearance transfer can serve as a powerful, modular "pre-filter" for existing localization pipelines. By forcing GANs to be "geometrically honest" through differentiable feature losses, robots can successfully navigate adverse conditions using a single "Golden Map" captured in ideal weather. Future work aims to consolidate multiple condition-specific generators into a single model capable of handling continuous environmental changes and optimizing the architecture for real-time deployment.
Summary updated