Adversarial Training for Adverse Conditions: Robust Metric Localisation Using Appearance Transfer

Horia Porav, Will Maddern, Paul Newman

TL;DR

Researchers from the Oxford Robotics Institute have developed a way to help robots "see" through adverse weather and lighting by using Generative Adversarial Networks (GANs). By transforming images of night, rain, or snow into synthetic "clear day" versions—specifically optimized to preserve matchable geometric features—they significantly improved the accuracy of camera-based localization without needing to re-map the world for every possible weather condition.

What Problem Does This Paper Address?

For a self-driving car or a mobile robot, knowing exactly where it is (metric localization) is a fundamental requirement. Most current systems rely on matching "keypoints" (distinctive visual landmarks) between a live camera feed and a pre-built map.

However, these systems are notoriously fragile. A map built on a sunny Tuesday afternoon is often useless on a snowy Friday night because the visual features—the shadows, the textures, and the illumination—change too drastically for traditional algorithms like SURF or ORB to recognize them. The standard "brute force" solution is to map the same route dozens of times in every possible condition, which is expensive, time-consuming, and often impractical. The authors ask: Can we instead transform the live "bad" image into a "good" one that matches our existing map?

Methods

The authors propose a two-stage training pipeline based on Appearance Transfer.

  1. Stage 1: Unsupervised Cycle-Consistency: Using a GAN architecture similar to CycleGAN, they train two generators ($G_{AB}$ and $G_{BA}$) to translate images between domains (e.g., Night $\to$ Day). Crucially, they don't just use a standard "looks real" loss. They introduce a differentiable SURF detector and descriptor loss. They implemented a convolutional version of the SURF algorithm, allowing the network to receive mathematical feedback on whether the synthetic image actually contains the same geometric features as the original.
  2. Stage 2: Supervised Fine-Tuning: Once the generators understand the general "style" of the transformation, they are fine-tuned using a small set of pixel-aligned image pairs (using GPS ground truth). This stage uses an L1 loss to ensure the feature maps of the synthetic image perfectly align with the target "clear day" image.

The "secret sauce" here is the Differentiable SURF stack. By making the feature extraction process part of the neural network's loss function, the generator learns to prioritize reconstructing the edges and corners that the localization system actually uses, rather than just making the image look pretty to a human.

Datasets

The study utilizes the Oxford RobotCar Dataset, a massive longitudinal collection of data captured over a year-long period in Oxford, UK.

  • Conditions: The authors focused on five specific pairs: day-night, day-snow, day-dawn, day-sun, and day-rain.
  • Ground Truth: They used RTK-GPS data (filtered for high precision) to benchmark the accuracy of their metric localization.

Key Insights and Ideas

  • Feature-Preserving Synthesis: Standard GANs often "hallucinate" details to make an image look realistic. In localization, hallucinating a window where there isn't one is catastrophic. By adding SURF-specific losses, the authors force the GAN to be "geometrically honest."
  • Modularity: Instead of inventing a brand-new localization algorithm, this method acts as a "pre-filter." You can take an existing, trusted localization pipeline and simply "plug in" this appearance transfer module at the front.
  • Efficiency in Mapping: The results suggest we don't need a map for every condition. A single, high-quality "Golden Map" (captured in clear day) can be used year-round if the robot can successfully "hallucinate" that clear day from its current rainy or dark environment.

Key Results

The improvements were dramatic, particularly in the most challenging "Day vs. Night" scenarios:

  • Localization Success: In the "Day vs. Night" test, the reference system (matching raw night images to a day map) only achieved a 19.5% success rate. Using the synthetic images from the Stage-2 fine-tuned model, the success rate jumped to 60%.
  • Metric Accuracy: For successful matches, the Root Mean Squared Error (RMSE) in translation dropped from 12.74 meters (raw) to just 1.99 meters (synthetic).
  • Robustness: Qualitative results showed that even in heavy snow or rain, the generator was able to reconstruct structural details like window frames and building outlines that were nearly invisible in the original "adverse" frames.

Limitations and Future Work

  • Discrete Bins: Currently, the system requires a specific generator for each condition (e.g., one for snow, one for night). The authors note that a more ideal system would handle continuous changes in illumination and weather within a single model.
  • Compute Overhead: While the generator runs at 1–3 Hz on an Nvidia Titan X, this might still be a bottleneck for high-speed robotics applications requiring real-time processing at 30+ FPS.
  • Hallucination Risks: While the SURF loss mitigates this, there is always a theoretical risk that a GAN might "correct" a scene by adding a feature that was removed in reality (e.g., a demolished building), leading to false confidence in localization.

Summary updated

Abstract

We present a method of improving visual place recognition and metric localisation under very strong appearance change. We learn an invertable generator that can transform the conditions of images, e.g. from day to night, summer to winter etc. This image transforming filter is explicitly designed to aid and abet feature-matching using a new loss based on SURF detector and dense descriptor maps. A network is trained to output synthetic images optimised for feature matching given only an input RGB image, and these generated images are used to localize the robot against a previously built map using traditional sparse matching approaches. We benchmark our results using multiple traversals of the Oxford RobotCar Dataset over a year-long period, using one traversal as a map and the other to localise. We show that this method significantly improves place recognition and localisation under changing and adverse conditions, while reducing the number of mapping runs needed to successfully achieve reliable localisation.
BibTeX
@inproceedings{porav2018adversarial,
  author = {Horia Porav and Will Maddern and Paul Newman},
  title = {{Adversarial Training for Adverse Conditions: Robust Metric Localisation Using Appearance Transfer}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=iEAAwPUPqJ},
}

Loading related content...

Figure