Dark Side Augmentation: Generating Diverse Night Examples for Metric Learning

Albert Mohwald, Tomás Jenícek, Ondrej ChumICCV 2023

TL;DR

This paper tackles the challenge of training robust image retrieval systems for day-night scenarios, where night images are scarce and less diverse. They propose a novel GAN-based augmentation technique to synthesize realistic night images from abundant day data, significantly improving retrieval performance without requiring paired day-night images.

What Problem Does This Paper Address?

Deep learning-based image retrieval systems, particularly those using metric learning, rely on large datasets of diverse positive and negative image pairs for effective training. However, domains like night-time imagery suffer from a scarcity of both quantity and variability, leading to poor retrieval performance even for methods that excel on standard benchmarks. This limitation hinders applications like visual place recognition and localization in challenging, low-light conditions. Existing domain adaptation methods often neglect the crucial aspect of preserving performance on the original (day) domain, or require a substantial number of paired day-night images, which are difficult and expensive to acquire.

Methods

The core contribution is a GAN-based synthetic image generation approach to augment training data for day-night image retrieval. Instead of relying on real, paired day-night images, the authors train a generator to translate available day images into synthetic night images. This synthetic data is then used for metric learning.

Key novelties include:

  • HEDNGAN Architecture: A lightweight GAN architecture that enforces edge consistency between the original day image and the generated night image. This is achieved by penalizing the dissimilarity between edge maps extracted by a differentiable edge detector (HED) from both images. This also allows for simultaneous training of a robust edge detector (HEDN) that works well on both day and night images.
  • Diverse Anchor Mining: A novel method to increase training data variability by moving beyond random selection of training examples. It employs a pseudo-random importance sampling strategy to prevent over-reliance on similar examples and ensure a more diverse set of anchors are used in each training epoch.

The authors also compare various existing image translation methods (CycleGAN, DRIT, CUT, CyEDA) for their effectiveness in this augmentation task.

Datasets

  • Training: Retrieval-SfM dataset [30] is used for metric learning and day-night generator training.
  • Evaluation:
    • Image Retrieval: Tokyo 24/7 [42, 13], revisited Oxford and Paris [26].
  • Visual Localization: Aachen Day-Night v1.1 [48, 33], RobotCar Seasons [33, 20].

The Retrieval-SfM N/D dataset [13] is mentioned as a dataset with paired day-night images, but the authors emphasize their method does not require such paired data.

Key Insights and Ideas

  • Synthetic data can outperform real paired data: The proposed GAN-augmented approach achieves better results than using real, paired day-night images from datasets like SfM-N/D. This highlights the power of generating diverse synthetic data.
  • Edge consistency is crucial for day-night translation: Enforcing edge consistency in the GAN generator significantly improves the quality of generated night images and, consequently, retrieval performance.
  • Photo-realism isn't everything: The study suggests that while the generated images need to be visually plausible, extreme photo-realism is not as critical as ensuring that the learned embedding is robust to illumination changes.
  • Diverse anchor mining matters: This technique demonstrably improves performance, especially on day-night retrieval tasks, by preventing the model from becoming overly specialized to certain views or conditions.

Key Results

  • State-of-the-art performance: The proposed HEDNGAN method achieves state-of-the-art results on the Tokyo 24/7 day-night retrieval benchmark, outperforming previous methods, including those that use real paired day-night data.
  • Preserves day performance: The method maintains strong performance on the standard Oxford and Paris datasets, demonstrating its ability to generalize across domains.
  • Faster training: The HEDNGAN architecture is significantly faster to train than methods like CycleGAN.
  • Improved visual localization: The approach also shows consistent performance improvements on visual localization tasks.
  • Edge detector benefits: The jointly trained HEDN edge detector shows superior performance compared to standard HED and RCF detectors on night images.

Limitations and Future Work

  • Generator training time: While HEDNGAN is faster, some other generators (e.g., CycleGAN) are still computationally intensive to train.
  • Night-to-day translation: The paper notes that some GAN architectures (HEDNGAN, RCFNGAN, CUT) do not inherently support night-to-day translation, which might be a limitation for certain applications.
  • ToDayGAN limitations: The ToDayGAN method for night-to-day translation was found to perform well only on specific datasets (like RobotCar) and can sometimes hallucinate objects.
  • Future work: The authors suggest that their diverse anchor mining technique is broadly applicable to other metric learning tasks and could be beneficial for improving robustness in scenarios with strong data distribution modes.

Summary updated

BibTeX
@inproceedings{mohwald2023dark,
  author = {Albert Mohwald and Tomás Jenícek and Ondrej Chum},
  title = {{Dark Side Augmentation: Generating Diverse Night Examples for Metric Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure