DNIT: Enhancing Day-Night Image-to-Image Translation through Fine-Grained Feature Handling (Student Abstract)

Hanyue Liu, Haonan Cheng, Long YeAAAI 2024 · DOI

TL;DR

This paper introduces DNIT, a novel approach to day-to-night image translation that tackles the challenges of low illumination and structural detail loss. By employing a Nighttime Image Preprocessing (NIP) module to enhance visibility and an Edge Fusion Detection (EFD) module to preserve structural integrity, DNIT significantly improves the realism and quality of translated images compared to existing methods.

What Problem Does This Paper Address?

Image-to-image translation (I2IT) aims to transform an image from one domain to another. While methods like CycleGAN are effective for general tasks, they struggle with the specific "day-night" translation scenario. Nighttime images suffer from low illumination, making feature extraction difficult and increasing the risk of losing crucial structural details. Existing methods often fail to adequately address these domain-specific challenges, leading to mistranslations and unrealistic outputs. This paper addresses the need for a more robust I2IT method that can effectively handle the unique difficulties presented by day-to-night image translation.

Methods

DNIT introduces two key novel modules to enhance day-to-night image translation:

  • Nighttime Image Preprocessing (NIP) Module: This module is designed to improve the visibility of nighttime images while minimizing noise. It comprises two sub-networks:
  • Self-Calibrated Illumination (SCI) Learning Network: Enhances brightness.
  • Dark Channel Prior (DCP) Defogging Network: Reduces noise introduced by the brightening process. The NIP module aims to provide the generator with better-defined content and style features from dark areas.

  • Edge Fusion Detection (EFD) Module: To combat the loss of structural details, the EFD module uses edge information as an additional constraint. It combines edge maps generated by both Sobel and Canny filters from real and translated images. An L1 loss is then applied to minimize the distance between these edge maps, encouraging the generator to preserve structural accuracy.

These modules are integrated into a larger I2IT framework, which also includes a semantic prior generator (SPG(GA)) for conversion.

Datasets

The paper evaluates DNIT on the ACDC dataset for day-to-night image translation.

Key Insights and Ideas

  • Fine-grained feature handling is crucial for challenging domains: The paper highlights that simply applying general I2IT methods to specific, difficult domains like day-to-night translation is insufficient. Tailored modules that address domain-specific challenges (low light, structural loss) are necessary.
  • Combining illumination enhancement and noise reduction is key: The NIP module's success demonstrates that simply brightening a dark image isn't enough; managing the introduced noise is equally important for effective feature extraction.
  • Edge information acts as a powerful structural constraint: The EFD module's effectiveness shows that explicitly guiding the generator to preserve edge details significantly improves the realism and structural integrity of the translated images.
  • User studies are vital for perceptual quality assessment: While quantitative metrics are useful, the paper emphasizes the importance of user studies for evaluating the perceived quality and naturalness of generated images, especially in subjective tasks like image translation.

Key Results

  • Quantitative Improvements: DNIT achieves state-of-the-art results, outperforming the baseline SPN2D-GAN across several metrics. Specifically, it reduces FID from 65.54 to 64.72, KID from 2.46 to 2.23, and BRISQUE from 20.36 to 19.04, while maintaining a similar number of model parameters.
  • Qualitative Improvements: Visual comparisons (Figure 3) show that DNIT generates more realistic images with improved texture, structure, and color fidelity compared to other methods.
  • User Study Validation: A user study with 20 evaluators ranked DNIT as the best in terms of image quality, with 51% of users preferring its outputs.

Limitations and Future Work

  • NIQE metric: The paper notes that the NIQE metric is slightly higher for DNIT, suggesting a potential decrease in the quality of natural scene statistics features.
  • Adaptive Canny Threshold: The authors propose future work to improve the adaptive selection of the Canny threshold for better generalization across diverse scenes.

Summary updated

BibTeX
@inproceedings{liu2024dnit,
  author = {Hanyue Liu and Haonan Cheng and Long Ye},
  title = {{DNIT: Enhancing Day-Night Image-to-Image Translation through Fine-Grained Feature Handling (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30474},
}
Figure