On the Difficulty of Unpaired Infrared-to-Visible Video Translation: Fine-Grained Content-Rich Patches Transfer

Zhenjie Yu, Shuang Li, Yirui Shen, Chi Harold Liu, Shuigen WangCVPR 2023 · DOI

Abstract

Unpaired infrared-to-visible (IR-to-visible) video translation is a critical task for enhancing autonomous driving and surveillance in low-visibility conditions. However, standard models often suffer from a "long-tail" pixel distribution problem, where they prioritize large, uniform areas like the sky over small, detail-rich objects like cars and road signs. This paper introduces CPTrans, a novel framework that utilizes content-aware optimization and temporal normalization to focus on these "content-rich" patches. CPTrans achieves state-of-the-art visual quality, significantly improves downstream object detection performance, and trains up to five times faster than existing methods.

The Long-Tail Problem in IR-to-Visible Translation

Infrared sensors are vital for operating in total darkness or adverse weather, but most computer vision systems are optimized for visible-light data. Bridging this gap via unpaired translation—converting IR video to visible video without matched pairs—is difficult because of the semantic imbalance in typical driving scenes.

Figure 5: The long-tail distribution of pixels in driving scenes and the use of GradCAM++ to identify content-rich patches.

As shown in Figure 5, "content-lacking" patches (e.g., sky, road) dominate the pixel count. During training, optimization gradients are overwhelmed by these regions, leading the model to produce blurry or semantically incorrect results for "content-rich" patches like pedestrians or vehicles. This "optimization prejudice" prevents the generator from learning the fine-grained textures necessary for high-quality translation.

The CPTrans Framework

To address these challenges, the authors propose two primary innovations within a Generative Adversarial Network (GAN) pipeline:

1. Content-aware Optimization (CO)

CPTrans identifies patches where the local gradient direction deviates significantly from the global average. By using GradCAM++ and cosine similarity, the model flags these deviated patches as content-rich areas. It then applies a weight map to amplify the gradients of these specific patches, forcing the generator to prioritize fine-grained details during the optimization process.

2. Content-aware Temporal Normalization (CTN)

Temporal consistency is essential to prevent flickering in translated videos. While previous methods used random "fake flows" to warp images, CPTrans guides these flows using the content-rich weight maps from the CO module. This ensures the model is specifically robust to the motion of important objects, such as moving cars, rather than just the static background.

Experimental Results

The framework was evaluated on three major benchmarks: IRVI, InfraredCity-Lite, and a new InfraredCity-Adverse dataset containing rain and snow conditions.

Qualitative and Quantitative Superiority

CPTrans achieved the best scores in Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) across all datasets. Qualitatively, the model produces much sharper lane lines and recognizable text on road signs compared to previous state-of-the-art methods like ROMA.

Figure 1: Qualitative results of the CPTrans framework showing clear reconstruction of road signs and vehicle textures.

Downstream Performance and Efficiency

When translated videos were processed by a pre-trained YOLOv3 detector, CPTrans achieved an Average Precision (AP) of 58.1, a significant jump from ROMA's 50.1. Furthermore, the framework is highly efficient, reaching peak performance approximately 5x faster than ROMA by focusing its computational effort on the most informative parts of the scene.

Conclusion

CPTrans demonstrates that "fairness" in pixel-wise training can be a hindrance in content-imbalanced scenes. By intentionally over-weighting minority, content-rich patches, the model achieves superior semantic alignment and visual clarity. Future work may explore optimizing the generator for real-time inference on edge devices to further support autonomous vehicle deployment.

Summary updated

BibTeX
@inproceedings{yu2023difficulty,
  author = {Zhenjie Yu and Shuang Li and Yirui Shen and Chi Harold Liu and Shuigen Wang},
  title = {{On the Difficulty of Unpaired Infrared-to-Visible Video Translation: Fine-Grained Content-Rich Patches Transfer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00163},
}
Figure