Abstract
Unpaired infrared-to-visible (IR-to-visible) video translation is a critical task for enhancing autonomous driving and surveillance in low-visibility conditions. However, standard models often suffer from a "long-tail" pixel distribution problem, where they prioritize large, uniform areas like the sky over small, detail-rich objects like cars and road signs. This paper introduces CPTrans, a novel framework that utilizes content-aware optimization and temporal normalization to focus on these "content-rich" patches. CPTrans achieves state-of-the-art visual quality, significantly improves downstream object detection performance, and trains up to five times faster than existing methods.
The Long-Tail Problem in IR-to-Visible Translation
Infrared sensors are vital for operating in total darkness or adverse weather, but most computer vision systems are optimized for visible-light data. Bridging this gap via unpaired translation—converting IR video to visible video without matched pairs—is difficult because of the semantic imbalance in typical driving scenes.
As shown in Figure 5, "content-lacking" patches (e.g., sky, road) dominate the pixel count. During training, optimization gradients are overwhelmed by these regions, leading the model to produce blurry or semantically incorrect results for "content-rich" patches like pedestrians or vehicles. This "optimization prejudice" prevents the generator from learning the fine-grained textures necessary for high-quality translation.
The CPTrans Framework
To address these challenges, the authors propose two primary innovations within a Generative Adversarial Network (GAN) pipeline:
1. Content-aware Optimization (CO)
CPTrans identifies patches where the local gradient direction deviates significantly from the global average. By using GradCAM++ and cosine similarity, the model flags these deviated patches as content-rich areas. It then applies a weight map to amplify the gradients of these specific patches, forcing the generator to prioritize fine-grained details during the optimization process.
2. Content-aware Temporal Normalization (CTN)
Temporal consistency is essential to prevent flickering in translated videos. While previous methods used random "fake flows" to warp images, CPTrans guides these flows using the content-rich weight maps from the CO module. This ensures the model is specifically robust to the motion of important objects, such as moving cars, rather than just the static background.
Experimental Results
The framework was evaluated on three major benchmarks: IRVI, InfraredCity-Lite, and a new InfraredCity-Adverse dataset containing rain and snow conditions.
Qualitative and Quantitative Superiority
CPTrans achieved the best scores in Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) across all datasets. Qualitatively, the model produces much sharper lane lines and recognizable text on road signs compared to previous state-of-the-art methods like ROMA.
Downstream Performance and Efficiency
When translated videos were processed by a pre-trained YOLOv3 detector, CPTrans achieved an Average Precision (AP) of 58.1, a significant jump from ROMA's 50.1. Furthermore, the framework is highly efficient, reaching peak performance approximately 5x faster than ROMA by focusing its computational effort on the most informative parts of the scene.
Conclusion
CPTrans demonstrates that "fairness" in pixel-wise training can be a hindrance in content-imbalanced scenes. By intentionally over-weighting minority, content-rich patches, the model achieves superior semantic alignment and visual clarity. Future work may explore optimizing the generator for real-time inference on edge devices to further support autonomous vehicle deployment.
Summary updated