TL;DR
This paper tackles the challenge of training effective infrared (IR) object detectors by leveraging vast amounts of readily available color (RGB) imagery. They propose a novel meta-learning approach, MLST, that learns to stylize RGB images into realistic IR images, significantly outperforming existing methods and enabling better IR object detection.
What Problem Does This Paper Address?
Deep neural networks (DNNs) have revolutionized object detection, but their success hinges on large, annotated datasets. While abundant RGB datasets exist, labeled IR datasets are scarce and limited in diversity. This bottleneck hinders the development of robust IR object detection systems crucial for applications like autonomous driving and surveillance. The paper addresses this by exploring Cross-Modal Style Transfer (CMST), specifically RGB-to-IR, to synthesize realistic IR imagery from abundant RGB data, thereby augmenting training datasets for IR object detectors.
Methods
The core of this work is Meta-Learning Style Transfer (MLST). MLST treats image stylization as a composition of simpler, well-behaved analytic functions (e.g., contrast adjustment, blur). Instead of manually crafting these compositions, MLST uses a meta-learning framework, inspired by Faster AutoAugment, to automatically learn the optimal sequence and parameters of these functions. This learned "Stylization Policy" is trained using an adversarial critic to distinguish between real IR and stylized RGB images, with an additional task-specific loss to preserve semantic content. The paper also benchmarks several existing CMST and general style transfer methods, including hand-crafted approaches, photorealistic style transfer (PST) methods like WCT2, and adversarial style transfer (AST) methods like CycleGAN and CyCADA.
Datasets
The paper utilizes the MS COCO dataset for RGB imagery, chosen for its extensive object detection labels and availability. For IR imagery and evaluation, they use four publicly available datasets: FLIR [2], DSIAC [1], CAMEL [11], and KAIST [6]. These datasets are standard benchmarks for IR object detection and have been used in prior CMST research.
Key Insights and Ideas
- Simple is sometimes better: Surprisingly, basic grayscale transformations often outperformed complex, data-driven CMST methods in terms of downstream IR object detection performance.
- Artifacts are detrimental: Existing data-driven CMST methods often introduce artifacts or reduce image dynamic range, negatively impacting the performance of object detectors trained on the synthesized data.
- Meta-learning for composition: MLST's strength lies in its ability to learn optimal compositions of simple functions, effectively creating complex stylizations without introducing artifacts.
- Task-driven stylization: Evaluating CMST methods based on their downstream task performance (IR object detection) is a more effective metric than purely visual assessment.
- Augmentation vs. Stylization: MLST can be applied to both stylization (RGB to IR) and augmentation (IR to IR). While both improve performance, stylization yields significantly larger gains, suggesting the primary benefit comes from bridging the modality gap.
Key Results
The paper presents a comprehensive benchmark of CMST methods.
- MLST (Unsupervised) achieved the highest average performance across all four IR datasets, outperforming all other methods by a significant margin.
- MLST (Supervised) further improved performance, demonstrating the benefit of incorporating downstream task labels during training.
- On average, MLST improved the [email protected] score by 13.5% (unsupervised) and 15.7% (supervised) compared to the baseline "None" method (training only on real IR data).
- MLST was the only data-driven method that consistently improved performance over the simple Grayscale (Inversion) baseline.
- Qualitatively, MLST produced complex, artifact-free stylizations that closely resembled real IR imagery, unlike some other methods that introduced artifacts or saturated images.
Limitations and Future Work
- Dictionary Dependence: MLST's effectiveness relies on a well-chosen dictionary of candidate functions. While suitable for RGB-to-IR, extending it to other stylization tasks might require different dictionaries.
- Generalization to Other Domains: The current work focuses on RGB-to-IR. Future work could explore extending MLST to other cross-modal tasks, such as hyperspectral or synthetic aperture radar (SAR) imagery.
- Improving Existing Models: The authors suggest future work could investigate how to improve existing, more expressive stylization models (like CycleGAN) to overcome their limitations in this specific RGB-to-IR context.
Summary updated