TL;DR
This paper introduces V2T-GAN, a novel three-level cascaded generative adversarial network designed for translating visible light images into realistic infrared images. By employing a multi-stage approach with semantic segmentation, grayscale inversion, and edge information as auxiliary guidance, V2T-GAN significantly improves the structural and textural accuracy of synthesized infrared images, outperforming existing methods.
What Problem Does This Paper Address?
Generating realistic infrared images from visible light images is a challenging task with significant applications in fields like military, medical, and industrial monitoring. Traditional methods often struggle with complex scene modeling or suffer from low accuracy and poor generalization when relying solely on visible light inputs. While deep learning approaches have shown promise, they can lead to blurred edges and loss of texture details. This paper aims to bridge this gap by proposing a more accurate, robust, and efficient method for visible-to-thermal image translation.
Methods
The core of the proposed solution is the V2T-GAN, a three-level cascaded generative adversarial network. Each level refines the infrared image generation process:
- Level 1: Uses semantic segmentation as an auxiliary task to guide the generator ($G_1$) in capturing accurate structural information. A semantic segmentation network ($G_5$) helps $G_1$ focus on structural details.
- Level 2: Leverages the grayscale inverted visible (GIV) image as an auxiliary task to enhance the texture details of the infrared image. A GIV network ($G_8$) guides the generator ($G_2$) to produce more detailed textures.
- Level 3: Incorporates visible light edge information and a displacement network ($G_d$) to refine edges and produce sharp, accurate outlines in the final infrared image. $G_d$ predicts offset maps to resample the output of the second level.
To improve efficiency, the network employs lightweight convolutions such as GConv, DSConv, and Ghost modules, reducing parameter count and computational cost. The entire network is trained end-to-end using a combination of adversarial and pixel-level losses, including L1 loss and a gradient loss for texture enhancement.
Datasets
The experiments were conducted on the Multispectral Pedestrian Dataset (MPD). This dataset contains pairs of visible light and infrared images. The authors used daytime images for training (33,399 pairs) and a subset of daytime images from the test set for evaluation (565 pairs). Images were resized to 256x256 for processing. Auxiliary information like semantic segmentation images was generated using a model trained on the Cityscapes dataset.
Key Insights and Ideas
- Cascaded Refinement: The three-level structure is crucial. Each level builds upon the previous one, progressively adding structural accuracy, then detailed textures, and finally sharp edges, leading to a more comprehensive and realistic infrared image.
- Auxiliary Tasks for Guidance: Using semantic segmentation and GIV images as auxiliary tasks effectively guides the generators to focus on specific aspects (structure and texture, respectively) that are difficult to capture directly.
- Edge Refinement with Displacement: The third level's use of edge information and a displacement network is a clever way to address the common issue of blurred edges in generated images, directly optimizing for edge sharpness.
- Lightweight Design: The integration of lightweight convolutions is key to making the complex V2T-GAN practical for real-world applications, balancing performance with computational efficiency.
Key Results
- Objective Metrics: V2T-GAN achieved the best results across several objective evaluation metrics (Abs Rel, Avg log10, RMS, δ < 1.25, PSNR, SSIM) compared to state-of-the-art methods like Pix2pix, X-Fork, Selection-GAN, SEAN, and LG-GAN. For instance, it achieved the lowest RMS error of 0.850 and the highest SSIM of 0.990.
- Computational Efficiency: The proposed V2T-GAN has significantly fewer network parameters (15.24 M) compared to other methods, demonstrating its lightweight nature.
- Qualitative Results: Visual comparisons showed that V2T-GAN produces infrared images with more accurate structures, finer textures, and sharper edges than its counterparts. Ablation studies confirmed the effectiveness of each level and the auxiliary tasks.
Limitations and Future Work
- Limitations: The paper notes that while objective metrics are improved, subjective perception of the generated images can still be an area for further enhancement, particularly in the third level's output before resampling. The reliance on paired training data for each level is also a characteristic of the approach.
- Future Work: The authors aim to explore converting visible light to infrared images without requiring one-to-one mapping in the training data. They also suggest applying the algorithm's principles to other areas like visible light and infrared image fusion and detailed enhancement of infrared images.
Summary updated