Enhancing Infrared Optical Flow Network Computation through RGB-IR Cross-Modal Image Generation

Feng Huang, Wei Huang, Xianyu WuSensors 2024 · DOI

TL;DR

This paper tackles the challenge of computing optical flow for infrared (IR) images, a task hindered by the lack of suitable datasets. The authors propose a novel method to generate synthetic IR optical flow datasets by transforming existing RGB datasets using a cross-modal image generation network. This approach significantly improves the performance of optical flow networks on IR imagery.

What Problem Does This Paper Address?

Optical flow, the estimation of motion between consecutive frames, is crucial for many computer vision tasks like navigation and motion inference. While deep learning has revolutionized optical flow computation for RGB images, its application to infrared (IR) imagery has lagged significantly. This is primarily due to the scarcity of IR optical flow datasets. Creating such datasets is labor-intensive and costly. Existing methods often rely on traditional algorithms for IR images or struggle to adapt deep learning models without sufficient IR-specific training data. This gap limits the practical use of advanced optical flow techniques in scenarios where IR imaging is advantageous, such as low-light conditions.

Methods

The core of this work lies in two main components:

  1. RGB-IR Cross-Modal Image Generation: The authors adapt the Pix2Pix architecture, a popular image-to-image translation framework, to transform RGB images into realistic IR images. This network is trained in a supervised manner using the M³FD dataset, which provides aligned RGB-IR image pairs. The loss function incorporates pixel consistency (L1 loss), gradient consistency (Sobel operator), and a generative adversarial loss (Patch GAN) to ensure high-quality IR image generation. To further improve the realism of generated IR images, especially for unlabeled datasets like KITTI, they introduce edge consistency loss (using HED features) and perceptual consistency loss (using Swin-T features) for fine-tuning.
  2. Optical Flow Network Fine-tuning: Once synthetic IR images are generated from existing RGB optical flow datasets (like KITTI), these synthetic IR images and their corresponding ground truth optical flow maps are used to train or fine-tune established optical flow networks. The paper evaluates two prominent networks: RAFT (Recurrent All-Pairs Field Transforms) and FastFlowNet. The goal is to demonstrate that training with these generated IR images enhances the performance of these networks when applied to real IR images.

Datasets

  • M³FD (Multi-Modal Fusion Dataset): This dataset is used for supervised training of the RGB-IR cross-modal image transformation network. It's highlighted as an aligned RGB-IR bimodal dataset.
  • KITTI: This well-known RGB optical flow dataset is used as the source for generating synthetic IR optical flow data. The RGB images from KITTI are transformed into IR images, and their corresponding optical flow maps are used for training.

Key Insights and Ideas

  • Bridging the Modality Gap: The paper demonstrates that a well-trained cross-modal generation network can effectively translate RGB image characteristics into plausible IR representations, creating a viable synthetic dataset for IR optical flow.
  • The Power of Synthetic Data: By generating IR optical flow data from existing RGB datasets, the authors circumvent the need for expensive real-world IR optical flow data collection, making advanced deep learning models accessible for IR applications.
  • Fine-tuning is Key: Simply applying RGB-trained optical flow models to IR images is insufficient. The study shows that fine-tuning these models with generated IR data significantly improves their accuracy and robustness in the IR domain.
  • Complementary Loss Functions: The combination of pixel, gradient, and adversarial losses, along with edge and perceptual losses for unlabeled data, is crucial for generating high-fidelity IR images that capture essential structural and textural information.

Key Results

  • Improved Optical Flow Accuracy: The fine-tuned optical flow networks (RAFT and FastFlowNet) using generated IR images achieved a significant reduction in endpoint error (EPE) on real IR images. For RAFT, EPE was reduced by 24% (from 64.8766 to 49.3069), and for FastFlowNet, it was reduced by 13.29% (from 8.6948 to 7.5395).
  • Visual Quality of Generated IR Images: Qualitative comparisons show that the proposed RGB-IR transformation network produces visually optimized and more realistic IR images compared to other methods like Pix2Pix, FDIT, and AlignGAN2, especially on challenging nighttime scenes.
  • Generalization: The method shows promising generalization capabilities, as evidenced by its performance on the KITTI dataset after training on M³FD.

Limitations and Future Work

  • Lack of Real IR Optical Flow Datasets: The authors acknowledge that the primary limitation remains the scarcity of real IR optical flow datasets. While synthetic data is effective, real data would provide a more robust benchmark.
  • Refinement of IR Image Generation: There is still a gap between simulated-generated IR images and real IR images. Further research is needed to refine the cross-modal transformation network to produce even more realistic IR images.
  • Future Directions: The paper suggests that collecting and producing real IR optical flow datasets would be a valuable contribution. Additionally, continued research into improving the realism of generated IR images is recommended to further enhance optical flow computation capabilities in IR scenes.

Summary updated

BibTeX
@article{huang2024enhancing,
  author = {Feng Huang and Wei Huang and Xianyu Wu},
  title = {{Enhancing Infrared Optical Flow Network Computation through RGB-IR Cross-Modal Image Generation}},
  journal = {Sensors},
  year = {2024},
  doi = {10.3390/s24051615},
}
Figure