Enhanced Target Tracking Algorithm for Autonomous Driving Based on Visible and Infrared Image Fusion

Quan Yuan, Haixu Shi, Ashton Tan Yu Xuan, Ming Gao, Qing Xu, Jianqiang WangJournal of Intelligent and Connected Vehicles 2023 · DOI

Abstract

In autonomous driving, target tracking is essential to environmental perception. The study of target tracking algorithms can improve the accuracy of an autonomous driving vehicle's perception, which is of great significance in ensuring the safety of autonomous driving and promoting the landing of technical applications. This study focuses on the fusion tracking algorithm based on visible and infrared images. The proposed approach utilizes a feature-level image fusion method, dividing the tracking process into two components: image fusion and target tracking. An unsupervised network, Visible and Infrared image Fusion Network (VIF-net), is employed for visible and infrared image fusion in the image fusion part. In the target tracking part, Siamese Region Proposal Network (SiamRPN), based on deep learning, tracks the target with fused images. The fusion tracking algorithm is trained and evaluated on the visible infrared image dataset RGBT234. Experimental results demonstrate that the algorithm outperforms training networks solely based on visible images, proving that the fusion of visible and infrared images in the target tracking algorithm can improve the accuracy of the target tracking even if it is like tracking-based visual images. This improvement is also attributed to the algorithm's ability to extract infrared image features, augmenting the target tracking accuracy.

Introduction

Reliable environmental perception serves as the "eyes" of an autonomous vehicle, providing the critical data necessary for navigation and obstacle avoidance. While standard visible-light (RGB) cameras offer rich detail and high-resolution texture, they are notoriously sensitive to environmental factors such as low-light conditions, nighttime, or heavy fog. Conversely, infrared (IR) sensors excel at detecting thermal signatures—such as pedestrians or running engines—regardless of lighting conditions, though they often lack the fine-grained resolution required for precise spatial tracking.

To address these limitations, this research explores multimodal fusion, specifically combining RGB and IR data streams into a single, high-quality representation. The challenge lies in integrating these disparate data types effectively without sacrificing the real-time processing speeds required for safe autonomous operation.

Methodology

The proposed framework adopts a decoupled, two-stage architecture that separates the image fusion process from the tracking task. This modularity allows for independent optimization of each component.

1. Image Fusion via VIF-net

Rather than employing simple pixel-level averaging, which often results in blurred outputs, the system utilizes the Visible and Infrared image Fusion Network (VIF-net). VIF-net is an unsupervised deep learning network designed to perform feature-level fusion. It employs convolutional layers and "dense blocks" to extract high-level features—such as sharp edges from RGB data and thermal intensity from IR data.

Because no "perfect" ground truth exists for fused imagery, VIF-net is trained using a custom loss function based on Structural Similarity (SSIM) and Total Variation (TV). This ensures that the resulting fused image preserves the structural integrity of the visible light feed while incorporating the salient thermal information from the infrared feed.

2. Target Tracking via SiamRPN

The fused output is then passed to the Siamese Region Proposal Network (SiamRPN). This architecture treats target tracking as a local detection problem. It functions by comparing a "template" (the specific object identified for tracking) against a "detection frame" (the current environmental view) to identify the best match. By incorporating a Region Proposal Network (RPN), the system can dynamically handle changes in the object's scale and aspect ratio, making it significantly more robust than traditional correlation filters.

Experimental Setup and Datasets

The model was primarily trained and evaluated using the RGBT234 dataset, which consists of 234 fully annotated visible and infrared video pairs. To ensure broad generalizability, the tracking component was pre-trained on large-scale datasets including COCO, ILSVRC, and Youtube-BB. Additionally, specialized infrared datasets (INO and TNO) were utilized during the initial training phases of the VIF-net fusion component.

Key Results and Discussion

The experimental results highlight the significant advantages of the fusion-based approach over unimodal systems:

  • Accuracy Improvement: The fusion-based model achieved a tracking accuracy of 0.577, representing a 7.4% improvement over the baseline model trained exclusively on visible light (0.537). This gain is directly attributed to the system's ability to maintain a lock on targets when visible features are obscured.
  • Real-Time Performance: Despite the computational overhead of a two-stage deep learning framework, the algorithm maintained an average speed of 125.5 FPS. This performance is well above the standard requirements for real-time autonomous driving, where low latency is paramount.
  • Nighttime Robustness: Qualitative analysis in low-light scenarios demonstrated that while RGB-only models frequently lost targets in the dark, the fusion model maintained high precision by leveraging infrared thermal data to distinguish pedestrians and vehicles from the background.

Conclusion and Future Work

This study demonstrates that feature-level fusion of visible and infrared imagery significantly enhances the accuracy and reliability of target tracking for autonomous driving. By decoupling fusion and tracking, the system achieves a balance between high-fidelity perception and real-time execution speeds.

Future research will focus on increasing the complexity of the Siamese network's backbone (e.g., transitioning from AlexNet to ResNet) to further boost accuracy. Additionally, the researchers aim to expand the training duration and explore larger, more diverse RGBT datasets to improve the model's performance in increasingly complex urban environments.

Summary updated

Abstract

In autonomous driving, target tracking is essential to environmental perception. The study of target tracking algorithms can improve the accuracy of an autonomous driving vehicle's perception, which is of great significance in ensuring the safety of autonomous driving and promoting the landing of technical applications. This study focuses on the fusion tracking algorithm based on visible and infrared images. The proposed approach utilizes a feature-level image fusion method, dividing the tracking process into two components: image fusion and target tracking. An unsupervised network, Visible and Infrared image Fusion Network (VIF-net), is employed for visible and infrared image fusion in the image fusion part. In the target tracking part, Siamese Region Proposal Network (SiamRPN), based on deep learning, tracks the target with fused images. The fusion tracking algorithm is trained and evaluated on the visible infrared image dataset RGBT234. Experimental results demonstrate that the algorithm outperforms training networks solely based on visible images, proving that the fusion of visible and infrared images in the target tracking algorithm can improve the accuracy of the target tracking even if it is like tracking-based visual images. This improvement is also attributed to the algorithm's ability to extract infrared image features, augmenting the target tracking accuracy.
BibTeX
@article{yuan2023enhanced,
  author = {Quan Yuan and Haixu Shi and Ashton Tan Yu Xuan and Ming Gao and Qing Xu and Jianqiang Wang},
  title = {{Enhanced Target Tracking Algorithm for Autonomous Driving Based on Visible and Infrared Image Fusion}},
  journal = {Journal of Intelligent and Connected Vehicles},
  year = {2023},
  doi = {10.26599/jicv.2023.9210018},
}
Figure