Corrections to "SiamFT: An RGB-Infrared Fusion Tracking Method via Fully Convolutional Siamese Networks"

Xingchen Zhang, Ping Ye, Shengyun Peng, Jun Liu, Ke Gong, Gang XiaoIEEE Access 2019 · DOI

Introduction

This article presents formal corrections to the 2019 publication "SiamFT: An RGB-Infrared Fusion Tracking Method via Fully Convolutional Siamese Networks." The original paper established a robust framework for multi-modal object tracking; however, a clerical error led to the publication of an incorrect Figure 5. This update provides the corrected visualizations, which are essential for understanding how the SiamFT model dynamically balances visible and infrared data in challenging environments.

The Problem: Robust Multi-modal Tracking

Object tracking is a cornerstone of computer vision, but "visible-only" trackers often fail in real-world scenarios. Environmental factors such as extreme over-exposure (e.g., glare from headlights) or total darkness can render visible sensors ineffective. To solve this, researchers utilize RGB-Infrared (RGB-T) fusion. Visible cameras provide high-resolution texture and color, while infrared (thermal) cameras capture heat signatures regardless of lighting. The primary challenge is the fusion strategy: determining which sensor to trust at any given moment. SiamFT addresses this by adaptively weighting the contribution of each modality.

Methods: Adaptive Weighting in Siamese Networks

The SiamFT framework utilizes Fully Convolutional Siamese Networks to process parallel streams of visible (RGB) and infrared (IR) data. In a standard Siamese tracker, the network compares a template against a search region to find the best match. SiamFT extends this by calculating a "reliability" score for each modality, known as the modality response value.

Instead of a static 50/50 split, the weights are dynamic and change frame-by-frame based on the confidence of the tracking prediction in each spectrum. If the visible light response is weak or noisy (e.g., due to glare), its weight is reduced, and the system shifts reliance to the infrared feed.

Analysis of Corrected Visualizations

The corrected figures provide a clear demonstration of how multi-modal fusion handles sensor failure in two distinct types of challenging sequences.

Case 1: The "Hand-off" in Glare

In sequences involving extreme over-exposure, such as a car moving through intense light glare, the model demonstrates a seamless "hand-off" between sensors. As shown in Figure 1, when the car enters a region of extreme glare (around frame #70), the visible modality weight (blue line) crashes toward zero. Simultaneously, the infrared weight (red dashed line) spikes to nearly 1.0, allowing the tracker to maintain its target using thermal data alone.

Figure 1: Adaptive weighting during over-exposure. As the car becomes a white blur in the visible spectrum, the model shifts weight to the infrared modality.

Case 2: Consistent Thermal Dominance in Darkness

In low-light environments where visible features are nearly invisible, the model correctly identifies the infrared sensor as the primary source of truth. As illustrated in Figure 2, the infrared modality (red dashed line) remains the primary source of truth as the visible modality (blue line) struggles with low contrast. The infrared weight stays consistently above 0.8, proving the model's ability to adaptively predict which modality is more trustworthy.

Figure 2: Adaptive weighting in darkness. The infrared modality remains the primary source of truth as the visible modality struggles with low contrast.

Conclusion

The quantitative weight curves in the corrected figures demonstrate the model's sensitivity and robustness. Modality weights are not binary; they fluctuate between 0 and 1, allowing for a "soft" fusion that blends the best of both worlds when both sensors are partially reliable. These corrections confirm the effectiveness of the SiamFT adaptive weighting mechanism in mitigating sensor-specific failures in real-world tracking scenarios.

Summary updated

BibTeX
@article{zhang2019corrections,
  author = {Xingchen Zhang and Ping Ye and Shengyun Peng and Jun Liu and Ke Gong and Gang Xiao},
  title = {{Corrections to "SiamFT: An RGB-Infrared Fusion Tracking Method via Fully Convolutional Siamese Networks"}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2944111},
}
Figure