Online Non-Negative Multi-Modality Feature Template Learning for RGB-Assisted Infrared Tracking

Xiangyuan Lan, Mang Ye, Rui Shao, Bineng Zhong, Deepak Kumar Jain, Huiyu Zhou

Abstract

Infrared (IR) sensors are critical for night-time surveillance but face significant challenges during thermal crossover events, where target and background temperatures converge. This paper proposes an online non-negative multi-modality feature template learning framework that integrates RGB data to enhance IR tracking. By enforcing non-negative constraints, the model learns robust part-based representations. An adaptive weighting mechanism dynamically prioritizes the most reliable modality, while Huber loss is utilized to mitigate the impact of contaminated data. Evaluations demonstrate that the proposed method achieves a success rate of 0.9, substantially outperforming existing state-of-the-art trackers.

Introduction

Infrared imaging is a staple of modern surveillance due to its ability to detect thermal signatures in low-light conditions. However, IR tracking is often compromised by thermal crossover—a phenomenon where the target's heat signature matches the background, causing the object to become indistinguishable. While multi-modality systems (RGB-IR) offer a solution, simple data fusion often fails in the presence of occlusions, cluttered backgrounds, and sensor noise. There is a critical need for a tracking framework that can intelligently weigh sensor reliability and learn robust object features in real-time.

Methodology

The proposed framework addresses these challenges through three core innovations:

Non-Negative Part-Based Learning

Unlike standard models that use abstract global features, this method applies Non-Negative Matrix Factorization (NMF) logic. By enforcing non-negative constraints, the tracker is forced to represent objects as a "sum of parts" (e.g., head, torso, limbs). This part-based approach ensures that if one part of the target is occluded, the remaining visible parts can maintain a stable track.

Adaptive Modality Weighting

The system does not treat RGB and IR data as equally reliable. Instead, it calculates a dynamic importance weight ($\alpha$) for each modality. When the IR signal degrades—such as during thermal crossover—the model automatically shifts its trust toward the RGB spectrum, ensuring continuous tracking.

Robust Decontamination with Huber Loss

To prevent the model from "learning" background noise or occlusions, the authors utilize the Huber loss function. While an explicit error term ($E$) was initially considered for noise handling, the Huber loss formulation provided equivalent "decontamination" with significantly lower computational overhead, streamlining the optimization process.

Experimental Results

The framework was evaluated on 15 aligned RGB-IR video pairs featuring challenges like scale variation, poor illumination, and thermal crossover. The proposed method demonstrated superior performance across key metrics:

  • Overlapping Rate: Achieved a score of 0.7, compared to 0.58 for the next best competitor (L1).
  • Success Rate: Reached 0.9, a significant improvement over the 0.69 achieved by state-of-the-art multi-modality trackers.
Tracking performance in challenging infrared scenarios demonstrating robustness to noise and low contrast.

Qualitative analysis shows that while competing trackers often drift toward the background during crossover events, the proposed model maintains a precise bounding box by leveraging its adaptive weighting and part-based templates.

Conclusion

This research introduces a robust multi-modality tracking framework that effectively mitigates the limitations of infrared sensors. By combining non-negative part-based learning with adaptive sensor weighting, the system achieves high accuracy in complex environments. Future work will focus on optimizing the iterative algorithms to achieve real-time processing speeds through hardware acceleration and algorithmic streamlining.

Summary updated

Abstract

Infrared sensors have been deployed in many video surveillance systems because of the insensibility of their imaging procedure to some extreme conditions (e.g. low illumination condition, dim environment). To reduce human labor in video monitoring and perform intelligent infrared video understanding, an important issue we need to consider is how to locate the object of interest in consecutive video frames accurately. Therefore, developing a robust object tracking algorithm for infrared videos is necessary. However, the infrared information may not be reliable (e.g. thermal crossover), and appearance modeling with only the infrared modality may not be able to achieve good results. To address these issues, with the wide deployment of RGB-infrared camera systems, this paper proposes an infrared tracking framework in which information from RGB-modality will be exploited to assist the infrared object tracking. Specifically, within the tracking framework, in order to deal with the contaminated features caused by large appearance variations, an online non-negative feature template learning model is designed. The non-negative constraint enables the model to capture the local part-based characteristic of the target appearance. To ensure more important modality contribute more in appearance representation, an adaptive modality importance weight learning scheme is also incorporated in the proposed feature learning model. To guarantee the model optimality, an iterative optimization algorithm is derived. The experimental results on various RGB-infrared videos show the effectiveness of the proposed method.
BibTeX
@article{lan2019online,
  author = {Xiangyuan Lan and Mang Ye and Rui Shao and Bineng Zhong and Deepak Kumar Jain and Huiyu Zhou},
  title = {{Online Non-Negative Multi-Modality Feature Template Learning for RGB-Assisted Infrared Tracking}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2916895},
}
Figure