Abstract
In challenging situations, such as low illumination, rain, and background clutter, the stability of the thermal infrared (TIR) spectrum can help red, green, blue (RGB) visible spectrum to improve tracking performance. However, the high-level image information and the modality-specific features have not been sufficiently studied. The proposed correlation filter uses the fused saliency content map to improve filter training and extracts different features of modalities. The fused content map is introduced into the spatial regularization term of correlation filter to highlight the training samples in the content region. Furthermore, the fused content map can avoid the incompleteness of the content region caused by challenging situations. Additionally, different features are extracted according to the modality characteristics and are fused by the designed response-level fusion strategy. The alternating direction method of multipliers (ADMM) algorithm is used to solve the tracker training efficiently. Experiments on the large-scale benchmark datasets show the effectiveness of the proposed tracker compared to the state-of-the-art traditional trackers and the deep learning based trackers.
Introduction: Seeing Through the Dark
Visual tracking is a cornerstone of computer vision, but standard RGB-only trackers often fail when the lights go out or the background becomes cluttered. Thermal infrared (TIR) sensors provide a solution by capturing heat signatures, which remain stable regardless of illumination. However, TIR images lack the rich texture and color information found in RGB.
While "RGB-T" tracking (combining both modalities) exists, current methods face two major hurdles:
- The Boundary Effect: Traditional Correlation Filters (CF) suffer from periodic boundary artifacts that degrade tracking accuracy.
- Suboptimal Fusion: Many trackers treat RGB and TIR data identically, failing to exploit the unique strengths of each modality (e.g., trying to extract "color" features from a grayscale thermal image).
Methodology: Content-Aware Fusion
The authors build upon the Spatially Regularized Discriminative Correlation Filter (SRDCF) and introduce several key innovations to address these hurdles.
Fused Saliency Content Map
Instead of using a fixed spatial penalty to handle boundary effects, the authors generate saliency maps for both RGB and TIR frames. These are fused into a "content map" ($w_f$) that tells the tracker exactly where the target is likely to be. This map is then used to dynamically adjust the spatial regularization, forcing the filter to focus on the target's actual shape rather than a generic central box.
Modality-Specific Features and Fusion
Recognizing that different sensors capture different information, the team uses tailored feature sets:
- RGB: Histogram of Oriented Gradients (HOG) + Color Names (CN).
- TIR: HOG + Histogram of Local Intensity (HI).
The tracker generates separate response maps for each feature. These are adaptively weighted and fused based on their maximum values, allowing the tracker to automatically rely more on the more "confident" modality in a given frame. To keep the tracker fast enough for practical use, the authors use the Alternating Direction Method of Multipliers (ADMM) to solve the filter training equations.
Key Insights
- Saliency as a Regularizer: Using high-level image content (saliency) to solve a low-level signal processing problem (boundary effects) makes the spatial regularization "content-aware," significantly improving robustness to deformations and partial occlusions.
- Complementary Incompleteness: Challenging conditions might cause a saliency map to be incomplete in one modality. By fusing the maps, the "complete" information from one side fills the gap in the other.
- Efficiency Matters: This paper shows that hand-crafted features, when fused intelligently and optimized with ADMM, can still compete with—and sometimes beat—deep neural networks in specific tracking tasks.
Experimental Results
The researchers evaluated their tracker (OursCAFF) on major benchmarks including GTOT, RGBT210, RGBT234, and VOT-RGBT2019.
The proposed tracker demonstrated impressive gains:
- Superior Accuracy: On the GTOT dataset, it achieved a Success Rate (SR) of 0.726 and a Precision Rate (PR) of 0.852, outperforming established trackers like ECO and BACF.
- Robustness to Challenges: In "Thermal Crossover" scenarios, the tracker maintained high precision by relying on RGB texture. Conversely, in "Low Illumination," it successfully pivoted to TIR data.
- Competitive with Deep Learning: In the VOT-RGBT2019 benchmark, OursCAFF achieved an Accuracy (A) of 0.569, surpassing several deep learning-based competitors like FANet and RT-MDNet.
Conclusion and Future Work
While the hand-crafted features are efficient, they may have reached a performance ceiling compared to deep features learned directly from data. Future directions include integrating multi-resolution deep features into the content-aware framework and exploring combinations with Siamese network architectures to further boost tracking stability and speed.
Summary updated