TL;DR
This paper introduces ProFormer, a novel Transformer-based RGBT tracking framework that addresses the challenge of significant appearance differences between RGB and thermal modalities. ProFormer progressively fuses modality-specific information through a multi-stage Transformer architecture and employs a dynamic learning strategy to optimize individual branches, achieving state-of-the-art results on multiple benchmarks.
What Problem Does This Paper Address?
Robust tracking in challenging scenarios like varying illumination, background clutter, and adverse weather is crucial for visual tracking systems. RGBT (RGB-Thermal) tracking leverages the complementary information from both visible and infrared sensors to improve accuracy and robustness. However, a significant challenge in RGBT tracking, especially with Transformer-based methods, is the inherent appearance gap between RGB and thermal images. This gap can limit the feature representation capabilities during modality fusion, leading to suboptimal tracking performance. Existing methods often struggle to effectively integrate modality-specific information into a unified representation or suffer from imbalanced learning between different branches.
Methods
The paper proposes a two-pronged approach: a novel fusion architecture called ProFormer and a Dynamically Guided Learning (DGL) algorithm.
ProFormer is a progressive fusion Transformer designed to gradually integrate single-modality information into a robust multimodal representation. It consists of three stages:
- Multimodal Self-Attention Fusion: Initially, RGB and thermal features are aggregated and then enhanced using a multi-head self-attention mechanism to create a unified multimodal representation.
- Modality-Specific Cross-Attention Fusion: To reactivate submerged modality-specific information, two cross-attention modules are introduced. The fused features act as queries, while unimodal features (RGB and thermal separately) act as keys and values, allowing interaction without direct cross-modal fusion at this stage.
- Multimodal Enhancement Fusion: The outputs from the cross-attention modules are combined and further processed by feed-forward networks to produce the final enhanced multimodal representation.
The Dynamically Guided Learning (DGL) algorithm addresses the issue of imbalanced learning between fused and modality-specific branches. During training, it monitors the classification losses of the fused and unimodal branches. The branch with a lower classification loss (i.e., performing better) is used to guide the learning of the other branches, promoting better representation ability across all branches.
Datasets
The proposed method is evaluated on four public RGBT tracking datasets:
- RGBT210: The first large-scale RGBT dataset.
- RGBT234: An extension of RGBT210.
- LasHeR: A large-scale dataset with a high diversity of sequences.
- VTUAV: A dataset specifically designed for UAV scenarios.
The paper also mentions using datasets like LaSOT, GOT10k, TrackingNet, and MS-COCO for pre-training single-modal data.
Key Insights and Ideas
- Progressive Fusion is Key: The paper highlights that a staged, progressive fusion approach is more effective than direct fusion for RGBT tracking, especially when dealing with significant modality appearance gaps.
- Cross-Attention with Fused Features: Using fused features as queries in cross-attention modules (rather than directly fusing RGB and thermal features) helps preserve modality-specific information while enriching the fused representation.
- Dynamic Learning for Balanced Optimization: The DGL algorithm is a clever way to ensure that all branches (fused and unimodal) are well-optimized, preventing one from dominating or failing to learn effectively. This is crucial for leveraging the full potential of multimodal information.
- Transformer's Strength in RGBT: The work reinforces the idea that Transformers, with their global context modeling capabilities, are well-suited for RGBT tracking, outperforming traditional CNN-based methods.
Key Results
ProFormer achieves state-of-the-art (SOTA) performance on all four evaluated RGBT tracking datasets (RGBT210, RGBT234, LasHeR, and VTUAV). Notably:
- On the LasHeR dataset, ProFormer improves PR/SR scores by +6.2% and +4.1% over existing best trackers.
- The paper demonstrates significant improvements over baseline methods, including a Transformer-based baseline (ToMP-RGBT*), showcasing the effectiveness of both the ProFormer architecture and the DGL algorithm.
- Qualitative results show ProFormer's robustness in challenging scenarios involving similar appearance, low resolution, thermal crossover, and high illumination.
Limitations and Future Work
- Inference Speed: The authors acknowledge that the current inference speed of ProFormer does not reach real-time. This is attributed to the computational cost of extracting bimodal features and the complexity of the Transformer-based fusion models.
- Future Directions:
- Simplifying and optimizing the structure and parameters of fusion models to reduce computation and memory consumption.
- Introducing novel Transformer acceleration methods to improve fusion effect and speed.
- Exploring the use of Transformers for multimodal tracking in general, as it shows significant promise.
Summary updated