Dense Feature Aggregation and Pruning for RGBT Tracking

Yabin Zhu, Chenglong Li, Bin Luo, Jin Tang, Xiao Wang

Abstract

How to perform effective information fusion of different modalities is a core factor in boosting the performance of RGBT tracking. This paper presents a novel deep fusion algorithm based on the representations from an end-to-end trained convolutional neural network. To deploy the complementarity of features of all layers, we propose a recursive strategy to densely aggregate these features that yield robust representations of target objects in each modality. In different modalities, we propose to prune the densely aggregated features of all modalities in a collaborative way. In a specific, we employ the operations of global average pooling and weighted random selection to perform channel scoring and selection, which could remove redundant and noisy features to achieve more robust feature representation. Experimental results on two RGBT tracking benchmark datasets suggest that our tracker achieves clear state-of-the-art against other RGB and RGBT tracking methods.

1. Introduction

Visual object tracking remains a challenging task in computer vision, particularly in environments characterized by low illumination, heavy fog, or intense glare. To overcome the limitations of standard visible light (RGB) cameras, RGBT tracking incorporates thermal infrared (T) sensors, which provide complementary information based on heat signatures. However, effectively fusing these disparate modalities and managing the resulting feature redundancy are significant hurdles. Traditional methods often rely on handcrafted weights or fuse only high-level semantic features, potentially losing critical spatial details. This paper introduces the Dense feature Aggregation and Pruning Network (DAPNet) to address these challenges through recursive feature fusion and collaborative pruning.

2. Methodology

2.1 Dense Feature Aggregation (DFA)

The DFA module is designed to leverage the complementary information across different network depths. While shallow layers capture fine-grained spatial details and appearance, deep layers provide high-level semantic information. DAPNet employs a recursive strategy to aggregate features from all layers (from shallow to deep) into a unified representation. By using 1x1 convolutions for dimensionality reduction and max-pooling for spatial alignment, the model constructs a rich, multi-scale feature space that enhances target localization and recognition.

2.2 Collaborative Feature Pruning (CFP)

Dense aggregation, while informative, can introduce significant noise and redundancy. To mitigate this, the authors propose the CFP module, which acts as a regularization mechanism during training.

  1. Channel Scoring: Global Average Pooling (GAP) is used to compute an importance score for each feature channel based on its activation state.
  2. Weighted Random Selection (WRS): Instead of deterministic pruning, WRS is used to stochastically inactivate channels during training iterations. This forces the network to learn more diverse and robust features, preventing over-reliance on a subset of "easy" features.
  3. Inference Efficiency: The pruning mechanism is active only during the training phase. During online tracking, all channels are utilized, ensuring no additional computational overhead during real-time inference.

3. Experimental Evaluation

3.1 Datasets and Metrics

The performance of DAPNet was evaluated on two major benchmarks:

  • GTOT (Grayscale-Thermal Object Tracking): Comprising 50 aligned RGBT video pairs.
  • RGBT234: A large-scale dataset with 234 sequences and approximately 234,000 frames, annotated with 12 environmental attributes.

3.2 Quantitative and Qualitative Results

DAPNet achieved state-of-the-art performance on both benchmarks. On the GTOT dataset, it reached a Precision Rate (PR) of 88.2% and a Success Rate (SR) of 70.7%. On the larger RGBT234 dataset, it achieved a PR of 76.6% and an SR of 53.7%.

Qualitative analysis (see Figure 1) demonstrates the tracker's robustness in challenging scenarios such as "Thermal Crossover" (where target and background temperatures are similar) and extreme low-light conditions. The collaborative pruning strategy effectively filters out modality-specific noise, allowing the tracker to maintain a stable bounding box even when one modality is degraded.

Qualitative comparison of DAPNet against baseline methods (MDNet variants) across multiple frames in both RGB and Thermal modalities.

4. Conclusion

This research presents DAPNet, a robust framework for RGBT tracking that utilizes dense feature aggregation and collaborative pruning. By effectively integrating multi-layer features and intelligently managing redundancy, DAPNet sets a new performance standard on major benchmarks. Future work will explore deeper backbone architectures and more precise localization techniques like RoIAlign to further enhance tracking accuracy in real-time applications.

Summary updated

Abstract

How to perform effective information fusion of different modalities is a core factor in boosting the performance of RGBT tracking. This paper presents a novel deep fusion algorithm based on the representations from an end-to-end trained convolutional neural network. To deploy the complementarity of features of all layers, we propose a recursive strategy to densely aggregate these features that yield robust representations of target objects in each modality. In different modalities, we propose to prune the densely aggregated features of all modalities in a collaborative way. In a specific, we employ the operations of global average pooling and weighted random selection to perform channel scoring and selection, which could remove redundant and noisy features to achieve more robust feature representation. Experimental results on two RGBT tracking benchmark datasets suggest that our tracker achieves clear state-of-the-art against other RGB and RGBT tracking methods.
BibTeX
@inproceedings{zhu2019dense,
  author = {Yabin Zhu and Chenglong Li and Bin Luo and Jin Tang and Xiao Wang},
  title = {{Dense Feature Aggregation and Pruning for RGBT Tracking}},
  booktitle = {ACM Multimedia},
  year = {2019},
  url = {https://openreview.net/forum?id=Ochtfw6Na0},
}
Figure