Dense Feature Aggregation and Pruning for RGBT Tracking

Yabin Zhu, Chenglong Li, Bin Luo, Jin Tang, Xiao Wang

TL;DR

Researchers from Anhui University have developed DAPNet, a deep learning framework that significantly boosts RGBT (RGB + Thermal) tracking performance. By recursively aggregating features from all network layers and applying a collaborative pruning strategy to eliminate noise, the tracker achieves state-of-the-art results on major benchmarks while maintaining a robust representation of target objects.

What Problem Does This Paper Address?

Visual tracking is difficult in "edge case" environments—think total darkness, heavy fog, or intense glare. RGBT tracking attempts to solve this by combining standard visible light (RGB) cameras with thermal infrared (T) sensors.

However, two major hurdles remain:

  1. Effective Fusion: How do you combine these two very different data streams? Most methods either use handcrafted weights or only fuse high-level semantic features, losing the fine-grained spatial details needed for precise localization.
  2. Feature Redundancy: Fusing multiple modalities often creates a "noisy" feature space. Many convolutional filters become redundant or even distracting, leading to overfitting and reduced tracking accuracy.

Methods

The authors propose the Dense feature Aggregation and Pruning Network (DAPNet), which consists of two primary innovations:

1. Dense Feature Aggregation (DFA)

Instead of just looking at the final layer of a neural network, DAPNet uses a recursive strategy to "stack" features from every layer (from shallow to deep).

  • Shallow layers provide spatial details and appearance, which are essential for pinpointing the exact location of a target.
  • Deep layers provide semantic information, helping the tracker understand what the object is, even if its appearance changes. By using 1x1 convolutions and max-pooling to align these features into a single space, the model creates a much richer representation of the target.

2. Collaborative Feature Pruning (CFP)

To handle the noise introduced by dense aggregation, the authors designed a pruning module inspired by dropout techniques.

  • Scoring: They use Global Average Pooling (GAP) to determine the "activation state" or importance of each feature channel.
  • Selection: Instead of just cutting the lowest-scoring channels, they use Weighted Random Selection (WRS). This allows the network to "inactivate" certain channels during each training iteration, forcing the model to learn more robust and diverse features.
  • Efficiency: Crucially, this pruning is only active during training. During online tracking (inference), the pruning is removed, meaning there is no extra computational overhead when it matters most.

Datasets

The model was evaluated on the two most prominent RGBT tracking benchmarks:

  • GTOT (Grayscale-Thermal Object Tracking): 50 aligned RGBT video pairs.
  • RGBT234: A large-scale dataset featuring 234 video sequences with approximately 234,000 frames, annotated with 12 different attributes (like occlusion, low illumination, and thermal crossover).

Key Insights and Ideas

  • The "More is More" Fallacy: Simply concatenating RGB and Thermal data (as seen in their MDNet+RGBT1 baseline) doesn't necessarily help. In fact, it can introduce noise that degrades performance. Intelligent fusion is mandatory.
  • Backbone Sharing: To reduce parameters and capture common properties between light and heat, the RGB and Thermal branches share the same backbone parameters.
  • Pruning as Regularization: The pruning module acts as a powerful regularizer. By randomly selecting channels based on their importance scores, the network avoids becoming overly reliant on a few "easy" features, making it more resilient to challenging environmental changes.

Key Results

DAPNet outperformed existing state-of-the-art (SOTA) trackers on both datasets:

  • On GTOT: It achieved a Precision Rate (PR) of 88.2% and a Success Rate (SR) of 70.7%, significantly outperforming the base MDNet and other RGBT-specific trackers like SGT.
  • On RGBT234: It reached a PR of 76.6% and an SR of 53.7%.
  • Attribute Performance: The tracker showed particular strength in handling "Thermal Crossover" (where the target and background have similar temperatures) and "Low Illumination," proving the effectiveness of its multi-modal fusion.

Limitations and Future Work

While DAPNet is highly effective, the authors note a few areas for growth:

  • Backbone Depth: The current model uses a relatively light VGG-M backbone. The authors suggest that moving to deeper or wider architectures (like ResNet or Siamese-based networks) could further boost performance.
  • Localization Precision: They propose integrating more advanced localization techniques, such as RoIAlign, to extract even more accurate target representations in future iterations.
  • Real-time Constraints: While the pruning is removed during inference to save time, the dense aggregation still adds complexity. Future work may focus on further optimizing the architecture for high-speed real-time applications.

Summary updated

Abstract

How to perform effective information fusion of different modalities is a core factor in boosting the performance of RGBT tracking. This paper presents a novel deep fusion algorithm based on the representations from an end-to-end trained convolutional neural network. To deploy the complementarity of features of all layers, we propose a recursive strategy to densely aggregate these features that yield robust representations of target objects in each modality. In different modalities, we propose to prune the densely aggregated features of all modalities in a collaborative way. In a specific, we employ the operations of global average pooling and weighted random selection to perform channel scoring and selection, which could remove redundant and noisy features to achieve more robust feature representation. Experimental results on two RGBT tracking benchmark datasets suggest that our tracker achieves clear state-of-the-art against other RGB and RGBT tracking methods.
BibTeX
@inproceedings{zhu2019dense,
  author = {Yabin Zhu and Chenglong Li and Bin Luo and Jin Tang and Xiao Wang},
  title = {{Dense Feature Aggregation and Pruning for RGBT Tracking}},
  booktitle = {ACM Multimedia},
  year = {2019},
  url = {https://openreview.net/forum?id=Ochtfw6Na0},
}

Loading related content...

Figure