RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss

Andong Lu, Chenglong Li, Yuqing Yan, Jin Tang, Bin Luo

TL;DR

The researchers present MANet++, a multi-adapter framework for RGBT (Visible + Thermal) tracking that effectively balances shared and modality-specific information. By using a novel Hierarchical Divergence (HD) loss and a parallel adapter architecture, the tracker achieves state-of-the-art accuracy while running significantly faster than previous iterations.

What Problem Does This Paper Address?

Object tracking using both RGB (visible) and Thermal (infrared) data is powerful because the modalities are complementary: RGB provides rich texture and color, while Thermal works in total darkness and handles shadows well.

However, existing RGBT trackers struggle with two extremes:

  1. Two-stream networks: These use separate branches for each modality, often leading to massive parameter redundancy and ignoring the fact that RGB and Thermal images share common structures (like object boundaries).
  2. Single-stream networks: These use the same parameters for both, which models shared information well but fails to capture the unique "heterogeneous" properties of each sensor.

The authors aim to bridge this gap by creating a network that extracts shared features, modality-specific features, and instance-specific features simultaneously without exploding the parameter count.

Methods

The core of the paper is the Multi-Adapter Network (MANet++), which consists of three specialized "adapters":

  • Generality Adapter (GA): A shared backbone (modified VGG-M) that extracts features common to both modalities, such as shapes and layouts.
  • Modality Adapter (MA): Small, parallel blocks added to each layer of the GA. These use very few parameters (only ~20% of the GA) to capture the unique nuances of RGB vs. Thermal data.
  • Instance Adapter (IA): A module that adapts to the specific object being tracked. It uses a dynamic fusion scheme to weight the importance of each modality based on the current scene (e.g., giving more weight to Thermal if the RGB frame is blurry or dark).

To make these adapters work, the authors introduced a Hierarchical Divergence (HD) loss. Based on Multiple Kernel Maximum Mean Discrepancy (MK-MMD), this loss "pushes" the network to minimize the difference between modalities in the GA (forcing it to learn shared traits) while maximizing the difference in the MA (forcing it to learn unique traits).

Finally, they integrated an adaptive RoIAlign layer, which allows the network to extract features for all candidate target samples from a single feature map, drastically improving computational efficiency.

Datasets

The authors evaluated MANet++ on three major RGBT tracking benchmarks:

  • GTOT: 50 video sequences of spatially and temporally aligned RGB and Thermal pairs.
  • RGBT234: A large-scale dataset with 234 sequences and ~200,000 frames, featuring challenges like motion blur and occlusion.
  • VOT-RGBT2019: A dataset from the Visual Object Tracking challenge consisting of 60 sequences.

Key Insights and Ideas

  • The Power of Parallelism: Instead of building two giant networks, the authors show that "plugging in" tiny modality-specific adapters into a shared backbone is sufficient. This keeps the model lightweight while maintaining high discriminative power.
  • Loss-Driven Feature Separation: The HD loss is the "secret sauce." Without it, the network might not naturally separate shared vs. specific information. By explicitly rewarding divergence in the Modality Adapters, they ensure the sensors aren't just duplicating each other's work.
  • Efficiency Matters: By switching to RoIAlign, the authors achieved an 8x speedup over their previous conference version (MANet) on the GTOT dataset, proving that architectural tweaks for speed don't have to come at the cost of accuracy.

Key Results

  • State-of-the-Art Performance: MANet++ achieved a Precision Rate (PR) of 80.0% and a Success Rate (SR) of 55.4% on the RGBT234 dataset, outperforming all other compared RGBT and RGB-only trackers.
  • Robustness: The tracker showed significant improvements in challenging scenarios like "Low Resolution," "Partial Occlusion," and "Thermal Crossover" (where the target's temperature matches the background).
  • Speed: On the GTOT dataset, MANet++ reached 27.3 FPS, making it much closer to real-time performance than many competing deep-learning-based trackers.

Limitations and Future Work

The authors note that while RoIAlign significantly boosts speed, it may lead to a "weak representation ability" for very low-resolution objects because the feature maps are pooled into a fixed size (e.g., 3x3). This was evident in the VOT-RGBT2019 results, where a version without RoIAlign actually performed slightly better on accuracy but was much slower.

For future work, the authors suggest exploring even deeper backbones (like ResNet) and incorporating additional modalities, such as depth or near-infrared data, into the multi-adapter framework.

Summary updated

Abstract

RGBT tracking has attracted increasing attention since RGB and thermal infrared data have strong complementary advantages, which could make trackers all-day and all-weather work. Existing works usually focus on extracting modality-shared or modality-specific information, but the potentials of these two cues are not well explored and exploited in RGBT tracking. In this paper, we propose a novel multi-adapter network to jointly perform modality-shared, modality-specific and instance-aware target representation learning for RGBT tracking. To this end, we design three kinds of adapters within an end-to-end deep learning framework. In specific, we use the modified VGG-M as the generality adapter to extract the modality-shared target representations. To extract the modality-specific features while reducing the computational complexity, we design a modality adapter, which adds a small block to the generality adapter in each layer and each modality in a parallel manner. Such a design could learn multilevel modality-specific representations with a modest number of parameters as the vast majority of parameters are shared with the generality adapter. We also design instance adapter to capture the appearance properties and temporal variations of a certain target. Moreover, to enhance the shared and specific features, we employ the loss of multiple kernel maximum mean discrepancy to measure the distribution divergence of different modal features and integrate it into each layer for more robust representation learning. Extensive experiments on two RGBT tracking benchmark datasets demonstrate the outstanding performance of the proposed tracker against the state-of-the-art methods.
BibTeX
@article{lu2021rgbt,
  author = {Andong Lu and Chenglong Li and Yuqing Yan and Jin Tang and Bin Luo},
  title = {{RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=cLuLQ0ah1ik},
}

Loading related content...

Figure