Fed-MAPTrack: Federated Modality-aware Prompt Tuning for RGB-X Tracking

Xiangyu Liu, Yijie Shen, Yue Yin, Zhitong Yin, Yongqi Pan, Wentao FanarXiv 2026 · arXiv

TL;DR

Fed-MAPTrack is a new federated learning framework designed for multimodal object tracking (RGB-X) that effectively handles "sensor-poor" (RGB-only) and "sensor-rich" (RGB + Thermal/Depth/Event) clients. By combining modality-aware prompt tuning with a selective aggregation strategy, it allows all clients to contribute to a shared RGB model while ensuring that modality-specific fusion parameters are only updated by clients with the relevant hardware.

What Problem Does This Paper Address?

Visual object tracking is significantly more robust when RGB data is supplemented by auxiliary sensors like thermal cameras (for low light), depth sensors (for 3D structure), or event cameras (for high-speed motion). However, training these "RGB-X" trackers typically requires centralized access to massive multimodal datasets, which raises significant privacy and bandwidth concerns.

Federated Learning (FL) offers a solution by keeping data local and only exchanging model updates. But standard FL assumes all clients are "homogeneous"—meaning they all have the same sensors. In reality, a network might consist of high-end drones with thermal sensors and budget smartphones with only RGB cameras. If an RGB-only client tries to update the "thermal fusion" part of a model, it introduces noise that degrades performance for everyone. The challenge is: How do we leverage the useful RGB data from every client without letting "sensor-poor" clients corrupt the modality-specific parts of the model?

Methods

The authors propose Fed-MAPTrack, which stands for Federated Modality-aware Prompt Tuning. Its architecture relies on three main pillars:

  1. Prompt-Based Tuning: Instead of updating the entire massive tracking model (which would be slow and bandwidth-heavy), the framework freezes a pre-trained Vision Transformer (ViT) backbone and prediction head. It only trains and exchanges small "prompt" parameters that adapt the model to specific modalities.
  2. Modality-Aware Bucketing: The server maintains a global bank for RGB-shared prompts and separate "modality banks" for Thermal, Depth, and Event sensors. These banks store parameters for auxiliary features, cross-modal fusion, and normalization.
  3. Selective Participation (The S-A Gate): This is the core innovation. While all clients contribute to the global RGB bank, the server uses a "Selective Aggregation" gate to filter updates. Only clients equipped with the actual auxiliary sensors are allowed to contribute to the modality-specific fusion and normalization banks.
Figure 2: Overall framework of Fed-MAPTrack. Clients keep training videos local and adapt prompt-related parameters on a frozen tracking backbone and prediction head. RGB-shared parameters use one global bank, while auxiliary-specific, fusion, and normalization parameters use modality banks. Server aggregation combines this sharing structure with sensor-dependent contributor eligibility.

Datasets

The framework was evaluated on three major RGB-X tracking benchmarks:

  • LasHeR: For RGB-Thermal (RGB-T) tracking.
  • VisEvent: For RGB-Event (RGB-E) tracking.
  • DepthTrack: For RGB-Depth (RGB-D) tracking.

To simulate a realistic heterogeneous environment, the researchers partitioned these datasets so that each benchmark was handled by a mix of three sensor-equipped clients and one RGB-only client.

Key Insights and Ideas

  • RGB is the Universal Language: Even if a device lacks a thermal sensor, its RGB video is still highly valuable for training the core appearance model. Fed-MAPTrack ensures this data isn't wasted.
  • Bucketing Alone is Insufficient: The authors found that simply creating separate buckets for different sensors (e.g., a "thermal bucket") actually decreases performance if you don't also restrict who can contribute to them. Without the "S-A Gate," RGB-only clients send "empty" updates that wash out the learning from sensor-rich clients.
  • Joint Design is Critical: The paper emphasizes that the sharing domain (where parameters go) and contributor eligibility (who sends them) must be designed as a single, unified strategy.
Figure 3: Unified aggregation design. RGB-shared parameters use the global bank, while auxiliary-specific, fusion, and normalization parameters use thermal, depth, and event banks. The participation gate retains valid RGB updates from RGB- only clients but excludes their modality-related contributions from server bucket averages. Dispatch and local trainability are distinct from server eligibility.

Key Results

Fed-MAPTrack consistently outperformed standard Federated Averaging (FedAvg) and several state-of-the-art federated baselines:

  • Performance Gains: Compared to FedAvg, it improved scores by 2.35 AUC points on LasHeR, 3.45 SR points on VisEvent, and 2.11 F-score points on DepthTrack.
  • Superiority over Personalization: It beat generic personalized FL methods (like FedPer and Ditto), proving that explicitly modeling the sensor modalities is more effective than just letting each client "do its own thing."
  • Efficiency: By only exchanging prompt parameters, the framework maintains a small communication footprint (roughly 3.36 MB per client per round) compared to full-model exchange.

Limitations and Future Work

  • Static Sensor Labels: The current system assumes a client's sensor configuration is fixed. It doesn't yet account for "dynamic" conditions, such as a sensor that becomes noisy or fails temporarily during a mission.
  • Formal Privacy: While FL keeps raw video local, the authors note that parameter exchange alone isn't a formal privacy guarantee. Future iterations might incorporate Differential Privacy or Secure Multi-Party Computation.
  • Evaluation Scope: The study used a fixed client partition. Testing across a wider variety of data distributions and "RGB-only" ratios would further validate the framework's generalizability.

Summary updated

Abstract

RGB-X tracking complements RGB appearance with thermal, depth, or event observations, but centralized training requires collecting multimodal videos at a single location. Federated training keeps raw videos local, but sensor heterogeneity can create a mismatch between parameter-sharing domains and the clients contributing to them. This mismatch is particularly relevant when RGB-only clients coexist with sensor-equipped clients: their RGB observations remain useful, but they provide no auxiliary measurements for learning sensor-dependent interactions. We propose Fed-MAPTrack, a modality-aware federated prompt tuning framework for RGB-X tracking. The pretrained tracking backbone and prediction head are frozen, and clients optimize and exchange only prompt-related parameters. The server aggregates RGB-shared parameters globally and maintains separate thermal, event, and depth banks for auxiliary-specific, fusion, and normalization parameters. Selective participation retains RGB-only clients' contributions to the global RGB bank while excluding their fusion and normalization updates from modality-specific aggregation. Across three matched training repetitions under a fixed client partition, Fed-MAPTrack improves the mean FedAvg scores by 2.35 AUC points on LasHeR, 3.45 SR points on VisEvent, and 2.11 F-score points on DepthTrack. Among the nine evaluated methods, it achieves the highest mean thermal AUC and depth F-score, while its mean event SR is 0.07 points below that of Independent FL. Controlled ablations show that extending the bucketing scope without a compatible participation rule can reduce performance. These results highlight the need to jointly define the sharing domain and eligible contributors for each parameter group.
BibTeX
@misc{liu2026fed,
  author = {Xiangyu Liu and Yijie Shen and Yue Yin and Zhitong Yin and Yongqi Pan and Wentao Fan},
  title = {{Fed-MAPTrack: Federated Modality-aware Prompt Tuning for RGB-X Tracking}},
  year = {2026},
  eprint = {2610.34939},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34939},
}
Figure