TL;DR
Fed-MAPTrack is a new federated learning framework designed for multimodal object tracking (RGB-X) that effectively handles "sensor-poor" (RGB-only) and "sensor-rich" (RGB + Thermal/Depth/Event) clients. By combining modality-aware prompt tuning with a selective aggregation strategy, it allows all clients to contribute to a shared RGB model while ensuring that modality-specific fusion parameters are only updated by clients with the relevant hardware.
What Problem Does This Paper Address?
Visual object tracking is significantly more robust when RGB data is supplemented by auxiliary sensors like thermal cameras (for low light), depth sensors (for 3D structure), or event cameras (for high-speed motion). However, training these "RGB-X" trackers typically requires centralized access to massive multimodal datasets, which raises significant privacy and bandwidth concerns.
Federated Learning (FL) offers a solution by keeping data local and only exchanging model updates. But standard FL assumes all clients are "homogeneous"—meaning they all have the same sensors. In reality, a network might consist of high-end drones with thermal sensors and budget smartphones with only RGB cameras. If an RGB-only client tries to update the "thermal fusion" part of a model, it introduces noise that degrades performance for everyone. The challenge is: How do we leverage the useful RGB data from every client without letting "sensor-poor" clients corrupt the modality-specific parts of the model?
Methods
The authors propose Fed-MAPTrack, which stands for Federated Modality-aware Prompt Tuning. Its architecture relies on three main pillars:
- Prompt-Based Tuning: Instead of updating the entire massive tracking model (which would be slow and bandwidth-heavy), the framework freezes a pre-trained Vision Transformer (ViT) backbone and prediction head. It only trains and exchanges small "prompt" parameters that adapt the model to specific modalities.
- Modality-Aware Bucketing: The server maintains a global bank for RGB-shared prompts and separate "modality banks" for Thermal, Depth, and Event sensors. These banks store parameters for auxiliary features, cross-modal fusion, and normalization.
- Selective Participation (The S-A Gate): This is the core innovation. While all clients contribute to the global RGB bank, the server uses a "Selective Aggregation" gate to filter updates. Only clients equipped with the actual auxiliary sensors are allowed to contribute to the modality-specific fusion and normalization banks.
Datasets
The framework was evaluated on three major RGB-X tracking benchmarks:
- LasHeR: For RGB-Thermal (RGB-T) tracking.
- VisEvent: For RGB-Event (RGB-E) tracking.
- DepthTrack: For RGB-Depth (RGB-D) tracking.
To simulate a realistic heterogeneous environment, the researchers partitioned these datasets so that each benchmark was handled by a mix of three sensor-equipped clients and one RGB-only client.
Key Insights and Ideas
- RGB is the Universal Language: Even if a device lacks a thermal sensor, its RGB video is still highly valuable for training the core appearance model. Fed-MAPTrack ensures this data isn't wasted.
- Bucketing Alone is Insufficient: The authors found that simply creating separate buckets for different sensors (e.g., a "thermal bucket") actually decreases performance if you don't also restrict who can contribute to them. Without the "S-A Gate," RGB-only clients send "empty" updates that wash out the learning from sensor-rich clients.
- Joint Design is Critical: The paper emphasizes that the sharing domain (where parameters go) and contributor eligibility (who sends them) must be designed as a single, unified strategy.
Key Results
Fed-MAPTrack consistently outperformed standard Federated Averaging (FedAvg) and several state-of-the-art federated baselines:
- Performance Gains: Compared to FedAvg, it improved scores by 2.35 AUC points on LasHeR, 3.45 SR points on VisEvent, and 2.11 F-score points on DepthTrack.
- Superiority over Personalization: It beat generic personalized FL methods (like FedPer and Ditto), proving that explicitly modeling the sensor modalities is more effective than just letting each client "do its own thing."
- Efficiency: By only exchanging prompt parameters, the framework maintains a small communication footprint (roughly 3.36 MB per client per round) compared to full-model exchange.
Limitations and Future Work
- Static Sensor Labels: The current system assumes a client's sensor configuration is fixed. It doesn't yet account for "dynamic" conditions, such as a sensor that becomes noisy or fails temporarily during a mission.
- Formal Privacy: While FL keeps raw video local, the authors note that parameter exchange alone isn't a formal privacy guarantee. Future iterations might incorporate Differential Privacy or Secure Multi-Party Computation.
- Evaluation Scope: The study used a fixed client partition. Testing across a wider variety of data distributions and "RGB-only" ratios would further validate the framework's generalizability.
Summary updated