Abstract
Multi-modal Federated Learning (MFL) often suffers from "modality imbalance," where dominant modalities overshadow weaker ones during local training, leading to biased global models. This paper introduces BMSFed (Balanced Modality Selection for Federated Learning), a framework that treats individual modalities as the primary unit of selection. By integrating a global prototype-based Modal Enhancement (ME) loss with a submodular diversity-driven selection strategy, BMSFed prevents strong modalities from suppressing weak ones. Experimental results across diverse datasets demonstrate that BMSFed significantly improves global model performance and effectively balances multi-modal representations.
Introduction
In Multi-modal Federated Learning (MFL), decentralized clients train models on heterogeneous data types, such as video and audio. While the goal is to leverage all modalities for optimal performance, different modalities often converge at inconsistent rates—a phenomenon termed modality imbalance.
Standard Federated Learning algorithms, like FedAvg, typically select a subset of clients and train on all their available modalities. However, this approach often allows a "strong" modality (e.g., audio in emotion recognition) to dominate the local optimization process, suppressing the learning of "weak" modalities (e.g., video). This results in a global model with severe modal bias, performing well on one data type but failing to utilize others effectively. Research suggests that training a client on only its weaker modality can sometimes benefit the global model more than training on all available data simultaneously.
Methodology: BMSFed
The BMSFed framework addresses modal bias through two primary mechanisms: local enhancement and strategic selection.
1. Local Imbalance Alleviation (ME Loss)
To support lagging modalities during local updates, BMSFed introduces a Modal Enhancement (ME) loss. This loss utilizes "global prototypes"—feature centroids for each class aggregated from the entire client network. By encouraging the local model's weaker modality to produce features aligned with these global consensus anchors, the ME loss effectively "stimulates" the lagging signal and helps it catch up to the dominant modality.
2. Balanced Modality Selection
BMSFed shifts the selection paradigm from entire clients to specific local modalities. The server selects updates based on a submodular optimization problem designed to maximize diversity and balance. The selection includes:
- Multi-modal updates: To capture joint representations.
- Uni-modal updates: Specifically selecting the "weak" modality from certain clients to strengthen its representation in the global model without interference from dominant modalities.
Experimental Evaluation
The framework was evaluated on four multi-modal datasets: CREMA-D (audio-visual emotion recognition), AVE (audio-visual event localization), CG-MNIST (synthetic biased images), and ModelNet40 (3D objects).
Key Results
BMSFed consistently outperformed eight state-of-the-art baselines. As shown in the performance table below, the method achieves significant gains in overall accuracy by specifically addressing the "weak link."
On the CREMA-D dataset, BMSFed reached 64.5% accuracy, a significant jump from FedAvg’s 50.7%. The most dramatic improvement occurred in the visual modality, where accuracy rose from approximately 20% to 42%, while the audio modality remained stable. This demonstrates that BMSFed successfully mitigates modal bias without sacrificing the performance of strong modalities.
Discussion and Insights
- The Uni-modal Advantage: A key insight is that "modality dropout"—intentionally ignoring a strong modality on some clients—can improve global multi-modal performance. This prevents dominant signals from hijacking the learning process.
- Global Prototypes as Anchors: Unlike previous methods using local prototypes, BMSFed’s use of global prototypes allows clients to benefit from collective knowledge, which is vital in non-IID (Independent and Identically Distributed) data scenarios.
- Robustness: BMSFed also proved robust in "modality incongruity" scenarios, where some clients lack certain modalities, outperforming specialized methods like FedMSplit.
Limitations
While effective, BMSFed currently focuses on dual-modality scenarios. Scaling the submodular selection process to three or more modalities may increase computational complexity. Additionally, identifying the "weak" modality requires clients to report local imbalance ratios, introducing a minor communication overhead. Future work will focus on optimizing server-side computation for large-scale networks.
Summary updated