Introduction
The pursuit of Artificial General Intelligence (AGI) necessitates the training of large-scale multimodal foundation models. However, the centralization of the vast, sensitive datasets required for such training poses significant privacy risks. Federated Learning (FL) has emerged as a promising paradigm to address these concerns by enabling collaborative model training across decentralized data sources without raw data exchange.
Despite its potential, traditional FL assumes a degree of symmetry—where clients perform similar tasks on similar data types—that is rarely present in the heterogeneous landscape of AGI. Researchers from the University of Virginia, in their AAAI-24 paper, introduce DisentAFL, a novel framework designed to handle the "messy" reality of Modality-Task Agnostic Federated Learning (AFL).
The Challenge of Asymmetry
In a truly agnostic FL environment, clients are highly diverse. For instance, one client might utilize cameras for image classification, while another uses audio sensors for speech translation. This diversity leads to Asymmetrical Knowledge Relationships (AKR), characterized by four primary types of heterogeneity (MTDC):
- Modality gaps: Disparities in input types (e.g., Video vs. Audio).
- Task gaps: Differences in objectives (e.g., Classification vs. Translation).
- Domain shifts: Environmental variations (e.g., Daytime vs. Nighttime driving).
- Concept drifts: Divergent interpretations of labels.
Standard FL methods typically rely on model averaging, which assumes that knowledge shared between clients is mutually beneficial. In the presence of AKR, however, this approach often results in negative transfer, where irrelevant or conflicting information from one client degrades the performance of another.
DisentAFL: A Two-Stage Solution
To mitigate negative transfer, DisentAFL employs a Knowledge Disentanglement and Gating (KDG) mechanism. This approach untangles shared, beneficial knowledge from client-specific noise through two distinct stages:
1. Coarse-grained Disentanglement
The framework first groups clients based on shared architectural requirements. Clients with similar input modalities (e.g., all image-based sensors) share specific encoder structures, even if their downstream tasks differ. This initial grouping simplifies the complex global problem into more manageable, symmetrical sub-problems.
2. Fine-grained Disentanglement
The core of DisentAFL lies in its server-side "supernetwork," which hosts specialized modules:
- Mixture of Domain Experts (MoDE): Captures environment-specific knowledge.
- Mixture of Task/Modality Experts (MoTE/MoME): Captures the specific interactions between different tasks and modalities.
- Routing Gates (IoGate and DomGate): These gates act as intelligent controllers, dynamically routing each client to the specific "experts" in the supernetwork that align with their local sensors and goals.
By incorporating an orthogonal regularization loss, DisentAFL ensures that private local knowledge and shared global knowledge remain distinct, preventing mutual interference.
Experimental Evaluation
The researchers evaluated DisentAFL across six rigorous simulation scenarios constructed from seven diverse datasets, including image classification, bimodal driving (Vehicle-xM), and multimedia emotion recognition.
In the highly complex MERGE-VM scenario—which involves cross-modal, multi-task learning—DisentAFL achieved an accuracy of 96.4%. This significantly outperformed standard FedAvg (84.6%) and advanced Personalized FL methods like Cross-PFL (90.1%). Ablation studies further confirmed that both stages of disentanglement are critical, with their removal leading to a 5-8% drop in performance.
Conclusion
DisentAFL represents a significant advancement in making Federated Learning viable for AGI. By acknowledging and addressing the inherent asymmetry in multimodal, multi-task environments, it enables efficient knowledge transfer while shielding clients from harmful noise. While the current reliance on a server-side supernetwork and pre-defined domains presents areas for future optimization, the framework provides a robust foundation for privacy-preserving, large-scale multimodal learning.
Summary updated