On Disentanglement of Asymmetrical Knowledge Transfer for Modality-Task Agnostic Federated Learning

Jiayi Chen, Aidong ZhangAAAI 2024 · DOI

Introduction

The pursuit of Artificial General Intelligence (AGI) necessitates the training of large-scale multimodal foundation models. However, the centralization of the vast, sensitive datasets required for such training poses significant privacy risks. Federated Learning (FL) has emerged as a promising paradigm to address these concerns by enabling collaborative model training across decentralized data sources without raw data exchange.

Despite its potential, traditional FL assumes a degree of symmetry—where clients perform similar tasks on similar data types—that is rarely present in the heterogeneous landscape of AGI. Researchers from the University of Virginia, in their AAAI-24 paper, introduce DisentAFL, a novel framework designed to handle the "messy" reality of Modality-Task Agnostic Federated Learning (AFL).

Overview of the Modality-Task Agnostic Federated Learning (AFL) framework.

The Challenge of Asymmetry

In a truly agnostic FL environment, clients are highly diverse. For instance, one client might utilize cameras for image classification, while another uses audio sensors for speech translation. This diversity leads to Asymmetrical Knowledge Relationships (AKR), characterized by four primary types of heterogeneity (MTDC):

  1. Modality gaps: Disparities in input types (e.g., Video vs. Audio).
  2. Task gaps: Differences in objectives (e.g., Classification vs. Translation).
  3. Domain shifts: Environmental variations (e.g., Daytime vs. Nighttime driving).
  4. Concept drifts: Divergent interpretations of labels.

Standard FL methods typically rely on model averaging, which assumes that knowledge shared between clients is mutually beneficial. In the presence of AKR, however, this approach often results in negative transfer, where irrelevant or conflicting information from one client degrades the performance of another.

Comparison between Symmetrical and Asymmetrical Knowledge Relationships in Federated Learning.

DisentAFL: A Two-Stage Solution

To mitigate negative transfer, DisentAFL employs a Knowledge Disentanglement and Gating (KDG) mechanism. This approach untangles shared, beneficial knowledge from client-specific noise through two distinct stages:

1. Coarse-grained Disentanglement

The framework first groups clients based on shared architectural requirements. Clients with similar input modalities (e.g., all image-based sensors) share specific encoder structures, even if their downstream tasks differ. This initial grouping simplifies the complex global problem into more manageable, symmetrical sub-problems.

2. Fine-grained Disentanglement

The core of DisentAFL lies in its server-side "supernetwork," which hosts specialized modules:

  • Mixture of Domain Experts (MoDE): Captures environment-specific knowledge.
  • Mixture of Task/Modality Experts (MoTE/MoME): Captures the specific interactions between different tasks and modalities.
  • Routing Gates (IoGate and DomGate): These gates act as intelligent controllers, dynamically routing each client to the specific "experts" in the supernetwork that align with their local sensors and goals.

By incorporating an orthogonal regularization loss, DisentAFL ensures that private local knowledge and shared global knowledge remain distinct, preventing mutual interference.

Experimental Evaluation

The researchers evaluated DisentAFL across six rigorous simulation scenarios constructed from seven diverse datasets, including image classification, bimodal driving (Vehicle-xM), and multimedia emotion recognition.

In the highly complex MERGE-VM scenario—which involves cross-modal, multi-task learning—DisentAFL achieved an accuracy of 96.4%. This significantly outperformed standard FedAvg (84.6%) and advanced Personalized FL methods like Cross-PFL (90.1%). Ablation studies further confirmed that both stages of disentanglement are critical, with their removal leading to a 5-8% drop in performance.

Conclusion

DisentAFL represents a significant advancement in making Federated Learning viable for AGI. By acknowledging and addressing the inherent asymmetry in multimodal, multi-task environments, it enables efficient knowledge transfer while shielding clients from harmful noise. While the current reliance on a server-side supernetwork and pre-defined domains presents areas for future optimization, the framework provides a robust foundation for privacy-preserving, large-scale multimodal learning.

Summary updated

BibTeX
@inproceedings{chen2024disentanglement,
  author = {Jiayi Chen and Aidong Zhang},
  title = {{On Disentanglement of Asymmetrical Knowledge Transfer for Modality-Task Agnostic Federated Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i10.29010},
}
Figure