MMGIA: Gradient Inversion Attack Against Multimodal Federated Learning via Intermodal Correlation

Lele Zheng, Yang Cao, Leo Yu Zhang, Wei Wang, Yulong Shen, Xiaochun CaoIJCAI 2025 · DOI · OpenReview

Abstract

Federated Learning (FL) is designed to preserve privacy by keeping raw data on local devices. However, Gradient Inversion Attacks (GIAs) have emerged as a significant threat, capable of reconstructing private data from shared gradients. This paper introduces MMGIA, a novel attack specifically targeting Multimodal Federated Learning (MMFL). By exploiting intermodal correlations—such as the semantic link between an image and its caption—MMGIA achieves superior reconstruction fidelity compared to existing methods. We propose a two-stage framework: modality-specific reconstruction followed by correlation-driven refinement using a pre-trained multimodal model like CLIP. Our evaluations across diverse datasets (CIFAR100, OpenI, Flickr30K) demonstrate that MMGIA consistently outperforms state-of-the-art baselines in both image and text reconstruction quality.

Introduction

Federated Learning (FL) allows multiple clients to collaboratively train a global model without sharing their local datasets. Instead, clients transmit model gradients to a central server. While this architecture provides a layer of privacy, Gradient Inversion Attacks (GIAs) can reverse-engineer these gradients to recover sensitive input data.

As applications increasingly rely on Multimodal Federated Learning (MMFL)—integrating data types like images and text—the privacy risks evolve. Contrary to the intuition that more complex data might be harder to invert, the inherent semantic correlations between modalities provide additional "leakage" points. Existing GIAs often treat modalities in isolation or make unrealistic assumptions about the attacker's knowledge. MMGIA addresses these gaps by systematically leveraging intermodal correlations to refine reconstructed data.

Methodology: The MMGIA Framework

MMGIA operates via a sophisticated two-stage optimization process designed to exploit the "anchor effect," where high-fidelity reconstruction in one modality guides the recovery of another.

Overview of the MMGIA framework, illustrating the two-stage gradient inversion attack progress from local training to cross-modal optimization.

Stage 1: Modality-Specific Reconstruction

The process begins by applying standard GIA techniques independently to the image and text gradients. This stage produces initial "rough drafts" of the private data.

Stage 2: Correlation-Driven Refinement

This stage is the core contribution of MMGIA. It utilizes a pre-trained multimodal model (e.g., CLIP) to map the initial reconstructions into a shared latent space.

  • Quality-Weighted Fusion: The system calculates quality scores for each modality based on how closely their generated gradients match the actual shared gradients.
  • Fused Representation: A global fused embedding is created, giving more weight to the higher-quality modality. This serves as a semantic "North Star" for further optimization.
  • Iterative Optimization: The data is iteratively refined to maximize semantic alignment between modalities, effectively using the stronger modality to "clean up" the weaker one.

Experimental Evaluation

We evaluated MMGIA across three distinct multimodal scenarios to prove its versatility:

  • CIFAR100: Standard images with template-based text descriptions.
  • OpenI: Real-world medical chest X-rays paired with actual radiology reports.
  • Flickr30K: Natural images with complex, human-annotated captions.

Impact of Auxiliary Models

The choice of the auxiliary model in Stage 2 is critical. While general-purpose models like CLIP are effective for natural images, domain-specific models significantly enhance performance in specialized fields.

Performance comparison of different auxiliary models (CLIP vs. PubMedCLIP) across three datasets, showing the importance of domain-specific models for medical data (OpenI).

Key Results and Insights

MMGIA consistently outperforms baselines like MGIS and DLG across various architectures (ResNet-50, BERT).

  • Superior Fidelity: On CIFAR100, MMGIA achieved an SSIM of 0.927, surpassing MGIS (0.869). In the medical OpenI dataset, text reconstruction reached a ROUGE-L score of 44.5, significantly higher than the 26.4 achieved by MGIS.
  • The Anchor Effect: A successful reconstruction in one modality (e.g., text) acts as a semantic anchor, drastically reducing the search space for the other modality (e.g., image). If the text identifies a "black dog," the optimization for the image is constrained to those features.
  • Robustness: MMGIA maintains higher reconstruction quality as batch sizes increase compared to traditional methods. It also shows resilience against standard Differential Privacy (DP) defenses, although performance naturally scales with the privacy budget.

Conclusion

MMGIA demonstrates that the complexity of multimodal data is a double-edged sword: while it enables powerful models, it also introduces significant privacy vulnerabilities through intermodal leakage. Future research will focus on extending this attack to video and audio modalities and developing specialized defense mechanisms that can disrupt the semantic correlations MMGIA exploits.

Summary updated

Abstract

Multimodal federated learning (MMFL) enables collaborative model training across multiple modalities, such as images and text, without requiring direct data sharing. However, the inherent correlations between modalities introduce new privacy vulnerabilities, making MMFL more susceptible to gradient inversion attacks. In this work, we propose MMGIA, an intermodal correlation-driven gradient inversion attack that systematically exploits multimodal correlation to enhance data reconstruction quality. MMGIA consists of a two-stage optimization framework: the first stage independently reconstructs each modality using traditional gradient inversion techniques, while the second stage refines these reconstructions through pre-trained feature extractors to align modalities in a shared latent space. To further improve reconstruction accuracy, we introduce a quality-weighted fusion strategy, which dynamically integrates multimodal embeddings into a global fused representation that serves as a guiding signal for refining each modality’s reconstruction. This ensures that high-quality reconstructions contribute more to the optimization process, preventing degradation in well-reconstructed modalities while enhancing weaker ones. We conduct extensive experiments on multiple multimodal scenarios, demonstrating that MMGIA outperforms both the only existing multimodal attack and state-of-the-art single-modal attacks, revealing the heightened privacy risks in MMFL.
BibTeX
@inproceedings{zheng2025mmgia,
  author = {Lele Zheng and Yang Cao and Leo Yu Zhang and Wei Wang and Yulong Shen and Xiaochun Cao},
  title = {{MMGIA: Gradient Inversion Attack Against Multimodal Federated Learning via Intermodal Correlation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/886},
}
Figure