Abstract
Federated Learning (FL) is designed to preserve privacy by keeping raw data on local devices. However, Gradient Inversion Attacks (GIAs) have emerged as a significant threat, capable of reconstructing private data from shared gradients. This paper introduces MMGIA, a novel attack specifically targeting Multimodal Federated Learning (MMFL). By exploiting intermodal correlations—such as the semantic link between an image and its caption—MMGIA achieves superior reconstruction fidelity compared to existing methods. We propose a two-stage framework: modality-specific reconstruction followed by correlation-driven refinement using a pre-trained multimodal model like CLIP. Our evaluations across diverse datasets (CIFAR100, OpenI, Flickr30K) demonstrate that MMGIA consistently outperforms state-of-the-art baselines in both image and text reconstruction quality.
Introduction
Federated Learning (FL) allows multiple clients to collaboratively train a global model without sharing their local datasets. Instead, clients transmit model gradients to a central server. While this architecture provides a layer of privacy, Gradient Inversion Attacks (GIAs) can reverse-engineer these gradients to recover sensitive input data.
As applications increasingly rely on Multimodal Federated Learning (MMFL)—integrating data types like images and text—the privacy risks evolve. Contrary to the intuition that more complex data might be harder to invert, the inherent semantic correlations between modalities provide additional "leakage" points. Existing GIAs often treat modalities in isolation or make unrealistic assumptions about the attacker's knowledge. MMGIA addresses these gaps by systematically leveraging intermodal correlations to refine reconstructed data.
Methodology: The MMGIA Framework
MMGIA operates via a sophisticated two-stage optimization process designed to exploit the "anchor effect," where high-fidelity reconstruction in one modality guides the recovery of another.
Stage 1: Modality-Specific Reconstruction
The process begins by applying standard GIA techniques independently to the image and text gradients. This stage produces initial "rough drafts" of the private data.
Stage 2: Correlation-Driven Refinement
This stage is the core contribution of MMGIA. It utilizes a pre-trained multimodal model (e.g., CLIP) to map the initial reconstructions into a shared latent space.
- Quality-Weighted Fusion: The system calculates quality scores for each modality based on how closely their generated gradients match the actual shared gradients.
- Fused Representation: A global fused embedding is created, giving more weight to the higher-quality modality. This serves as a semantic "North Star" for further optimization.
- Iterative Optimization: The data is iteratively refined to maximize semantic alignment between modalities, effectively using the stronger modality to "clean up" the weaker one.
Experimental Evaluation
We evaluated MMGIA across three distinct multimodal scenarios to prove its versatility:
- CIFAR100: Standard images with template-based text descriptions.
- OpenI: Real-world medical chest X-rays paired with actual radiology reports.
- Flickr30K: Natural images with complex, human-annotated captions.
Impact of Auxiliary Models
The choice of the auxiliary model in Stage 2 is critical. While general-purpose models like CLIP are effective for natural images, domain-specific models significantly enhance performance in specialized fields.
Key Results and Insights
MMGIA consistently outperforms baselines like MGIS and DLG across various architectures (ResNet-50, BERT).
- Superior Fidelity: On CIFAR100, MMGIA achieved an SSIM of 0.927, surpassing MGIS (0.869). In the medical OpenI dataset, text reconstruction reached a ROUGE-L score of 44.5, significantly higher than the 26.4 achieved by MGIS.
- The Anchor Effect: A successful reconstruction in one modality (e.g., text) acts as a semantic anchor, drastically reducing the search space for the other modality (e.g., image). If the text identifies a "black dog," the optimization for the image is constrained to those features.
- Robustness: MMGIA maintains higher reconstruction quality as batch sizes increase compared to traditional methods. It also shows resilience against standard Differential Privacy (DP) defenses, although performance naturally scales with the privacy budget.
Conclusion
MMGIA demonstrates that the complexity of multimodal data is a double-edged sword: while it enables powerful models, it also introduces significant privacy vulnerabilities through intermodal leakage. Future research will focus on extending this attack to video and audio modalities and developing specialized defense mechanisms that can disrupt the semantic correlations MMGIA exploits.
Summary updated