Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models

Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova

Introduction

Multimodal Large Language Models (LLMs), such as CLIP, are incredibly powerful but massive, making them difficult to fine-tune on edge devices. Federated Prompt Learning (FPL) offers an elegant solution: instead of updating billions of parameters, clients collaboratively learn a small "soft prompt" (a sequence of tunable vectors) while keeping the main model frozen.

However, FPL faces a three-way tug-of-war—an "impossible trinity":

  1. Personalization: Each client has unique data distributions, requiring a model tailored to their specific needs.
  2. Generalization: The model must still perform well on unseen data by leveraging the collective knowledge of the federation.
  3. Privacy: Soft prompts are surprisingly vulnerable. Adversaries can use Membership Inference Attacks (MIA) to determine if a specific image was used in a client's training set just by analyzing the trained prompt.

Standard privacy techniques, like Differential Privacy (DP), typically involve adding noise to the training process. Unfortunately, this noise often acts as a "utility killer," destroying both the fine-grained details needed for personalization and the broad patterns required for generalization. This paper introduces Differentially Private Federated Prompt Learning (DP-FPL), a framework designed to successfully balance this impossible trinity.

The DP-FPL Architecture

To achieve state-of-the-art accuracy on vision-language tasks under strict privacy constraints, the authors propose an architecture built on three main pillars:

DP-FPL Architecture Overview
  • Iterative Low-Rank Factorization: In every training round, the local prompt is decomposed into two low-rank matrices ($u$ and $v$). Low-rank structures are naturally better at capturing broad, generalizable patterns and are significantly more resilient to DP noise.
  • The Residual Term ($r$): Because low-rank factorization inevitably loses some information, the authors maintain a residual term that captures the "leftovers." This term stays local to the client and preserves the high-frequency, expressive details crucial for personalization.
  • Hybrid Differential Privacy: To protect against various attack vectors, the framework applies a dual-layered privacy approach:
  • Local DP (LDP): Noise is added specifically to the low-rank components of the local prompt before they update the model.
  • Global DP (GDP): Noise is added to the gradients sent to the central server to protect the shared global knowledge.
  • Crucially, by injecting noise only into the low-rank components rather than the full prompt, the framework mitigates the overall impact of noise on model performance.

Key Insights and Performance

The researchers evaluated DP-FPL on five widely used image classification benchmarks (Caltech101, OxfordPets, OxfordFlowers, Food101, and CIFAR-100), using splits to simulate real-world data heterogeneity.

A surprising insight from the study is the concept of Privacy as Regularization. At moderate levels, DP noise actually improved generalization on some datasets. The noise acts as a form of stochastic regularization, preventing the model from over-focusing on local quirks. Furthermore, ablation studies highlighted the power of the residual term: without it, performance plummeted under high privacy noise. The residual acts as a buffer, allowing the model to stay expressive even when shared components are heavily blurred by noise.

Empirical Results

DP-FPL consistently outperformed existing benchmarks like PromptFL, FedOTP, and FedPGP, especially as privacy requirements became stricter.

Performance comparison across datasets and privacy budgets
  • Superior Accuracy Under Strict Privacy: At a very strict privacy level ($\epsilon = 0.01$), DP-FPL maintained 85.21% accuracy on Caltech101, while the nearest competitor dropped to roughly 78%. Similar robust margins were observed on OxfordPets and Food101.
  • Robust Privacy Guarantees: The framework effectively neutralized Membership Inference Attacks. At $\epsilon = 0.1$, the attack success rate was pushed down near 50% (equivalent to a random guess), meaning the attacker learned virtually nothing about the private data.
  • Balanced Utility: Unlike other methods that sacrifice generalization for personalization (or vice versa), DP-FPL showed the highest utility across both local client classes and broader neighbor classes.

Limitations and Future Work

While highly effective, the framework does have limitations. The authors noted that certain datasets, like Oxford Flowers, were particularly challenging and required higher privacy budgets ($\epsilon \geq 0.1$) to achieve a good balance. Additionally, the current study uses a fixed rank (e.g., rank 8) for factorization. Future iterations might explore dynamic rank adjustment based on the complexity of a client's local data. Finally, while the residual term works well in practice, a deeper theoretical analysis of exactly why it compensates so effectively for regularization-like error remains a prime candidate for future research.

Summary updated

Abstract

Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.
BibTeX
@inproceedings{tran2025privacy,
  author = {Linh Tran and Wei Sun and Stacy Patterson and Ana Milanova},
  title = {{Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=Equ277PBN0},
}
Figure