TL;DR
This paper introduces PriME, a novel method for personalizing large language models (LLMs) by merging pre-trained modules from multiple users. PriME uses evolutionary algorithms to optimize for both user-specific performance and privacy, offering a better trade-off than existing approaches. This work is significant because it addresses the growing need for personalized LLMs while tackling the critical challenge of user data privacy.
What Problem Does This Paper Address?
Large language models (LLMs) are incredibly powerful but often too general for specific user needs. Personalizing LLMs aims to tailor their behavior to individual users or groups, enhancing their utility in domains like education or content recommendation. Current personalization methods fall into a few categories:
- Prompt-based methods: These inject user preferences into the input prompt. While simple, they are limited by context length and can increase inference costs.
- Training-based methods: These encode user preferences directly into model parameters, often by fine-tuning a small module (like LoRA) for each user. This can be challenging with limited user data and limits collaborative benefits.
- Model merging: This approach combines pre-trained modules from multiple users to create a personalized one. While promising for limited data scenarios, existing methods don't directly optimize for task-specific utility or explicitly address privacy risks for the users whose data contributed to the merged module.
The core challenge PriME tackles is how to achieve effective LLM personalization that simultaneously maximizes user utility and minimizes privacy risks for the users who share their data to enable this personalization.
Methods
PriME employs an evolutionary model merging approach. Here's how it works:
- User Embedding and Selection: First, user embeddings are generated using a language model. For a target user, a subset of "sharing users" whose preferences are most similar to the target user are selected. This is crucial for privacy, as it limits the number of shared modules involved.
- Evolutionary Merging: PriME then uses an evolutionary algorithm (specifically, CMA-ES is mentioned) to find optimal weights for linearly interpolating the selected sharing users' Parameter-Efficient Fine-Tuning (PEFT) modules (like LoRA). This process is "gradient-free," meaning it doesn't rely on traditional backpropagation, allowing it to directly optimize non-differentiable metrics.
- Privacy-Preserving Objective: The key innovation is the optimization objective. PriME aims to maximize the utility (e.g., ROUGE scores for generation, accuracy for classification) for the target user while simultaneously minimizing a privacy leakage measure. This privacy measure quantifies the similarity between the merged module's predictions and the sharing users' predictions on the target user's data. This creates a direct trade-off between utility and privacy.
- Personalized Module Creation: The evolutionary algorithm searches for the best interpolation weights that balance these competing objectives, resulting in a personalized PEFT module for the target user.
Datasets
The primary dataset used for evaluation is LaMP (Language Model Personalization Benchmark). LaMP consists of various personalized text classification and generation tasks. The paper focuses on three classification tasks (LaMP-1, LaMP-2, LaMP-3) and three generation tasks (LaMP-4, LaMP-5, LaMP-7). The authors followed prior work for data splitting and selecting sharing users.
Key Insights and Ideas
- Evolutionary Algorithms for Privacy-Utility Trade-off: The paper demonstrates that evolutionary algorithms are well-suited for optimizing complex, multi-objective problems like privacy-preserving LLM personalization, especially when dealing with non-differentiable utility metrics.
- Direct Privacy Optimization is Key: By explicitly incorporating a privacy leakage metric into the optimization objective, PriME can achieve a superior privacy-utility trade-off compared to methods that don't directly consider privacy during merging.
- Gradient-Free Methods Enable Flexibility: The use of gradient-free optimization allows PriME to directly optimize metrics like ROUGE, which are crucial for evaluating generation quality but are typically non-differentiable.
- Model Merging Can Be Privacy-Aware: This work shows that model merging, often seen as a potential privacy risk, can be adapted to be privacy-preserving.
Key Results
- Superior Performance: PriME consistently outperforms prompt-based (RAG, PAG) and a competing merging-based baseline (Per-Pcs) on the LaMP benchmark, achieving up to a 45% improvement in task performance.
- Better Privacy-Utility Trade-off: PriME offers a controllable trade-off between user preferences and privacy. By adjusting a privacy coefficient ($\alpha$), users can balance utility gains against privacy risks. At higher $\alpha$ values, PriME achieves significantly better privacy (lower MIA risk) while maintaining comparable utility.
- Robustness to MIA: PriME demonstrates enhanced robustness to Membership Inference Attacks (MIA), a common privacy threat, compared to prior methods.
- Efficiency with Less Data: PriME can achieve comparable or better performance than Per-Pcs even when using significantly less training data (down to 20% in some cases), highlighting its efficiency.
- Model Agnostic: Experiments with Llama 3 models show PriME's effectiveness across different architectures and sizes.
Limitations and Future Work
- Scalability: While PriME shows effectiveness with reduced data, further investigation is needed to assess its scalability with larger models and datasets. Developing more sophisticated data selection strategies could improve scalability.
- Formal Privacy Guarantees: The current approach mitigates MIA risks empirically but does not offer formal privacy guarantees. Future work could explore integrating mechanisms like differential privacy.
- Advanced Merging Techniques: Exploring advanced model merging techniques, such as random weight dropping or removing conflicting weights, could further enhance results.
- Parametric vs. Non-Parametric Knowledge: Addressing conflicts between parametric (model weights) and non-parametric (prompt-based) knowledge sources remains an area for future research to further improve LLM personalization.
Summary updated