TL;DR
This paper introduces DP-GTR, a novel framework for protecting the privacy of prompts sent to Large Language Models (LLMs). By leveraging group text rewriting and in-context learning, DP-GTR offers a fine-grained privacy-utility trade-off, outperforming existing methods in realistic question-answering scenarios.
What Problem Does This Paper Address?
The increasing use of LLMs in various applications has raised significant concerns about prompt privacy. Prompts can contain sensitive personal information, and LLMs, while powerful, can inadvertently leak this information. Existing methods for protecting prompt privacy often fall into a few categories: training-based optimizations (computationally expensive), embedding perturbations (can harm semantic coherence), and document-level paraphrasing (often lacks fine-grained control over privacy-utility trade-offs and can overlook word-level vulnerabilities). Furthermore, many approaches focus on global differential privacy (DP), which requires data aggregation, while client-side applications need local DP (LDP) for individual data point protection. This creates a gap in practical, client-side prompt privacy solutions that balance privacy with utility.
Methods
DP-GTR is a three-stage framework designed to bridge the gap between global and local DP mechanisms for prompt protection.
- Stage 1: Group Text Rewriting (GTR): This novel mechanism generates multiple paraphrased versions of the original prompt. This "rewriting group" preserves rich contextual information and enables bag-of-words-like analysis. This stage is designed to work with existing DP paraphrasing techniques.
- Stage 2: Utility and Privacy Control: This stage leverages the rewriting group for fine-grained control.
- Utility Enhancement: The paraphrase with the lowest perplexity is selected as an in-context learning (ICL) exemplar. This exemplar is used to guide the LLM, maximizing output quality without incurring additional DP cost due to the post-processing property of DP.
- Privacy Protection: Word counts are aggregated across the paraphrased prompts to identify "consensus keywords" – words that frequently appear despite paraphrasing attempts. These are considered potentially sensitive. The framework offers two methods for releasing these keywords:
- DP-GTR-NDP: Uses non-DP post-processing for keyword release, leveraging the post-processing property of DP to avoid additional privacy budget.
- DP-GTR-JEM: Employs a Joint Exponential Mechanism (Joint-EM) for privacy-preserving release of top-K keywords, incurring an additional DP budget.
- Stage 3: Privacy-Preserving Prompting: The selected ICL exemplar and the released private keywords are combined into a prompt template. This template is then fed to the LLM to generate the final, differentially private prompt, effectively suppressing sensitive information.
Datasets
The paper evaluates DP-GTR on two question-answering datasets:
- Commonsense QA (CSQA): A 5-choice closed-answer dataset.
- PFL-DocVQA (VQA): An open-answer dataset that provides pre-extracted OCR tokens.
The authors selected 200 random items from the validation set of each dataset for their experiments.
Key Insights and Ideas
- Bridging Local and Global DP: DP-GTR's group text rewriting approach ingeniously connects local DP principles (applied to individual paraphrases) with global DP concepts (through aggregated keyword analysis), offering a more comprehensive privacy solution.
- Fine-grained Control: By identifying and suppressing specific "consensus keywords" rather than just applying a blanket privacy budget to the entire prompt, DP-GTR allows for a more nuanced privacy-utility trade-off.
- Leveraging ICL for Utility: Using the lowest-perplexity paraphrase as an ICL exemplar is a clever way to boost the utility of the rewritten prompt without adding to the privacy burden, thanks to the post-processing property of DP.
- Generalizable Plug-in: The modular design of DP-GTR, particularly Stage 1, allows it to be integrated with various existing DP paraphrasing techniques, enhancing their privacy capabilities.
- Limitations of Existing Metrics: The paper highlights that traditional privacy evaluation metrics (like adversarial classifiers) and utility metrics (like sentiment analysis) are often too simplistic and fail to capture the nuanced impact of privacy-preserving techniques on text meaning and style.
Key Results
- Superior Privacy-Utility Trade-off: DP-GTR consistently outperforms baseline methods (DP-Prompt, DP-MLM, DP-Paraphrase) in achieving a better balance between privacy and utility across both open-answer (VQA) and closed-answer (CSQA) QA datasets.
- Improved Privacy with Comparable Utility: DP-GTR often achieves better privacy than DP-Prompt at similar utility levels, and sometimes higher utility at lower privacy.
- Stable Convergence: On the CSQA dataset, DP-GTR shows a more stable and robust trade-off convergence compared to DP-Prompt, which exhibits a "Rapid Equilibrium Deterioration Interval" (REDI) where privacy degrades sharply with minor utility gains.
- Robustness to Adversarial Attacks: DP-GTR demonstrates strong robustness against both static and adaptive adversarial attacks, with no observed increase in question similarity, indicating no privacy leakage.
- Non-Uniform Rewriting Benefits: Investigating non-uniform rewriting strategies (varying temperatures) showed that DP-GTR achieves favorable privacy-utility trade-offs, with significant privacy leakage reduction and only minor utility loss.
Limitations and Future Work
- LLM Instruction Following: A key limitation is that LLMs may sometimes fail to strictly follow instructions, potentially leading to privacy leakage or an inability to learn from ICL exemplars.
- Computational Overhead: DP-GTR introduces additional computational overhead compared to baseline inference, though the authors argue this is justified by the enhanced privacy protection.
- Model Capability: The use of the Llama-3.1-8B model, while capable, might not fully leverage the framework's potential due to its limitations in learning from prompts.
- Future Work: The authors suggest shifting control from outlier prompting to an internal LLM generation configuration to fundamentally address prompt failure issues. They also believe that using more capable open-source models could further improve performance.
Summary updated