Abstract
Prompt privacy is crucial, especially when using online large language models (LLMs), due to the sensitive information often contained within prompts. While LLMs can enhance prompt privacy through text rewriting, existing methods primarily focus on document-level rewriting, neglecting the rich, multi-granular representations of text. This limitation restricts LLM utilization to specific tasks, overlooking their generalization and in-context learning capabilities, thus hindering practical application. To address this gap, we introduce DP-GTR, a novel three-stage framework that leverages local differential privacy (DP) and the composition theorem via group text rewriting. DP-GTR is the first framework to integrate both document-level and word-level information while exploiting in-context learning to simultaneously improve privacy and utility, effectively bridging local and global DP mechanisms at the individual data point level. Experiments on CommonSense QA and DocVQA demonstrate that DP-GTR outperforms existing approaches, achieving a superior privacy-utility trade-off. Furthermore, our framework is compatible with existing rewriting techniques, serving as a plug-in to enhance privacy protection. Our code is publicly available at https://github.com/FatShion-FTD/DP-GTR for reproducibility.
Introduction
As users increasingly interact with cloud-based Large Language Models (LLMs), their prompts often contain sensitive or personally identifiable information (PII). While Differential Privacy (DP) is the gold standard for data protection, applying it to natural language is notoriously difficult. Existing methods generally rely on coarse-grained document-level rewriting, which often introduces excessive noise, or resource-intensive fine-tuning that lacks flexibility.
A significant challenge in current techniques is the "Privacy-Utility Trade-off." Many baseline models suffer from a "Rapid Equilibrium Deterioration Interval" (REDI), where privacy protection degrades sharply for only minor gains in utility. To bridge this gap, we introduce DP-GTR (Differentially Private Group Text Rewriting), a training-free framework that optimizes both privacy and utility through multi-granular text representation.
The DP-GTR Framework
DP-GTR is a three-stage pipeline that runs locally on the user's side before any data is sent to the cloud, ensuring that sensitive information is scrubbed at the source.
- Stage 1: Group Text Rewriting (GTR): Instead of generating a single paraphrase, a local LLM generates a group of $n$ paraphrases. Each is generated using an Exponential Mechanism (EM) to ensure a formal Local Differential Privacy (LDP) guarantee.
- Stage 2: Utility and Privacy Control: The system selects the "best" paraphrase (lowest perplexity) to serve as a high-quality exemplar. Simultaneously, it performs a "bag-of-words" analysis across the group to identify "consensus keywords"—stubborn terms that appear frequently despite DP noise and likely represent sensitive identifiers.
- Stage 3: Privacy-Preserving Prompting: The final prompt uses a template that includes the best paraphrase as an In-Context Learning (ICL) exemplar. It explicitly instructs the cloud LLM to avoid using the identified consensus keywords, using negative constraints to prevent the model from leaking sensitive terms in its output.
Experimental Evaluation
We evaluated the framework on two primary benchmarks: Commonsense QA (CSQA), requiring complex reasoning, and PFL-DocVQA, focused on visual document understanding. Experiments were conducted using GPT-3.5 and Llama-3.1-8B as target cloud models, comparing DP-GTR against baselines like DP-Prompt and DP-MLM.
Key Results and Insights
DP-GTR consistently achieves a superior privacy-utility trade-off. As shown in the performance curves, our method maintains higher utility (accuracy) at lower privacy leakage levels (Rouge1 similarity) compared to existing approaches.
- Utility Recovery via ICL: Using In-Context Learning helps "fix" the damage done by DP noise. By showing the cloud LLM a high-quality private example, the model better understands the desired context and task logic.
- Stability: DP-GTR avoids the REDI effect seen in baselines, converging predictably around 30% Rouge1 similarity while maintaining 55-60% utility.
- Adversarial Robustness: The system remains resilient against "Adaptive Attacks" where an attacker knows the DP-GTR mechanism and attempts to reverse-engineer the original prompt.
Limitations and Future Work
The framework's effectiveness relies on the cloud LLM's ability to follow "negative commands." Furthermore, performing group rewriting requires local hardware capable of running models like Llama-3.1-8B. Future work will focus on optimizing these local resource requirements and exploring how scaling to even larger local models can further improve the quality of the generated exemplars.
Summary updated