PrivInfer: Privacy-Preserving Inference for Black-box Large Language Model

Meng Tong, Kejiang Chen, Yuang Qi, Jie Zhang, Weiming Zhang, Nenghai Yu

TL;DR

This paper introduces InferDPT, a novel framework for privacy-preserving inference with black-box Large Language Models (LLMs) like ChatGPT. It uses a two-module approach: a perturbation module to add differential privacy to user prompts and an extraction module to reconstruct coherent text, achieving a strong balance between privacy and utility. Their proposed RANTEXT mechanism significantly improves privacy against common attacks compared to existing methods.

What Problem Does This Paper Address?

The widespread adoption of powerful LLMs like ChatGPT has brought immense utility but also significant privacy concerns. Users often input sensitive information into these models, risking data leakage or unauthorized collection. While differential privacy (DP) offers a solution, existing DP methods for LLM inference face practical challenges: they are often too slow, require too much communication, or degrade the quality of generated text too much, especially for open-ended generation tasks. Current approaches are also not robust enough against sophisticated attacks like embedding inversion. This paper aims to bridge this gap by providing a practical, efficient, and robust privacy-preserving inference framework for black-box LLMs.

Methods

InferDPT is built on two core modules:

  1. Perturbation Module: This module applies differential privacy to the user's input prompt. Instead of directly sending the raw prompt to the LLM, it first perturbs the text. The authors introduce RANTEXT, a novel DP mechanism that uses a "random adjacency list" for token perturbation. This differs from prior methods (like SANTEXT+ and CUSTEXT+) which used static lists. RANTEXT dynamically generates a list of potential replacement tokens for each original token, aiming for a better trade-off between semantic similarity (utility) and privacy. This module then sends the perturbed prompt to the black-box LLM.
  2. Extraction Module: The LLM returns a "perturbed generation" based on the perturbed prompt. This output might be less coherent due to the privacy perturbations. The extraction module uses a smaller, local language model to process this perturbed generation. Inspired by knowledge distillation, it extracts coherent and consistent text from the LLM's output and reconstructs it into a final, aligned generation that preserves the original intent while maintaining privacy.

Datasets

The authors evaluated their framework on several standard NLP datasets:

  • CNN/Daily Mail: A common dataset for news summarization and text generation.
  • Wikitext-103-v1: A benchmark dataset for language modeling.
  • ArXiv Dataset: A collection of scientific papers.
  • FinRED: A dataset for financial relation extraction.
  • MedQA: A dataset for medical question answering.

These datasets are used for evaluating the utility (quality of generated text) of the InferDPT framework.

Key Insights and Ideas

  • The "Random Adjacency List" is Key: The core innovation, RANTEXT, moves away from static lists of replacement tokens. By dynamically generating random adjacency lists, it offers a more flexible and robust perturbation, leading to better privacy guarantees and improved utility compared to fixed lists.
  • Perturbed Generation as a "Hint": The perturbed generation from the LLM, while potentially noisy, isn't discarded. Instead, it acts as a valuable "hint" for the local extraction model. This allows the smaller local model to achieve generation quality comparable to the much larger remote LLM, a form of knowledge distillation.
  • Black-Box LLMs are a Target: The framework is specifically designed for "black-box" LLMs (like ChatGPT), where the internal model architecture and parameters are not accessible. This makes it applicable to widely used commercial services.
  • Adaptive Attacks Matter: The authors highlight that traditional DP attacks might not be sufficient. They propose and test an "adaptive attack" (GPT inference attack) that leverages the capabilities of the LLM itself to try and recover the original prompt, demonstrating RANTEXT's resilience.

Key Results

  • Superior Privacy: RANTEXT significantly outperforms SANTEXT+ and CUSTEXT+ in privacy protection. For instance, with an epsilon (ε) of 6.0, RANTEXT achieved over 90% privacy protection against embedding inversion attacks, which is 3.35x higher than CUSTEXT+ and 0.58x higher than SANTEXT+.
  • Comparable Utility: The text generation quality of InferDPT (using RANTEXT) is comparable to that of non-private GPT-4 across various metrics (diversity, MAUVE, coherence) on multiple datasets.
  • Efficiency: While there's an overhead for perturbation and extraction, the overall inference time is practical, with the extraction module taking less than 4 seconds per inference.
  • Robustness: InferDPT, particularly with RANTEXT, demonstrates strong privacy protection against BERT inference attacks, embedding inversion attacks, and GPT inference attacks.

Limitations and Future Work

  • Local Model Requirement: InferDPT requires a local language model for the extraction module. This might be a limitation in extremely resource-constrained environments (e.g., smartwatches).
  • MAUVE Score Gap: There's a slight gap in MAUVE scores compared to non-private GPT-4. The authors suggest that future work could focus on improving the extraction module's capabilities or developing more advanced DP perturbation mechanisms to better preserve semantic nuances.
  • Prompt Optimization: Further research into optimizing the system prompt for the extraction module could enhance performance.
  • Optimal Perturbation: Exploring more optimal perturbation mechanisms beyond the current exponential mechanism could further improve the privacy-utility trade-off.

Summary updated

Abstract

Large language models (LLMs), like ChatGPT, have greatly simplified text generation tasks. However, they have also raised concerns about privacy risks such as data leakage and unauthorized data collection. Existing solutions for privacy-preserving inference face practical challenges related to computation time and communication costs. In this paper, we propose InferDPT, the first practical framework for the privacy-preserving Inference of black-box LLMs, implementing Differential Privacy in Text generation. InferDPT comprises two key modules: the "perturbation module" utilizes the exponential mechanism to generate a perturbed prompt, facilitating privacy-preserving inference with black-box LLMs, and the "extraction module", inspired by knowledge distillation and retrieval-augmented generation, extracts coherent and consistent text from the perturbed generation result, ensuring successful text generation completion. To address privacy concerns related to previous exponential mechanisms' susceptibility to embedding revision attacks, we introduce RANTEXT, a novel differential privacy mechanism integrated into the perturbation module of InferDPT, which introduces the concept of "RANdom adjacency" for TEXT perturbation within the prompt. Experimental results across three datasets demonstrate that the text generation quality of InferDPT is comparable to that of non-private GPT-4, and RANTEXT surpasses existing state-of-the-art mechanisms, namely, SANTEXT+ and CUSTEXT+ in the trade-off between privacy and utility. Even with an privacy parameter epsilon value of 6.0, RANTEXT achieves an average privacy protection rate exceeding 90% against embedding revision attacks, which is 0.58 times higher than that of SANTEXT+ and 3.35 times higher than that of CUSTEXT+.
BibTeX
@article{tong2025privinfer,
  author = {Meng Tong and Kejiang Chen and Yuang Qi and Jie Zhang and Weiming Zhang and Nenghai Yu},
  title = {{PrivInfer: Privacy-Preserving Inference for Black-box Large Language Model}},
  journal = {IEEE Trans. Dependable Secur. Comput.},
  year = {2025},
  doi = {10.1109/tdsc.2025.3550389},
}
Figure