LLMs are Privacy Erasable

Zipeng Ye, Wenjian LuoEMNLP 2025

TL;DR

This paper introduces a practical, pipeline-parallel inference method to protect user privacy when using cloud-based Large Language Models (LLMs). It achieves this by strategically modifying intermediate representations, effectively thwarting sophisticated reconstruction attacks with minimal impact on LLM performance.

What Problem Does This Paper Address?

The increasing reliance on cloud-based LLM services for tasks like document summarization and content generation presents a significant privacy challenge. Users often need to upload sensitive personal or proprietary data as context for these LLMs. While LLMs offer immense convenience, the potential for malicious service providers to reconstruct this private data from intermediate model states (hidden states) is a serious concern. Existing privacy-preserving methods, such as differential privacy, often degrade LLM performance too much for fine-grained tasks, or are too complex to implement. This paper aims to bridge the gap between LLM utility and user privacy by proposing a practical and effective solution.

Methods

The core of the proposed solution is a distributed inference paradigm where a small portion of the LLM (embedding layer and a few attention layers) is run locally on the user's device. The user's input is processed by this local module, and the resulting hidden state is then modified before being sent to the cloud LLM. This modification involves erasing a portion of the original input's representation and then randomly introducing token embeddings from the vocabulary. This "misdirection" strategy makes it significantly harder for an adversary to reconstruct the original private data from the hidden states, even if they are aware of the defense mechanism. The authors emphasize that this approach is "plug-and-play," requires no additional training, and is compatible with low-bit quantization for efficiency.

Datasets

The paper evaluates its method on a variety of mainstream LLM benchmarks, including:

  • BoolQ: A question-answering dataset for determining if a statement is true or false based on context.
  • SQuAD: A reading comprehension dataset for extracting answers from text.
  • HellaSwag: A commonsense reasoning dataset for inferring the most plausible continuation of a sentence.
  • GSM8K: A dataset of grade-school math problems.
  • HumanEval: A dataset for evaluating code generation capabilities.
  • MMLU: A broad benchmark covering diverse subjects.
  • BBH: A benchmark for evaluating in-context learning.

No new datasets were introduced.

Key Insights and Ideas

  • LLMs are inherently "privacy erasable": The paper demonstrates that even after passing through multiple layers of an LLM, the original input's information is remarkably well-preserved in hidden states, making it susceptible to reconstruction attacks.
  • Orthogonality of semantic domains: The authors hypothesize and provide evidence that different semantic domains within LLM embeddings tend to occupy nearly orthogonal subspaces. This property is crucial for their defense, as it means introducing random token embeddings has a minimal impact on the overall semantic representation.
  • Cosine distance is key for attackers: The paper highlights that cosine distance is a more effective metric than Euclidean distance for attackers trying to match hidden states to original tokens, especially under noise.
  • "Misdirection" is more effective than pure noise: Simply adding random noise (like in differential privacy) requires large amounts of noise to be effective, significantly degrading performance. The proposed method's "misdirection" strategy, by introducing token embeddings, offers a better balance between privacy and utility.

Key Results

  • Effective Attack Thwarting: The proposed method successfully prevents reconstruction attacks, with attack metrics (like Rouge scores) dropping significantly (e.g., from ~0.9 to ~0.1-0.3) after defense.
  • Minimal Performance Degradation: The method shows negligible impact on LLM performance across various tasks, and in some cases, even slightly improves it. For example, on SQuAD, accuracies remained high (e.g., 83.8% without defense vs. 83.7% with defense for Llama-3-8B).
  • Compatibility with Quantization: The defense is compatible with low-bit quantization (e.g., 4-bit), allowing for reduced runtime memory requirements (1-2 GB) on user devices without sacrificing performance.
  • Robustness to Scale Coefficient: The defense remains effective even with variations in the scale coefficient (δ), demonstrating its robustness.

Limitations and Future Work

  • User-end Computation: The current method requires some local computation on the user's device, which might be a barrier for extremely resource-constrained environments.
  • Integration with TEE: The authors plan to explore integrating their architecture with Trusted Execution Environments (TEEs) to alleviate user-end computational pressure and potentially deploy the user-end module to the cloud's TEE.
  • Formal Proofs: While the method is empirically validated, the authors aim to provide further formal proofs for its privacy guarantees, potentially linking it more formally to differential privacy concepts.

Summary updated

BibTeX
@inproceedings{ye2025llms,
  author = {Zipeng Ye and Wenjian Luo},
  title = {{LLMs are Privacy Erasable}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure