SoK: Semantic Privacy in Large Language Models

Baihe Ma, Yanna Jiang, Xu Wang, Guangshen Yu, Qin Wang, Caijun Sun, Chen Li, Xuelei Qi, Ying He, Wei Ni, Ren Ping LiuarXiv 2025 · DOI · arXiv

Introduction

Large Language Models (LLMs) have revolutionized natural language processing, but their ability to memorize and synthesize vast amounts of data introduces significant privacy risks. While traditional privacy research focuses on "Data Privacy"—the protection of explicit strings like Social Security numbers—this paper introduces a Systematization of Knowledge (SoK) for Semantic Privacy. Semantic privacy addresses a more insidious threat: the leakage of sensitive information through paraphrasing, logical inference, and stylistic cues across the entire model lifecycle.

Comparison between Data Privacy and Semantic Privacy

The Semantic Privacy Gap

Traditional anonymization techniques, such as removing Personally Identifiable Information (PII), often fail to protect users from sophisticated inference attacks. LLMs are "meaning machines" that capture the underlying relationships between data points. Even if a name is redacted, a model might still reveal a person's identity by describing their unique achievements, location, or writing style.

Illustrating semantic leakage and the goal of semantic privacy protection

As shown in Figure 1, a model without protection can reconstruct a detailed profile of an individual ("Alice") from disparate training data points. Semantic privacy aims to move beyond simple redaction toward "Ideal Semantic Privacy," where the output remains useful but generalizes sensitive details to prevent re-identification.

A Lifecycle-Centric Framework

The authors propose a multi-dimensional framework to track semantic information flow through four key stages of the LLM lifecycle:

  1. S1: Input Processing: The initial stage where raw data is tokenized and converted into embeddings.
  2. S2: Pre-training: The phase where the model learns hierarchical semantics and general world knowledge, often "entangling" sensitive facts with general information.
  3. S3: Fine-tuning: Reorienting the model's internal representations for specific downstream tasks, which can inadvertently sharpen its focus on sensitive attributes.
  4. S4: Alignment: The final tuning for safety and human feedback (e.g., RLHF), which attempts to steer the model toward desirable behaviors but may not fully address latent semantic leakage.

Threat Taxonomy and Defense Landscape

The paper categorizes four primary semantic threat vectors: Membership Inference, Attribute Inference, Model Inversion, and Backdoor Attacks.

In evaluating the defense landscape, the authors systematically review techniques such as Differential Privacy (DP), Split Learning, and Knowledge Unlearning. A critical finding is the persistent Privacy-Utility Tension: traditional token-level noise often destroys the very "meaning" that makes LLMs useful. The authors highlight a need for "semantic-aware" DP that protects the intent and sensitive relationships without breaking the model's fluency.

Key Insights and Future Directions

A major challenge identified is the Quantification Gap. Current metrics like BLEU or Perplexity are poor proxies for semantic privacy. The authors advocate for new, "entailment-aware" scoring systems based on embedding similarity to better measure how much meaning is actually leaking.

Furthermore, as models become multimodal, the risk of cross-modal leakage (e.g., an image caption revealing identity details not present in the text) grows. Future research must focus on:

  • Mathematically Rigorous Metrics: Developing a semantic equivalent to the $\epsilon$ (epsilon) used in Differential Privacy.
  • Inference-Time Controllers: Dynamic systems that can detect and block semantically sensitive queries in real-time.
  • Interpretability vs. Privacy: Balancing the need for "white box" transparency for safety with "black box" protection for privacy.

This SoK serves as a roadmap for the next generation of privacy-preserving AI, shifting the focus from protecting strings to protecting meaning.

Summary updated

BibTeX
@misc{ma2025sok,
  author = {Baihe Ma and Yanna Jiang and Xu Wang and Guangshen Yu and Qin Wang and Caijun Sun and Chen Li and Xuelei Qi and Ying He and Wei Ni and Ren Ping Liu},
  title = {{SoK: Semantic Privacy in Large Language Models}},
  year = {2025},
  eprint = {2506.23603},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.23603},
}
Figure