A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage

Rui Xin, Niloofar Mireshghallah, Shuyue Stella Li, Michael Duan, Hyunwoo Kim, Yejin Choi, Yulia Tsvetkov, Sewoong Oh, Pang Wei KohSubmission to ICLR 2025 · Rejected

TL;DR

Standard text anonymization and PII-scrubbing tools create a dangerous "false sense of privacy" by focusing only on surface-level, exact lexical matches. This paper introduces the first semantic re-identification framework to evaluate text sanitization, showing that commercial PII scrubbers leave up to 89% of sensitive information inferable, while differential privacy (DP) text generation protects data only at the cost of crippling utility and generating incoherent text.

What Problem Does This Paper Address?

Releasing sensitive text data (e.g., medical records, conversational logs) requires sanitization to protect user confidentiality. Practitioners typically rely on two approaches:

  1. Identifier removal / scrubbing: Redacting Personally Identifiable Information (PII) like names, phone numbers, or dates.
  2. Synthetic data generation: Training generative language models to produce artificial records mimicking the original distribution.

However, standard privacy audits measure privacy using lexical overlap (e.g., ROUGE scores or PII detection lists). The authors show this ignores quasi-identifiers and semantic leakage: an adversary armed with innocuous auxiliary information (e.g., noticing someone drinks wine or has a specific speech pattern) can link records and infer sensitive attributes (like mental health diagnoses or substance use) even if exact names or keywords are completely removed or paraphrased.

Methods

Grounded in Statistical Disclosure Control (SDC) principles used in tabular data reconstruction attacks, the authors design a two-stage evaluation framework:

[Auxiliary Claims] ──> (1. Sparse Linking: BM25) ──> [Matched Sanitized Record]
                                                               │
[Original Document Claims] <── (2. Semantic Distance: LLM) ────┘
  1. Document Atomization: Documents are decomposed into discrete, indivisible factual claims (following the atomic claim formulation of FActScore). An adversary is assumed to know a small subset of claims (e.g., 3 claims) as auxiliary side information.
  2. Linking Stage ($L$): The auxiliary claims are queried against the sanitized dataset using a sparse retriever (BM25) to identify the candidate record.
  3. Semantic Matching Metric ($\mu$): Rather than checking word overlap, an evaluator LLM (LLaMA 3.1 8B, validated against human annotations with Fleiss’ $\kappa = 0.875$) compares the remaining original claims against the retrieved record on a 3-point scale (Same Information, Different but Similar, Unsupported), normalized to a $[0, 1]$ semantic distance (higher = more private).

The authors benchmark two primary classes of sanitization:

  • Identifier Removal (Sample-level): Commercial Azure AI PII Redaction, Span Sanitization (Dou et al., 2024), Iterative LLM Anonymization (Staab et al., 2024), and LLM Sanitize & Paraphrase.
  • Data Synthesis (Dataset-level): GPT-2 fine-tuning without differential privacy, and with DP fine-tuning across privacy budgets $\epsilon \in \{3, 64, 1024, \infty\}$.

Datasets

The authors evaluate downstream utility and privacy on two benchmark domains:

  • MedQA: 11,450 USMLE medical licensing exam profiles and multiple-choice questions. Utility is measured by LLaMA 3.1 8B QA accuracy given sanitized clinical vignettes.
  • WildChat: 1 million real-world human–ChatGPT conversations containing self-disclosures. Downstream utility is measured by chi-squared divergence in conversation task distributions, evaluated with GPT-4o.
  • Text Coherence: A 1–5 Likert scale evaluated by GPT-4o across both datasets.

Key Insights and Ideas

  • The Lexical Illusion: Lexical distance metrics (ROUGE-L) consistently report high privacy scores, masking massive semantic leakage. For instance, Azure AI PII redaction achieves a lexical distance of 0.16 on MedQA, but its semantic distance is only 0.11—leaving roughly 89% of private information inferable.
  • Unprotected Synthesis Leaks: Generating synthetic text without differential privacy does not ensure anonymity; ~55% of the original information remained re-identifiable in MedQA.
  • The DP Utility Collapse: Enforcing formal differential privacy ($\epsilon \in [3, 1024]$) successfully halts semantic reconstruction ($\mu \approx 0.90$), but destroys data utility. On MedQA, DP synthesis accuracy drops to 41–42%—worse than an empty baseline (44%) where the medical text is entirely removed and the model guesses purely from background knowledge.
  • Coherence Degrades under DP: Strong DP noise causes text generation quality to fall by ~36% (from 3.28 down to 1.63–2.04), yielding severe clinical hallucinations and contradictory medical statements.

Key Results

Dataset Sanitization Method Lexical Dist. ↑ Semantic Dist. ↑ Task Utility ↑ Text Coherence ↑
MedQA No Sanitization 0.10 0.09 0.69 3.79
Remove All Info — — 0.44 —
Azure AI PII Tool 0.16 0.11 0.67 3.27
Span Sanitization 0.61 0.43 0.62 2.84
Vanilla Data Synthesis ($\epsilon = \infty$) 0.41 0.45 0.61 3.48
DP Synthesis ($\epsilon = 3$) 0.84 0.91 0.41 2.04
WildChat No Sanitization 0.31 0.19 0.96 4.09
Azure AI PII Tool 0.35 0.22 0.96 3.59
Iterative Anonymization 0.58 0.41 0.92 3.51
DP Synthesis ($\epsilon = 3$) 0.89 0.89 -0.46 1.63

Limitations and Future Work

  • Domain Scope: Evaluation is limited to medical exam notes (MedQA) and chatbot logs (WildChat); findings need testing on legal, financial, and code domains.
  • Contextual Nuance: The metric treats semantic similarity uniformly, whereas real-world privacy harm is highly context-dependent (e.g., revealing an SSRI prescription carries different harm than revealing age).
  • Disentangling Confounders: The framework struggles to cleanly differentiate between facts learned strictly from the sanitized dataset versus reasonable priors inferred directly from the auxiliary query itself.

Summary updated

Abstract

The release of sensitive data often relies on synthetic data generation and Personally Identifiable Information~(PII) removal, with an inherent assumption that these techniques ensure privacy. However, the effectiveness of sanitization methods for text datasets has not been thoroughly evaluated. To address this critical gap, we propose the first privacy evaluation framework for the release of sanitized textual datasets. In our framework, a sparse retriever initially links sanitized records with target individuals based on known auxiliary information. Subsequently, semantic matching quantifies the extent of additional information that can be inferred about these individuals from the matched records. We apply our framework to two datasets: MedQA, containing medical records, and WildChat, comprising individual conversations with ChatGPT. Our results demonstrate that seemingly innocuous auxiliary information, such as specific speech patterns, can be used to deduce personal attributes like age or substance use history from the synthesized dataset. We show that private information can persist in sanitized records at a semantic level, even in synthetic data. Our findings highlight that current data sanitization methods create a false sense of privacy by making only surface-level textual manipulations. This underscores the urgent need for more robust protection methods that address semantic-level information leakage.
BibTeX
@inproceedings{xin2024false,
  author = {Rui Xin and Niloofar Mireshghallah and Shuyue Stella Li and Michael Duan and Hyunwoo Kim and Yejin Choi and Yulia Tsvetkov and Sewoong Oh and Pang Wei Koh},
  title = {{A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage}},
  booktitle = {RegML @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=p9KIxJmKpy},
}
Figure