TL;DR
Standard text anonymization and PII-scrubbing tools create a dangerous "false sense of privacy" by focusing only on surface-level, exact lexical matches. This paper introduces the first semantic re-identification framework to evaluate text sanitization, showing that commercial PII scrubbers leave up to 89% of sensitive information inferable, while differential privacy (DP) text generation protects data only at the cost of crippling utility and generating incoherent text.
What Problem Does This Paper Address?
Releasing sensitive text data (e.g., medical records, conversational logs) requires sanitization to protect user confidentiality. Practitioners typically rely on two approaches:
- Identifier removal / scrubbing: Redacting Personally Identifiable Information (PII) like names, phone numbers, or dates.
- Synthetic data generation: Training generative language models to produce artificial records mimicking the original distribution.
However, standard privacy audits measure privacy using lexical overlap (e.g., ROUGE scores or PII detection lists). The authors show this ignores quasi-identifiers and semantic leakage: an adversary armed with innocuous auxiliary information (e.g., noticing someone drinks wine or has a specific speech pattern) can link records and infer sensitive attributes (like mental health diagnoses or substance use) even if exact names or keywords are completely removed or paraphrased.
Methods
Grounded in Statistical Disclosure Control (SDC) principles used in tabular data reconstruction attacks, the authors design a two-stage evaluation framework:
[Auxiliary Claims] ──> (1. Sparse Linking: BM25) ──> [Matched Sanitized Record]
│
[Original Document Claims] <── (2. Semantic Distance: LLM) ────┘
- Document Atomization: Documents are decomposed into discrete, indivisible factual claims (following the atomic claim formulation of FActScore). An adversary is assumed to know a small subset of claims (e.g., 3 claims) as auxiliary side information.
- Linking Stage ($L$): The auxiliary claims are queried against the sanitized dataset using a sparse retriever (BM25) to identify the candidate record.
- Semantic Matching Metric ($\mu$): Rather than checking word overlap, an evaluator LLM (LLaMA 3.1 8B, validated against human annotations with Fleiss’ $\kappa = 0.875$) compares the remaining original claims against the retrieved record on a 3-point scale (Same Information, Different but Similar, Unsupported), normalized to a $[0, 1]$ semantic distance (higher = more private).
The authors benchmark two primary classes of sanitization:
- Identifier Removal (Sample-level): Commercial Azure AI PII Redaction, Span Sanitization (Dou et al., 2024), Iterative LLM Anonymization (Staab et al., 2024), and LLM Sanitize & Paraphrase.
- Data Synthesis (Dataset-level): GPT-2 fine-tuning without differential privacy, and with DP fine-tuning across privacy budgets $\epsilon \in \{3, 64, 1024, \infty\}$.
Datasets
The authors evaluate downstream utility and privacy on two benchmark domains:
- MedQA: 11,450 USMLE medical licensing exam profiles and multiple-choice questions. Utility is measured by LLaMA 3.1 8B QA accuracy given sanitized clinical vignettes.
- WildChat: 1 million real-world human–ChatGPT conversations containing self-disclosures. Downstream utility is measured by chi-squared divergence in conversation task distributions, evaluated with GPT-4o.
- Text Coherence: A 1–5 Likert scale evaluated by GPT-4o across both datasets.
Key Insights and Ideas
- The Lexical Illusion: Lexical distance metrics (ROUGE-L) consistently report high privacy scores, masking massive semantic leakage. For instance, Azure AI PII redaction achieves a lexical distance of 0.16 on MedQA, but its semantic distance is only 0.11—leaving roughly 89% of private information inferable.
- Unprotected Synthesis Leaks: Generating synthetic text without differential privacy does not ensure anonymity; ~55% of the original information remained re-identifiable in MedQA.
- The DP Utility Collapse: Enforcing formal differential privacy ($\epsilon \in [3, 1024]$) successfully halts semantic reconstruction ($\mu \approx 0.90$), but destroys data utility. On MedQA, DP synthesis accuracy drops to 41–42%—worse than an empty baseline (44%) where the medical text is entirely removed and the model guesses purely from background knowledge.
- Coherence Degrades under DP: Strong DP noise causes text generation quality to fall by ~36% (from 3.28 down to 1.63–2.04), yielding severe clinical hallucinations and contradictory medical statements.
Key Results
| Dataset | Sanitization Method | Lexical Dist. ↑ | Semantic Dist. ↑ | Task Utility ↑ | Text Coherence ↑ |
|---|---|---|---|---|---|
| MedQA | No Sanitization | 0.10 | 0.09 | 0.69 | 3.79 |
| Remove All Info | — | — | 0.44 | — | |
| Azure AI PII Tool | 0.16 | 0.11 | 0.67 | 3.27 | |
| Span Sanitization | 0.61 | 0.43 | 0.62 | 2.84 | |
| Vanilla Data Synthesis ($\epsilon = \infty$) | 0.41 | 0.45 | 0.61 | 3.48 | |
| DP Synthesis ($\epsilon = 3$) | 0.84 | 0.91 | 0.41 | 2.04 | |
| WildChat | No Sanitization | 0.31 | 0.19 | 0.96 | 4.09 |
| Azure AI PII Tool | 0.35 | 0.22 | 0.96 | 3.59 | |
| Iterative Anonymization | 0.58 | 0.41 | 0.92 | 3.51 | |
| DP Synthesis ($\epsilon = 3$) | 0.89 | 0.89 | -0.46 | 1.63 |
Limitations and Future Work
- Domain Scope: Evaluation is limited to medical exam notes (MedQA) and chatbot logs (WildChat); findings need testing on legal, financial, and code domains.
- Contextual Nuance: The metric treats semantic similarity uniformly, whereas real-world privacy harm is highly context-dependent (e.g., revealing an SSRI prescription carries different harm than revealing age).
- Disentangling Confounders: The framework struggles to cleanly differentiate between facts learned strictly from the sanitized dataset versus reasonable priors inferred directly from the auxiliary query itself.
Summary updated