Evaluating how well we protect privacy in text is harder than it looks. When we "de-identify" a document by removing names or dates, residual cues like pronouns ("she"), kinship terms ("husband"), or role nouns ("physician") often remain.
A new paper from researchers at Korea University, "Residual Attribute Leakage Is an Interface: Auditing Privacy Evaluation After Text De-identification," argues that our current ways of measuring this "leakage" are dangerously sensitive to the evaluation setup—the "interface"—and can even reward methods that merely mislead an adversary rather than actually protecting information.
TL;DR
Privacy evaluations for de-identified text are highly sensitive to the "evaluation interface" (adversary model, scoring rules, and priors), which can lead to unstable rankings of de-identification methods. The authors demonstrate that standard metrics can be fooled by "misdirection"—confidently tricking an adversary into the wrong answer—and propose a new auditing framework and a traceable editor, SACC, to expose these vulnerabilities.
What Problem Does This Paper Address?
The standard way to evaluate text privacy is to use a "language model adversary." You give the de-identified text to an LLM and ask it to guess a protected attribute (like sex or occupation). If the LLM fails, we say the text is private.
However, the authors identify a major flaw: Misdirection is not the same as information removal.
Imagine an editor that simply swaps every "he" for "she." A naive LLM adversary will confidently guess the wrong sex. If your privacy score only looks at whether the LLM got the right answer, this "SWAP" editor looks incredibly private. But the information hasn't been removed; it has just been flipped. A "rule-aware" adversary who knows about the swap can recover the true attribute perfectly.
The paper tackles the lack of a systematic way to audit how these evaluation choices—like which LLM you use or how you score its confidence—change the verdict on which de-identification method is "best."
Methods
The authors introduce three key components to their audit:
- The Clipped Leakage Score ($L$): A new metric that measures the adversary's confidence gain on the true attribute value, relative to a document-free prior. Crucially, it is clipped at zero. This prevents the score from giving "privacy credit" when an editor pushes the adversary toward a wrong belief (misdirection).
- SACC (Scoped Attribute-Cue Closure): A deterministic, rule-based editor used as a research instrument. SACC uses dependency parsing to identify cues and either redacts them (SACC-mask) or rewrites them into neutral forms (e.g., "he" $\rightarrow$ "they").
- The SWAP Stress Test: A deliberate counter-example that replaces sex cues with the opposite-sex form. This is used to test if scoring rules can distinguish between removing information and providing false information.
The authors then perform a "multiverse" style audit, varying six protocol choices (adversary family, subject reference, candidate order, etc.) across three different corpora to see how often the privacy "verdict" changes.
Datasets
The study evaluates editors on three primary corpora:
- MedQA: Clinical vignettes where the goal is to hide the patient's sex and occupation.
- TAB (Text Anonymization Benchmark): Court judgments from the European Court of Human Rights.
- SynthPAI: A synthetic dataset of social media posts.
They also test for generalization on two "unseen" corpora: encyclopedic biographies and film plot summaries.
Key Insights and Ideas
- The Interface Matters More Than the Text: Privacy rankings are not absolute properties of the de-identified text. They are properties of the interface. For example, changing the adversary model (e.g., from GPT-4o-mini to Llama-3) caused the most reversals in which editor was considered more private.
- Scoring Rules Can Be Naive: Metrics like attack accuracy or Brier scores can reward misdirection. If an editor makes an LLM 99% sure of the wrong answer, these metrics might rank it as more private than an editor that makes the LLM 50/50 uncertain.
- Subject Misattribution: LLMs often guess a subject's attributes based on cues belonging to other people in the text. In court judgments, an LLM might guess a defendant's sex based on the pronouns used for their lawyer.
- Rewriting vs. Redaction: SACC shows that rewriting cues into neutral forms (like "they") generally preserves more utility (content) than redaction ("REDACTED") without significantly increasing privacy risk, provided the rewriting is done carefully.
Key Results
- SWAP Failure: On MedQA, the SWAP editor ranked first (most private) under the clipped leakage score $L$ and attack accuracy. However, when using a rule-aware adversary, its leakage score jumped from .240 to .654, proving the information was never gone.
- SACC Performance: SACC reduced leakage by approximately 25% compared to the raw de-identified input across MedQA and TAB, while maintaining higher utility than "global" editors that redact every keyword regardless of context.
- Verdict Stability: Across 988 contrasts between released texts, the choice of adversary family changed the verdict in 25% of cases. Most other choices affected "separability" (whether the difference was statistically significant) rather than reversing the direction of the result.
Limitations and Future Work
- Lexical Focus: SACC is a lexical editor; it cannot catch "contextual" leakage. For example, mentioning "prostate cancer" still leaks that the subject is likely male, even if all pronouns are removed.
- English-Centric: The study is limited to English. Languages with grammatical gender (like French or German) would likely show even higher sensitivity to these evaluation choices.
- Adversary Drift: The authors noted that proprietary API-based adversaries (like GPT-4o-mini) exhibit drift over time, making long-term benchmarking difficult.
- Future Directions: The authors suggest that future privacy evaluations must report two-sided displacement (to flag misdirection) and use "rule-aware" adversaries whenever an editor's logic is public.
Summary updated