Residual Attribute Leakage Is an Interface: Auditing Privacy Evaluation After Text De-identification

Tuan Thanh Duong, Joongmin Shin, Jeongho Yoon, Junyoung Son, Heuiseok LimarXiv 2026 · arXiv

Evaluating how well we protect privacy in text is harder than it looks. When we "de-identify" a document by removing names or dates, residual cues like pronouns ("she"), kinship terms ("husband"), or role nouns ("physician") often remain.

A new paper from researchers at Korea University, "Residual Attribute Leakage Is an Interface: Auditing Privacy Evaluation After Text De-identification," argues that our current ways of measuring this "leakage" are dangerously sensitive to the evaluation setup—the "interface"—and can even reward methods that merely mislead an adversary rather than actually protecting information.

TL;DR

Privacy evaluations for de-identified text are highly sensitive to the "evaluation interface" (adversary model, scoring rules, and priors), which can lead to unstable rankings of de-identification methods. The authors demonstrate that standard metrics can be fooled by "misdirection"—confidently tricking an adversary into the wrong answer—and propose a new auditing framework and a traceable editor, SACC, to expose these vulnerabilities.

What Problem Does This Paper Address?

The standard way to evaluate text privacy is to use a "language model adversary." You give the de-identified text to an LLM and ask it to guess a protected attribute (like sex or occupation). If the LLM fails, we say the text is private.

However, the authors identify a major flaw: Misdirection is not the same as information removal.

Imagine an editor that simply swaps every "he" for "she." A naive LLM adversary will confidently guess the wrong sex. If your privacy score only looks at whether the LLM got the right answer, this "SWAP" editor looks incredibly private. But the information hasn't been removed; it has just been flipped. A "rule-aware" adversary who knows about the swap can recover the true attribute perfectly.

The paper tackles the lack of a systematic way to audit how these evaluation choices—like which LLM you use or how you score its confidence—change the verdict on which de-identification method is "best."

Methods

The authors introduce three key components to their audit:

  1. The Clipped Leakage Score ($L$): A new metric that measures the adversary's confidence gain on the true attribute value, relative to a document-free prior. Crucially, it is clipped at zero. This prevents the score from giving "privacy credit" when an editor pushes the adversary toward a wrong belief (misdirection).
  2. SACC (Scoped Attribute-Cue Closure): A deterministic, rule-based editor used as a research instrument. SACC uses dependency parsing to identify cues and either redacts them (SACC-mask) or rewrites them into neutral forms (e.g., "he" $\rightarrow$ "they").
  3. The SWAP Stress Test: A deliberate counter-example that replaces sex cues with the opposite-sex form. This is used to test if scoring rules can distinguish between removing information and providing false information.
Figure 1: The authors use the SWAP editor to show how naive metrics reward misdirection. While SWAP tricks the adversary, the information remains recoverable if the rule is known.

The authors then perform a "multiverse" style audit, varying six protocol choices (adversary family, subject reference, candidate order, etc.) across three different corpora to see how often the privacy "verdict" changes.

Datasets

The study evaluates editors on three primary corpora:

  • MedQA: Clinical vignettes where the goal is to hide the patient's sex and occupation.
  • TAB (Text Anonymization Benchmark): Court judgments from the European Court of Human Rights.
  • SynthPAI: A synthetic dataset of social media posts.

They also test for generalization on two "unseen" corpora: encyclopedic biographies and film plot summaries.

Key Insights and Ideas

  • The Interface Matters More Than the Text: Privacy rankings are not absolute properties of the de-identified text. They are properties of the interface. For example, changing the adversary model (e.g., from GPT-4o-mini to Llama-3) caused the most reversals in which editor was considered more private.
  • Scoring Rules Can Be Naive: Metrics like attack accuracy or Brier scores can reward misdirection. If an editor makes an LLM 99% sure of the wrong answer, these metrics might rank it as more private than an editor that makes the LLM 50/50 uncertain.
  • Subject Misattribution: LLMs often guess a subject's attributes based on cues belonging to other people in the text. In court judgments, an LLM might guess a defendant's sex based on the pronouns used for their lawyer.
  • Rewriting vs. Redaction: SACC shows that rewriting cues into neutral forms (like "they") generally preserves more utility (content) than redaction ("REDACTED") without significantly increasing privacy risk, provided the rewriting is done carefully.
Figure 3: Verdict sensitivity to protocol choices over all pairs of released texts (988 contrasts; 161 involve SACC). Bars: reversals and separations gained or lost; labels: changed/compared counts, median |∆L| and Kendall’s τb of the seven-text ranking (MedQA / TAB / SynthPAI). Values are descriptive; all exceed the bootstrap-reseeding rate (.007 / .012 / .024). No median for candidate order (verdicts only) or the subject choice (read on AUC); τb needs seven shared texts. ∗Both read- ings before the binder fix of Appendix V.

Key Results

  • SWAP Failure: On MedQA, the SWAP editor ranked first (most private) under the clipped leakage score $L$ and attack accuracy. However, when using a rule-aware adversary, its leakage score jumped from .240 to .654, proving the information was never gone.
  • SACC Performance: SACC reduced leakage by approximately 25% compared to the raw de-identified input across MedQA and TAB, while maintaining higher utility than "global" editors that redact every keyword regardless of context.
  • Verdict Stability: Across 988 contrasts between released texts, the choice of adversary family changed the verdict in 25% of cases. Most other choices affected "separability" (whether the difference was statistically significant) rather than reversing the direction of the result.

Limitations and Future Work

  • Lexical Focus: SACC is a lexical editor; it cannot catch "contextual" leakage. For example, mentioning "prostate cancer" still leaks that the subject is likely male, even if all pronouns are removed.
  • English-Centric: The study is limited to English. Languages with grammatical gender (like French or German) would likely show even higher sensitivity to these evaluation choices.
  • Adversary Drift: The authors noted that proprietary API-based adversaries (like GPT-4o-mini) exhibit drift over time, making long-term benchmarking difficult.
  • Future Directions: The authors suggest that future privacy evaluations must report two-sided displacement (to flag misdirection) and use "rule-aware" adversaries whenever an editor's logic is public.

Summary updated

Abstract

A privacy evaluation of de-identified text asks a language model (the adversary) to guess a person's protected attribute. We introduce a prior-normalised clipped leakage score of its confidence gain on the true value and audit the conclusions drawn from it. Each verdict depends on an evaluation interface (the queried person, answer options, prior, adversary model and scoring rule), which we vary on fixed releases of three corpora. In a stress test, an editor that writes the opposite-sex form at the same spans a redaction editor would redact misleads the adversary about most people, yet ranks first among editors scored on the same MedQA entries under this score, as under attack accuracy and proper scoring rules read as privacy. A relabelling-invariant reading of AUC ranks that editor sixth of eight, and a rule-aware adversary recovers the attribute from the same text. On ordinary releases most verdict changes are gains or losses of separability; of six choices varied over all pairs, only the adversary reverses comparisons (5 of its 124 changes), and the query can target the wrong person. SACC, a traceable editor we build as the instrument, lets us ablate two editing principles and show where its advantage does not hold.
BibTeX
@misc{duong2026residual,
  author = {Tuan Thanh Duong and Joongmin Shin and Jeongho Yoon and Junyoung Son and Heuiseok Lim},
  title = {{Residual Attribute Leakage Is an Interface: Auditing Privacy Evaluation After Text De-identification}},
  year = {2026},
  eprint = {2611.04941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04941},
}
Figure