Does the Adversary Read the Text? Counterfactual Audits of Privacy Leakage

Modern text anonymizers are graded by a second language model, called the adversary, which reads the rewritten text and tries to guess private facts about its author, such as age, location, or occupation. If the adversary guesses right, the anonymizer is said to have leaked. This only works as a measurement if a correct guess comes from evidence still present in the text. A strong model can also guess correctly for two other reasons. It may simply be playing the odds: most authors of a given forum belong to a common demographic, so a population-level guess is often right. Or it may have seen the source document during pre-training: the court cases in TAB are public judgments, and PersonalReddit is drawn from real Reddit. In either case a "leak" is counted that no amount of rewriting can remove. We propose to separate these three sources by editing the evidence itself. For each document we build a minimally edited twin whose clues point to a different attribute value. We then ask whether the adversary's guess moves with the evidence, both before and after anonymization.

The payoff is a leakage number that means the same thing across adversaries, datasets, and papers. Today headline results are not comparable on exactly this point. GPT-4-based adversarial anonymization reduced adversarial accuracy on PersonalReddit from 86.7% to 41.6% (Language Models are Advanced Anonymizers). A distilled 8B model reached a privacy score of 0.263 against GPT-4o's 0.434 on SynthPAI (Self-Refining Language Model Anonymizers via Adversarial Distillation). Neither paper says how much of the residual figure an anonymizer could ever have removed. Benchmark contamination is the threat to validity here. Reproducibility is the constraint: real private text cannot be released, so whatever metric we adopt has to stay meaningful on synthetic stand-ins and in-house audits.

Core Claim

Correct adversarial guesses on anonymized text break into three parts: evidence-following, population prior, and recall of the source. Only the first is leakage that an anonymizer controls. We claim the other two are large enough on real, publicly crawled corpora (PersonalReddit, TAB) to distort comparisons between anonymizers, and smaller on synthetic corpora (SynthPAI, RAT-Bench). If so, published anonymizer rankings partly reflect which documents happen to be memorized or stereotypical, not how well each tool rewrites.

The measurement works as follows. Let $x$ be a document with true attribute value $a$, and let $x'$ be its counterfactual twin, edited by a separate model and checked by humans so that the textual evidence supports a different value $a'$. Let $T$ be the anonymizer and $A$ the adversary. Grounded leakage is

$$L_{\text{grounded}}(T, A) = \Pr\big[A(T(x)) = a \ \text{and}\ A(T(x')) = a'\big],$$

the probability that the adversary tracks the evidence in both versions. The complementary "sticky" rate, $\Pr[A(T(x)) = a \text{ and } A(T(x')) = a]$, collects guesses that ignore the edit. We split it further with two probes. A prior-only probe gives the adversary no text, only the attribute question and the platform context. A recall probe asks the adversary to name the case, thread, or applicant from the redacted text. We correct for chance using the number of values each attribute can take.

Relation to Prior Work

Two papers motivate the gap. GuardianAgent documents "hallucinated leakage", meaning correct adversary guesses that come from world knowledge rather than the text. It adds an LLM verifier that judges whether a guess is grounded before escalating redaction. That verifier is itself a language model that may have read the same documents. Our test does not ask any model whether a guess is grounded; it intervenes on the evidence and watches the output. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples shows that paraphrased test items survive n-gram decontamination: Llama-2-13B rose from about 55 to about 86 on MMLU after training on rephrased test questions. That finding matters here because anonymization is itself a paraphrase. Rewriting may break a memorized string without removing the underlying evidence, or the reverse. That work studies academic question answering, not privacy audits.

Three papers constrain the design. Unique between the lines shows that indirect identifiers keep re-identification risk near 27–29% even for strong tools. We must therefore not overcorrect and declare residual leakage illusory; its census-grounded synthetic texts are a setting where recall should be near zero, which makes them a control. TAB reports that annotators disagree about what counts as a quasi-identifier, so our counterfactual edits need human verification and should not be treated as automatically correct. PromptBench shows that small perturbations change model outputs. The edit itself may therefore move the adversary for reasons unrelated to the evidence, and we control for this with a null-edit condition described below.

The method comes from the contamination literature's habit of testing on inputs a model cannot have memorized. Examples include post-cutoff events in Unveiling Causal Reasoning in Large Language Models, phase-transition instances in Can Large Language Models Reason? A Characterization via 3-SAT, and fresh arXiv-derived tasks in ArxivRoll. We cannot scrape fresh personal data. A counterfactual twin is the closest substitute we have: it is new text with a known, controlled answer.

Research Program

The first step is building the paired corpus. We draw stratified samples from PersonalReddit, TAB, SynthPAI, and RAT-Bench. For each attribute, an editor model generates a minimal edit that redirects the evidence. The editor must come from a different model family than every adversary under test. For example, it might change a reference to a regional landmark or a life-stage detail. Annotators confirm three things: that $x'$ plausibly supports $a'$, that it is fluent, and that nothing else changed. Pairs they disagree on are dropped and counted, because the disagreement rate is itself a reportable result given TAB's findings. We also build null-edit twins, which are edits of similar size that leave the evidence pointing to $a$. These measure how much the adversary moves just because the text changed.

The second step is a decomposition study. We run several adversaries on raw text to establish the uncorrected picture: a frontier API model in the style of the PersonalReddit evaluation, a strong open model, and an 8B model. For each we estimate the grounded, prior, and recall components. The hypothesis predicts a larger sticky share on PersonalReddit and TAB than on SynthPAI and RAT-Bench. It also predicts that the sticky share grows with adversary scale on real corpora, since larger models memorize more, but not on synthetic ones.

The third step re-scores anonymizers. We run a span-masking NER scrubber, Azure-style commercial redaction, an iterative adversarial-feedback anonymizer reimplemented from Language Models are Advanced Anonymizers, a SEAL-style distilled 8B self-refiner (Self-Refining Language Model Anonymizers via Adversarial Distillation), and a GuardianAgent-style verified-escalation pipeline. We report raw adversarial accuracy alongside $L_{\text{grounded}}$ and ask whether rankings change. We also check a specific side effect. Anonymizers trained or tuned against a memorizing adversary may learn to scramble recognizable strings instead of removing evidence. If so, their raw scores improve while grounded leakage on the counterfactual twins stays flat.

The fourth step is a reproducibility protocol. We release the paired corpora built from the public and synthetic sources, along with editor prompts, the human verification guidelines, and scoring code. The design lets a data holder who cannot release text, such as a hospital auditing de-identified notes in a MIMIC-III-like setting (MIMIC-III), report grounded leakage computed in-house. That number does not depend on which adversary they happened to have, so it can be compared with published figures.

Baselines

The naive reference point is raw adversarial accuracy on anonymized text, computed exactly as in Language Models are Advanced Anonymizers, which reports 86.7% falling to 41.6% on PersonalReddit for its GPT-4 anonymizer. Every field result on PersonalReddit and SynthPAI is currently stated in these terms. We compute it on the same documents, with the same adversary prompts and decoding settings, as our metric. Our proposal "beats" it if the two metrics disagree on anonymizer rankings in a way that human-verified evidence supports. Matching it would mean the raw score was already fine.

The strongest existing correction is GuardianAgent's grounded-evidence verifier (GuardianAgent). It is the only method in the file that explicitly targets guesses not supported by the text. GuardianAgent reports at least 0.90 privacy across TAB, SynthPAI, and PII-Masking-300k. We found no reported result on Lacuna for how accurately its verifier separates grounded from ungrounded guesses, so we measure that ourselves. We treat its verifier as a classifier of grounded guesses and score it against counterfactual ground truth: a guess counts as grounded if it flips with the evidence and not under the null edit. We run it with the same backbone model and call budget as our adversary. Our metric beats it if, on documents the recall probe flags as memorized, the verifier labels sticky guesses as grounded significantly more often than on non-memorized documents. That would show the verifier inherits contamination.

The ablation that isolates our key ingredient is prior subtraction alone: raw accuracy minus a calibrated no-text guess, without counterfactual twins. If prior subtraction matches $L_{\text{grounded}}$ to within annotation noise on every corpus, the counterfactual machinery is not needed. We will report that outcome as a result.

Why not just…?

Why not just evaluate on synthetic corpora like SynthPAI, which no model memorized? This is a real threat, and the cross-corpus comparison tests it directly. Synthetic data removes source recall but not population priors. SynthPAI was also written by frontier models whose stylistic habits for each persona may be exactly what a same-family adversary picks up. Synthetic benchmarks also cannot stand in for the real, unreleasable text that institutions must audit. If the sticky share on SynthPAI and RAT-Bench is near zero and rankings there match grounded rankings on real data, the recommendation becomes "use synthetic benchmarks," and our contribution shrinks to confirming that choice.

Why not just subtract a no-text prior baseline? It is cheap and accounts for base-rate guessing, so we include it as the main ablation. It cannot detect memorization: a model that recognizes a specific ECHR judgment gives a confident, document-specific answer that no population prior predicts. It also works only on average and cannot flag individual guesses. The test is whether the residual gap between prior-adjusted accuracy and $L_{\text{grounded}}$ concentrates on documents flagged by the recall probe.

Why not just run a contamination detector on the benchmark and drop seen documents? The embedding-plus-judge detector from Rethinking Benchmark and Contamination reports F1 of 0.94–1.0 on its own test cases. But it answers "was this document seen," not "does this guess depend on having seen it." Filtering would discard most of TAB and PersonalReddit, which are public by construction, and would destroy the benchmarks without telling us how biased the remaining scores were. We still use detector flags as a stratification variable and report results on the filtered subset as an extra comparison.

Why not just use GuardianAgent's verifier as the grounding test? Its verifier is a language model judging grounding through the same channels that may be contaminated. The baseline comparison above is designed to show whether it fails in exactly the cases where memorization is likely.

Proposed Experiments and Decisive Checks

The first decisive check concerns validity of the edits. On null-edit twins the adversary should hold its guess. If its answer changes on more than a small fraction of null edits, edit artifacts dominate the signal and the metric is invalid as designed. We would then shrink edits or move to template-controlled rewriting.

The second check is the cross-corpus contrast. The sticky, non-prior share should be clearly larger on TAB and PersonalReddit than on RAT-Bench and SynthPAI, and it should grow with adversary scale only on the real corpora. If the shares are equal, memorization is not the story and the proposal reduces to prior correction.

The third check is whether rankings change. We compute Kendall rank correlation between raw-accuracy and grounded-leakage orderings of the five anonymizers on each corpus. We also test the specific prediction that adversarially tuned rewriters gain more on raw accuracy than on grounded leakage, relative to span-masking tools. A correlation near one everywhere is a clean negative result.

The fourth check concerns hard cases. SEAL reports that 40% of hard SynthPAI examples stay leaky after six rewriting rounds. We will measure what fraction of that residue is grounded. If it is mostly grounded, it is likely a case where the attribute is entangled with the document's purpose and cannot be hidden without destroying the text. If it is mostly sticky, it is evaluator error, and further rewriting is the wrong response.