Large language models can read an anonymous forum comment and guess the author's age, sex, city or income, even after names and other direct identifiers are removed. This finding has driven a line of LLM-based anonymizers whose success is scored by one number: how often an adversary model still guesses the attribute correctly. That number mixes two different things. A correct guess can mean the text gave the author away. It can also mean the model fell back on a stereotype that happened to be right, such as guessing "male" in a car-repair forum. Recent work already suspects the second case is common. The proposal here measures it directly and cheaply. For each synthetic author we generate a "twin" who differs in exactly one attribute, write both their comment histories with the same random seeds and thread context, and ask whether the adversary's guess follows the attribute or stays the same across the twins.
The core claim is that counterfactual twin evaluation gives the first ground-truth separation between text-grounded leakage and prior-driven guessing in personal attribute inference. Two further problems can then be measured that current benchmarks hide. The first is how much of the reported privacy gain from anonymizers comes from removing real cues rather than from pushing the adversary back onto its priors. The second is how often anonymizers "succeed" by fabricating false cues that point to a different attribute value, which is a hallucination that looks like privacy. Because everything is built from synthetic personas, the benchmark can be released, unlike the private corpus on which most headline numbers rest.
What the four directions contribute
Implicit attribute inference is the core mechanism. Language Models are Advanced Anonymizers runs an iterative game in which an adversary LLM infers attributes and explains its reasoning, and an editor rewrites the cited cues. This lowers GPT-4 inference accuracy on the private PersonalReddit corpus from 86.7% to 41.6% over five rounds, compared with 66.3% after Azure span masking. Large Language Models are Anonymizers uses the same adversary–editor loop. Both treat every correct guess as a leak to be edited away, and that assumption is what this proposal tests.
The hallucination direction supplies the doubt. GuardianAgent argues that adversaries often guess correctly from general priors or invented connections, and it adds an LLM verifier that checks for textual grounding before the anonymizer escalates. This is the closest existing work. It differs from ours in that the verifier is itself an LLM judgment with no external ground truth, so it can hallucinate grounding just as the adversary can. Do LLMs Really Memorize Personally Identifiable Information? supplies the method template. It controls for lexical cues and shows that apparent PII memorization is mostly reconstruction from naming patterns, with recovery collapsing to near zero under cue control. We carry that "control the cue, then re-measure" logic from explicit identifiers in pretraining data over to implicit attributes in conversation. Two papers motivate the fabrication analysis. RedacBench reports that rewriting models invent false content to hide secrets. It also finds that aggressive iterative redaction trades a security score of 80.9% for a utility score of 37.6% with GPT-5-mini. Do Language Models Know When They're Hallucinating References? shows that consistency under repeated indirect questioning can detect fabrication. We use that as a baseline, though we expect it to fail here for a specific reason given below.
The reproducibility direction sets the regime. A Synthetic Dataset for Personal Attribute Inference introduced SynthPAI: 7,823 GPT-4-generated comments from 300 personas, which humans could tell from real Reddit only 51.9% of the time. Eighteen models rank the same way on SynthPAI as on PersonalReddit. That rank agreement validates the leaderboard, but it does not show that the measured accuracy reflects leakage. Our twin construction needs control over the generating persona, and only a synthetic pipeline can give that. In this design, synthetic data is the experimental instrument, not just a substitute for data that cannot be shared. Privasis acts as a constraint. It warns that synthetic generators carry the demographic biases of their seeds, and we have to measure that risk rather than assume it away.
Engagement–quality misalignment plays a smaller, hypothesis-generating role. PopALM finds that engagement on Weibo rewards specific, concrete viewpoints over generic, safe replies. SynthPAI explicitly omits upvotes and karma. The concrete personal detail that earns engagement is plausibly the same detail that leaks attributes. If so, synthetic benchmarks that lack engagement pressure underestimate grounded leakage. We test this with a single controlled knob and do not claim more.
Core Claim and Measure
Take a persona $p$, an attribute $A$ with true value $a$, and a twin persona $p'$ identical to $p$ except that $A = a'$. Generate comment histories $x$ and $x'$ with the same thread context, sampling seeds and other persona fields. Let $f$ be the adversary's predicted value. Counterfactual leakage is the rate at which the adversary is right on both twins:
$$L_{\text{cf}} = \Pr\big[f(x) = a \;\wedge\; f(x') = a'\big].$$
A guess driven only by priors gives the same answer for both twins, so it can be right on at most one of them and adds nothing to $L_{\text{cf}}$. Ordinary accuracy gives credit for that same guess half the time. The gap between raw accuracy and $L_{\text{cf}}$, scaled by the chance of hitting one twin by luck, estimates how much of a reported accuracy is prior-driven. For attributes with many values, such as location or occupation, we generate twin sets of size $k$ and count joint correctness across the set. Here prior-only guessing scores at most $1/k$ of ordinary accuracy.
Style differences that the generator produces because of the attribute count as leakage. That is the intended behavior, since real authors also write differently by age or sex. The risk is that the generator over-produces stereotyped cues. We address that in the research program.
Research Program
The first component is the twin generator. We extend the SynthPAI pipeline so that each of the 300 personas spawns twins along each of the 8 attributes. Thread structure, subreddit and the other personas' replies stay fixed, and only the target author's comments are regenerated. We use the same generator family as SynthPAI so that results stay comparable to its leaderboard. We audit the generator in two ways. The first audit checks minimality: twin comments should differ only in spans an annotator links to the changed attribute. The second checks stereotype inflation: human raters compare twin comments against real-world plausibility, and we report per-attribute rates of exaggerated cues. If inflation is high, $L_{\text{cf}}$ overstates leakage, and we will say so per attribute.
The second component re-scores the evaluation of anonymizers. We run the adversarial anonymizer of Language Models are Advanced Anonymizers, GuardianAgent, span masking and a single-pass prompt on both twins of each pair. We then report raw accuracy and $L_{\text{cf}}$ before and after each round. The quantity of interest is the share of each method's accuracy drop that comes from destroying real signal (a fall in $L_{\text{cf}}$) rather than from removing priors. Removing priors cannot happen through text editing in the twin design, so any accuracy drop with no change in $L_{\text{cf}}$ points to measurement noise or to fabrication.
The third component measures decoy fabrication. An anonymizer can lower accuracy by inserting a false cue, for example turning a nurse into a teacher. We flag a pair as a decoy when two things hold. The adversary confidently predicts a specific wrong value on the anonymized text. And a proposition-level diff, in the style of RedacBench, finds an inserted proposition that supports that wrong value and is absent from the original. Decoy rate becomes a reported hallucination metric next to privacy and utility.
The fourth component is the engagement probe. We generate a second twin corpus in which personas are told to write to win upvotes in their subreddit, and we compare $L_{\text{cf}}$ with the neutral corpus. This tests the hypothesis drawn from PopALM inside a simulation. Its value is in ruling the effect in or out for synthetic benchmarks. It makes no claim about real platforms.
Baselines
The naive reference point is raw adversary accuracy as reported today, with GPT-4 as the adversary. On SynthPAI it reaches 76% overall, with sex above 90% and income around 66% (A Synthetic Dataset for Personal Attribute Inference). We rerun this on the twin corpus with the same adversary prompts. The proposal "beats" it if $L_{\text{cf}}$ and raw accuracy rank attributes or anonymizers differently in a way the audits confirm.
The simplest real competitor is a prior-only adversary. It sees the thread, the subreddit and the other users' comments, but not the target author's text, and its accuracy is subtracted from raw accuracy. It uses the same model and decoding settings as the main adversary. If this subtraction tracks $L_{\text{cf}}$ closely across attributes and anonymizers (rank correlation near one), the twin machinery is unnecessary, and we would recommend the cheaper baseline. No reported result found on Lacuna for a prior-only adversary on SynthPAI or PersonalReddit.
The strongest existing method for separating grounded from lucky inferences is GuardianAgent's grounding verifier. It reports privacy of at least 0.90 on TAB, SynthPAI and PII-Masking-300k (GuardianAgent), but the file gives no figure for verifier agreement with ground-truth grounding. We run its verifier on every correct adversary guess and score its "grounded" labels against twin outcomes. A guess counts as truly grounded when the adversary is right on both twins. We use precision and recall at matched LLM-call budgets. Our method beats it if twin-based labels expose systematic verifier errors, such as accepting a stereotype as grounding.
The self-consistency baseline follows Do Language Models Know When They're Hallucinating References?. It ensembles direct and indirect repeated queries and treats inconsistent answers as ungrounded. The paper reports that the ensemble gives the best detection AUC, but the file gives no numeric value. We score it with the same AUC protocol against twin labels.
The key ablation removes seed and context matching. Twins are generated with independent seeds and threads, which leaves only the attribute swap. If $L_{\text{cf}}$ estimates become much noisier or shift, matched generation is the ingredient doing the work.
Why not just…?
Why not just trust GuardianAgent's verifier? The verifier asks an LLM whether a guess is grounded in the text, and that is exactly the kind of judgment the hallucination literature says models make with unearned confidence. Evaluating large language models on medical evidence summarization documents models stating tentative findings as certain. This is a real threat, though, because the verifier may be good enough. The verifier-agreement experiment settles it. High precision and recall against twin labels would turn this project into a calibration study that validates the verifier, and we would report it that way.
Why not just subtract a majority-class or prior-only baseline? Priors are conditional on context. A prior-only adversary that sees the subreddit already captures much of this, which is why it is our primary simple baseline and not a strawman. It can still fail in two ways. It cannot tell when the author's text reinforces a stereotype rather than adding information. And it says nothing about how anonymization moves the adversary between the two kinds of guess. The rank-correlation test decides whether these failures matter in practice.
Why not just use self-consistency to detect lucky guesses? Consistency measures how stable the model's belief is, not where it comes from. A model guessing "male" from forum priors will give that answer every time it is asked. We therefore expect self-consistency to misclassify confident, prior-driven guesses as grounded, and the twin labels test that prediction directly.
Why not just evaluate on real data with human leakage annotations? PersonalReddit cannot be released, and humans annotating which cues are "really" informative face the same confound between cue and prior. Real data also offers no twin counterfactual. The cost is realism: synthetic twins may inflate cues. That is why the stereotype audit is reported per attribute and not just averaged away.
Decisive Checks
The project succeeds if three things hold. First, on at least some attributes, ideally the ones with strong population skews such as sex and income, a substantial share of raw accuracy disappears under $L_{\text{cf}}$, and this survives the stereotype audit. Second, for the best iterative anonymizer, the share of its accuracy reduction explained by a fall in $L_{\text{cf}}$ differs measurably from the share for span masking, and its decoy rate is reported as nonzero or bounded. Third, the cheaper baselines, prior-only subtraction and the GuardianAgent verifier, either agree with twin labels, which makes them validated tools, or disagree in identifiable ways. The project fails cleanly if the generator cannot produce minimal twins, meaning the audits show twins differing in far more than the attribute. In that case the measure is uninterpretable, and the finding is a limitation of current synthetic privacy benchmarks. The engagement probe is a secondary result. A rise in $L_{\text{cf}}$ under engagement pressure would argue that SynthPAI-style benchmarks should model platform incentives. No change would remove one concern about them.
References
- Language Models are Advanced Anonymizers
- GuardianAgent: Policy-Conditioned Risk-Adaptive Anonymization with Verified Adversarial Escalation
- Do LLMs Really Memorize Personally Identifiable Information? Revisiting PII Leakage with a Cue-Controlled Memorization Framework
- RedacBench: Can AI Erase Your Secrets?
- Do Language Models Know When They're Hallucinating References?
- A Synthetic Dataset for Personal Attribute Inference
- Privasis: Synthesizing the Largest "Public" Private Dataset from Scratch
- PopALM: Popularity-Aligned Language Models for Social Media Trendy Response Prediction
- Evaluating large language models on medical evidence summarization