What the Review Hierarchy Tells an Attacker: Structure-Conditioned Author Re-Identification in Double-Blind Review

Double-blind review removes author names from a manuscript. The editorial system then attaches new information to that same manuscript as it routes it: a primary subject area, keywords, an area chair whose batch holds twenty or so closely related papers, and reviewers chosen because they work nearby. Each of these labels is a quasi-identifier, meaning an attribute that is harmless alone but narrows down who someone is when combined with others. Prior attacks on double-blind review guess the author from text and citations against a large, flat pool of candidate authors. Our claim is that the realistic attacker is someone sitting at a given tier of the hierarchy, and the routing information that tier sees shrinks the candidate pool far more than any text feature does. The question is how much this raises re-identification risk, and for whom.

We propose to measure re-identification risk tier by tier, as reviewer, area chair, and program chair, on real OpenReview submissions with recoverable ground-truth authors. The central test is decisive and cheap: compare attacks over pools defined by the hierarchy against attacks over random pools of the same size. Structural pools are small, which helps the attacker. They are also topically homogeneous, since everyone in the area uses the same vocabulary and cites the same papers, which hurts the attacker. Which effect wins cannot be read off existing accuracy-versus-pool-size curves. The answer also determines whether venues should treat area labels and assignment batches as privacy-relevant disclosures rather than neutral logistics.

Core Claim

Re-identification risk in double-blind review is a property of each tier's information set, not of the manuscript alone. We expect the routing metadata exposed at a tier to multiply with stylistic and bibliographic signals. If so, the effective anonymity set shrinks from thousands of plausible authors to a handful at the area-chair tier, and the shrinkage is concentrated on prolific authors. We define the effective anonymity set for a submission as $\hat{k} = 2^{H}$, where $H$ is the entropy in bits of the attacker's posterior distribution over candidate authors. When the attacker is certain, $\hat{k}=1$; when it spreads probability evenly over $n$ authors, $\hat{k}=n$. This puts our results on the same scale as k-anonymity: a record is k-anonymous when it is indistinguishable from at least $k-1$ others.

Relation to Prior Work

Three empirical studies motivate the question, and each holds the hierarchy fixed or ignores it. Effectiveness of Anonymization in Double-Blind Review surveyed human reviewers at ASE, OOPSLA and PLDI 2016. It found that only 1.5% to 13% of papers were unblinded by any reviewer, and that self-declared experts guessed more often without guessing better. That study measures what humans choose to do, not what a tier's information permits. De-anonymization of authors through arXiv submissions during double-blind review showed on 5,050 ICLR submissions that arXiv preprints act as a linkage database, and that the resulting acceptance boost goes to high-reputation authors. That work is observational and treats the preprint as the only channel. Cracking Double-Blind Review: Authorship Attribution with Deep Learning supplies our main attack method. It is a two-stream classifier that combines a DistilBERT encoding of the first 512 words with a histogram over cited authors. It reaches 73.4% top-1 and 87.6% top-5 accuracy over 2,070 arXiv authors. Its pools range from 100 to 2,070 authors but are not defined by any editorial structure, so its pool-size curve conflates "smaller" with "random." We separate the two.

Two lines of work supply method or constrain design. Large-scale online deanonymization with LLMs builds an extract-search-verify pipeline: a language model extracts traits, embedding search retrieves candidates, and a reasoning model re-ranks them. It reaches 67% recall at 90% precision linking Hacker News users to LinkedIn. We reuse this pipeline as a second, stronger attacker. Probabilistic Reasoning with LLMs for Privacy Risk Estimation shows that plain chain-of-thought prompting underestimates joint-attribute risk in 28% of cases, against 12% for structured Bayesian decomposition. This constrains us. We will not ask a language model to judge "how identifying" a paper is. We measure attacks against ground truth and compute risk from attacker posteriors.

The peer-review mechanism-design literature treats reviewer inputs as scores or assignment graphs. That includes Calibration with Privacy in Peer Review, which hides reviewer calibration parameters, and On Strategyproof Conference Peer Review, which designs assignments resistant to self-interested reviewing. Neither treats the assignment itself as a disclosure about authors. We do. Finally, the text-sanitization work on the Text Anonymization Benchmark and Neural Text Sanitization with Explicit Measures of Privacy Risk establishes that privacy is entity-level and that blanket masking wastes utility. We borrow that view to argue that the right defense may sit in routing metadata rather than in the manuscript text.

Research Program

Data and ground truth. We start from the OpenReview forums underlying ReviewArena, which covers 51,529 papers across NeurIPS, ICLR, ICML, CoRL, COLM, EMNLP and TMLR. We re-pull each forum's primary-area and keyword fields. Ground-truth authors come from de-anonymized accepted papers and, at venues that publish them, rejected papers. For each candidate author we build a writing-and-citation profile from arXiv papers posted strictly before that venue's submission deadline. We exclude the submission's own preprint so that the text-and-structure channel is measured separately from the preprint channel studied by Bharadhwaj et al.

Tier information sets. We simulate three tiers. The reviewer tier sees the anonymized text plus the area and keyword labels; at bidding time it also sees titles and abstracts across the area. The area-chair tier additionally sees its assignment batch. We reconstruct batches by clustering same-area submissions into groups of realistic size, because true batches are private. It also sees the expertise profiles of the assigned reviewers, approximated by those reviewers' published papers. The program-chair tier sees all area structure. For every tier, the candidate pool is the set of authors whose prior work places them inside that tier's visible structure. Because authors in the same area tend to share prose and citation habits, we keep candidate profiles disjoint from the test paper's content.

Attackers. We run four attackers of increasing strength: citation-overlap nearest neighbour, TF-IDF prose nearest neighbour, the two-stream classifier of Bauersfeld et al. retrained per pool, and the extract-search-verify pipeline restricted to the tier's pool. All four output a posterior over candidates, from which we compute top-1 and top-5 accuracy and $\hat{k}$.

Policy levers. We test three counterfactual routing designs. The first coarsens the area labels shown to reviewers by merging adjacent areas. The second builds area-chair batches that mix two areas. The third hides keywords until after bidding. For each, we measure the drop in risk and the cost in reviewer-to-paper expertise match, computed as text similarity between reviewer papers and the submission. The lever is worth adopting only if it removes much of the structural risk at small matching cost.

Baselines

The naive reference point is the current flat-pool assumption. We run the attack against all authors active at the venue that year, using text alone with no routing metadata. This corresponds to what a venue implicitly assumes when it says that removing names is enough. Every structural gain is reported relative to it.

The strongest existing method is the two-stream attribution model of Cracking Double-Blind Review. It reports 73.4% top-1 and 87.6% top-5 accuracy over 2,070 candidate authors on its arXiv benchmark. We re-implement it and train it on our pre-deadline arXiv profiles with the same encoder size, the same 512-word input, and the same tuning budget for every pool definition. Any difference between conditions then comes from the pool rather than the model. We found no reported result on Lacuna for attribution over structurally defined review pools. Beating it means a statistically reliable rise in top-1 accuracy, or a fall in median $\hat{k}$, at the area-chair tier relative to this model on the flat venue pool. The extract-search-verify pipeline of Large-scale online deanonymization with LLMs is a second strong competitor and runs under the same pools and a fixed per-paper query budget. Its reported 67% recall at 90% precision was measured on a different task, so it serves as context, not a target.

The ablation that isolates our key ingredient is the size-matched random pool. For every structural pool of size $n$, we draw candidate pools of $n$ venue authors at random and rerun the identical attacker. The gap between structural and random pools of the same size is the effect of routing structure itself, net of mere pool shrinkage. A citation-overlap-only attacker is also run in every condition, to test whether bibliography alone explains the result.

Why not just…?

Why not just read structural risk off the existing accuracy-versus-pool-size curve? Bauersfeld et al. already report accuracy for pools from 100 to 2,070 authors, so one could plug in an area's size. Their pools are not topically homogeneous, however, and same-area authors share vocabulary and citations, which should make them harder to tell apart. The size-matched ablation settles this directly. If structural and random pools of equal size give the same accuracy, the curve suffices and our contribution reduces to measuring pool sizes per tier. We would report that honestly.

Why not just rely on reviewers not trying? Effectiveness of Anonymization in Double-Blind Review found that most reviewers make no guess. That measures intent under 2016 conditions, when running an attack took effort. Today an attack is a script, and area chairs and bidding systems handle many papers at once. Our tier-level numbers bound what is possible, not what is typical. The two are complementary: if possible risk is high while observed guessing is low, policy should not rest on goodwill.

Why not just ban preprints or strip self-citations? Bauersfeld et al. report that removing self-citations cuts reference-stream accuracy by over 10 points, while prose alone still exceeds 60%. Our design already excludes the submission's own preprint, so the residual risk we measure is precisely what such policies leave behind. We additionally run every condition with self-citations removed to quantify what remains.

Why not just use citation overlap? This is a real threat to the framing. In a narrow area, a bibliography may pin down the research group on its own, making neural stylometry unnecessary. We run the citation-only attacker everywhere. If it matches the two-stream model at the area-chair tier, the finding becomes "routing plus bibliography suffices." That would point the defense at bibliographies and labels, not prose rewriting.

Proposed Experiments

The first experiment tests the core claim. We report top-1 and top-5 accuracy and median $\hat{k}$ at each tier for all four attackers, and compare structural pools against size-matched random pools. The hypothesis fails if the area-chair tier gives no gain over the flat venue pool. It is substantively refined if structural pools underperform random pools of the same size.

The second experiment stratifies risk by author productivity, using the count of prior arXiv papers. This tests whether structural re-identification falls hardest on the prolific authors who, per Bharadhwaj et al., benefit when unblinded. Concentrated risk would mean the hierarchy amplifies prestige bias rather than merely leaking identity.

The third experiment evaluates the three routing levers. For each, we report the change in median $\hat{k}$ against the loss in reviewer expertise match. The success criterion is a lever that restores at least half of the flat-pool anonymity set while costing little in matching quality.

The main validity check concerns the reconstructed area-chair batches, which approximate private assignments. We vary batch size and clustering method and report how sensitive the area-chair-tier results are. Results are claimed only where conclusions hold across these choices.