Effectiveness of anonymization in double-blind review

Claire Le Goues, Yuriy Brun, Sven Apel, Emery D. Berger, Sarfraz Khurshid, Yannis Smaragdakis

TL;DR

Double-blind review (DBR) is a powerful tool for mitigating reviewer bias, but does it actually work in practice? This study of three major computer science conferences finds that anonymization is remarkably effective: between 74% and 90% of reviews contained no correct author guesses. Even self-identified "experts" were no better at unmasking authors than non-experts, suggesting that the "open secret" of author identity in niche fields is largely a myth.

What Problem Does This Paper Address?

Peer review is the gatekeeper of science, yet it is far from objective. Previous studies have shown that knowing an author's identity can trigger subconscious biases regarding prestige, gender, and nationality. For instance, one study found reviewers were 1.76x more likely to recommend a paper if the authors were "famous."

Double-blind review—where both authors and reviewers are anonymous—is the standard solution. However, critics often argue that DBR is "security theater." They claim that in specialized research communities, reviewers can easily guess who wrote a paper based on the topic, writing style, or citations. This paper investigates whether those claims hold water by analyzing real-world data from three top-tier conferences.

Methods

The authors collected data from the 2016 editions of three prominent conferences:

  • ASE (Automated Software Engineering)
  • OOPSLA (Object-Oriented Programming, Systems, Languages, and Applications)
  • PLDI (Programming Language Design and Implementation)

Reviewers were asked optional questions on their review forms: Do you think you know who the authors are? If so, who? What informed your guess? The researchers then compared these guesses against the actual author lists. They also surveyed the Program Committee (PC) chairs to assess the administrative burden of managing a double-blind process.

Datasets

The study utilized the internal review data from the three aforementioned conferences. This included:

  • 2,819 total reviews across the three venues.
  • 818 total papers reviewed.
  • Self-reported reviewer expertise levels (Expert, Knowledgeable, Informed Outsider).
  • Acceptance/rejection decisions for all submissions.

Key Insights and Ideas

  • The Expertise Paradox: Reviewers who identified as "experts" (X-level) were significantly more likely to attempt a guess than their less-specialized colleagues. However, they were no more likely to be correct. This suggests that while experts feel they can identify their peers' work, they are often overconfident.
  • Anonymity is the Norm, Not the Exception: In the vast majority of cases (70%–86%), reviewers didn't even try to guess. This implies that reviewers are generally focused on the technical content rather than playing detective.
  • The "Easy to Unblind" Minority: A small subset of papers (about 1.5% in ASE to 13% in PLDI) had three or more reviewers guess correctly. These papers often represented clear extensions of well-known ongoing projects or utilized specific, unique datasets.
  • Low Administrative Cost: Despite fears that DBR is a logistical nightmare, PC chairs reported that the extra effort—mostly managing conflicts of interest—was "minimal" and "well-worth the benefits."

Key Results

  • High Success Rate: 90.2% of ASE reviews, 74.4% of OOPSLA reviews, and 81.0% of PLDI reviews contained no correct author guess.
  • Guess Accuracy: When reviewers did venture a guess, they were correct 72%–85% of the time. However, because so few reviewers guessed at all, the overall "unblinding" rate remained low.
  • Acceptance Bias: In OOPSLA and PLDI, papers where at least one reviewer guessed correctly were more likely to be accepted. The authors suggest this might be because higher-quality work is more likely to come from established researchers whose style is recognizable, or because reviewers are more willing to guess (and guess correctly) for papers they already like.

Limitations and Future Work

The authors acknowledge several limitations:

  • The "Observer Effect": Asking reviewers if they can guess might have prompted them to look for clues they otherwise would have ignored.
  • Self-Reporting Bias: Reviewers might have been hesitant to admit they knew the authors if they felt it compromised their perceived objectivity.
  • Evolution of Anonymization: As these conferences had only recently adopted DBR in 2016, the authors posit that "anonymization hygiene" among authors will likely improve over time, making DBR even more effective.

Future work could involve more granular analysis of why certain papers are easier to unblind and whether specific anonymization guidelines (like those used in PLDI) are more effective than others.

Summary updated

Abstract

Double-blind review relies on the authors' ability and willingness to effectively anonymize their submissions. We explore anonymization effectiveness at ASE 2016, OOPSLA 2016, and PLDI 2016 by asking reviewers if they can guess author identities. We find that 74%-90% of reviews contain no correct guess and that reviewers who self-identify as experts on a paper's topic are more likely to attempt to guess, but no more likely to guess correctly. We present our findings, summarize the PC chairs' comments about administering double-blind review, discuss the advantages and disadvantages of revealing author identities part of the way through the process, and conclude by advocating for the continued use of double-blind review.
BibTeX
@article{goues2018effectiveness,
  author = {Claire Le Goues and Yuriy Brun and Sven Apel and Emery D. Berger and Sarfraz Khurshid and Yannis Smaragdakis},
  title = {{Effectiveness of anonymization in double-blind review}},
  journal = {Communications of the ACM},
  year = {2018},
  url = {https://openreview.net/forum?id=OW72kd3GCH},
}

Loading related content...

Figure