Abstract: Double-blind review relies on the authors' ability and willingness to effectively anonymize their submissions. We explore anonymization effectiveness at ASE 2016, OOPSLA 2016, and PLDI 2016 by asking reviewers if they can guess author identities. We find that 74%-90% of reviews contain no correct guess and that reviewers who self-identify as experts on a paper's topic are more likely to attempt to guess, but no more likely to guess correctly. We present our findings, summarize the PC chairs' comments about administering double-blind review, discuss the advantages and disadvantages of revealing author identities part of the way through the process, and conclude by advocating for the continued use of double-blind review.
Introduction
Peer review serves as the gatekeeper of scientific integrity, yet it is susceptible to subconscious biases. Previous research indicates that knowledge of an author's identity can influence a reviewer's judgment regarding prestige, gender, and nationality. For example, studies have shown reviewers are significantly more likely to recommend a paper if the authors are well-known in the field.
Double-blind review (DBR)—where both authors and reviewers remain anonymous—is the primary mechanism used to mitigate these biases. However, critics often dismiss DBR as "security theater," arguing that in specialized research communities, reviewers can easily identify authors through topic, style, or citations. This study investigates the empirical effectiveness of DBR across three top-tier computer science conferences.
Methodology and Datasets
The researchers analyzed data from the 2016 editions of three prominent conferences: Automated Software Engineering (ASE), Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), and Programming Language Design and Implementation (PLDI).
The dataset comprised 2,819 total reviews for 818 papers. Reviewers were asked optional questions regarding whether they believed they knew the authors' identities and what informed their guesses. These responses were then cross-referenced with actual author lists and reviewer expertise levels (Expert, Knowledgeable, Informed Outsider).
Analysis of Reviewer Expertise
A key finding of the study is the "Expertise Paradox." Reviewers who identified as experts (X-level) were significantly more likely to attempt a guess than their less-specialized colleagues. However, as shown in the data, they were no more likely to be correct in their identifications. This suggests that while experts feel they can identify their peers' work, they are often overconfident in their ability to do so.
Overall Effectiveness of Anonymization
The results demonstrate that anonymity is the norm rather than the exception. In the vast majority of cases (70%–86%), reviewers did not even attempt to guess the authors' identities. This implies that reviewers are generally focused on technical content rather than attempting to unmask authors.
Key statistical highlights include:
- High Anonymity Success: 90.2% of ASE reviews, 74.4% of OOPSLA reviews, and 81.0% of PLDI reviews contained no correct author guess.
- Guess Accuracy: When reviewers did venture a guess, they were correct 72%–85% of the time. However, because guessing was infrequent, the overall unblinding rate remained low.
- Acceptance Correlation: In OOPSLA and PLDI, papers where at least one reviewer guessed correctly were more likely to be accepted. This may be because higher-quality work is more likely to come from established researchers with recognizable styles, or because reviewers are more willing to guess for papers they already favor.
Discussion and Conclusion
Despite concerns regarding administrative burden, Program Committee (PC) chairs reported that the extra effort required to manage DBR—primarily in handling conflicts of interest—was minimal and well-worth the benefits of reduced bias.
While a small minority of papers (1.5% to 13%) are easily unblinded due to being clear extensions of well-known projects, DBR remains remarkably effective for the vast majority of submissions. The authors conclude that DBR is a robust tool for maintaining fairness in the review process and advocate for its continued and expanded use in the scientific community.
Summary updated