When Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews

Sandeep Kumar, Tirthankar Ghosal, Asif Ekbal

Introduction

The peer-review process is the cornerstone of scientific publishing, yet it faces unprecedented pressure. With the surge in submissions to major AI conferences like ICLR and NeurIPS, conference chairs and editors are increasingly overwhelmed. A critical bottleneck in this process is identifying and mediating "reviewer disagreements"—instances where reviewers hold fundamentally opposing views on a paper's merits, such as its theoretical soundness or experimental validity.

Manually identifying these contradictions across thousands of technical, multi-page reviews is labor-intensive. To address this, researchers have introduced ContraSciView, a large-scale dataset and a corresponding automated system designed to flag these "lock horn" moments, enabling editors to focus on the most contentious aspects of a submission.

Methodology: A Two-Stage Detection System

The authors propose a baseline system that moves beyond simple numerical score discrepancies to analyze the semantic content of reviews. The process involves two distinct stages:

  1. Aspect Sentiment Model: Utilizing a Multi-Instance Multi-Label Learning Network (MIMLLN), the system decomposes reviews into individual comments. Each comment is categorized into one of eight aspects (e.g., Clarity, Soundness, Substance) and assigned a sentiment (Positive or Negative).
  2. Contradiction Classifier: The system identifies "Sentiment Disparity Aspect Pairs" (SDAPs)—cases where two reviewers address the same aspect with opposing sentiments. These pairs are processed by a transformer-based classifier (e.g., RoBERTa or XLNet) trained on Natural Language Inference (NLI) tasks to determine if a logical contradiction exists.
The two-stage pipeline for contradiction detection, moving from raw reviews to identified contradictions.

The ContraSciView Dataset

To train and evaluate these models, the study introduces ContraSciView, a comprehensive dataset curated from open-review platforms:

  • Source: Approximately 8.5k papers from ICLR (2017–2020) and NeurIPS (2016–2019).
  • Scale: The dataset comprises roughly 28k review pairs and nearly 50k individual comment pairs.
  • Annotation: Doctoral students manually labeled contradictions, achieving a substantial inter-annotator agreement (Cohen’s kappa of 0.62).

Analysis of the dataset reveals that Clarity is the primary "battlefield" for reviewer disagreements, while aspects like Replicability see fewer contradictions, often because they are mentioned less frequently.

Experimental Results

The experiments highlight the importance of domain-specific training. Standard NLI models trained on general text (like Wikipedia) struggle with the technical nuances of peer reviews.

Performance comparison of various transformer models. Models trained on the ContraSciView dataset significantly outperform those fine-tuned on general NLI benchmarks.

Key findings include:

  • Superior Performance: A RoBERTa-large model trained on ContraSciView achieved the highest performance, with an F1-score of 77.55% and an accuracy of 91.50%.
  • Dataset Impact: Training on the specialized ContraSciView dataset provided a significant boost over general NLI datasets (e.g., RoBERTa-large F1 improved from 71.14% to 77.55%).
  • LLM Comparison: Zero-shot Large Language Models like ChatGPT and Google BARD lagged behind the fine-tuned baseline by 9 to 12 F1 points, respectively, often producing false positives by misinterpreting differing perspectives as logical contradictions.

Limitations and Future Directions

While the current system excels at identifying explicit contradictions, several challenges remain. The model currently struggles with "implicit" contradictions that require deeper contextual understanding. Furthermore, the baseline evaluates comment pairs in isolation; future iterations aim to incorporate the full review context and the paper's content. Finally, the researchers plan to categorize disagreements by severity to help editors prioritize the most critical conflicts.

Summary updated

Abstract

To this date, the efficacy of the scientific publishing enterprise fundamentally rests on the strength of the peer review process. The journal editor or the conference chair primarily relies on the expert reviewers' assessment, $\textit{identify points of agreement and disagreement}$ and try to reach a consensus to make a fair and informed decision on whether to accept or reject a paper. However, with the escalating number of submissions requiring review, especially in top-tier Artificial Intelligence (AI) conferences, the editor/chair, among many other works, invests a significant, sometimes stressful effort to mitigate reviewer disagreements. Here in this work, we introduce a novel task of automatically identifying contradictions among reviewers on a given article. To this end, we introduce $\textit{ContraSciView}$, a comprehensive review-pair contradiction dataset on around 8.5k papers (with around 28k review pairs containing nearly 50k review pair comments) from the open review-based ICLR and NeurIPS conferences. We further propose a baseline model that detects contradictory statements from the review pairs. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically. We make our dataset and code public for further investigations.
BibTeX
@inproceedings{kumar2023when,
  author = {Sandeep Kumar and Tirthankar Ghosal and Asif Ekbal},
  title = {{When Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=eFnBXtZXIH},
}
Figure