Introduction
The peer-review process is the cornerstone of scientific publishing, yet it faces unprecedented pressure. With the surge in submissions to major AI conferences like ICLR and NeurIPS, conference chairs and editors are increasingly overwhelmed. A critical bottleneck in this process is identifying and mediating "reviewer disagreements"—instances where reviewers hold fundamentally opposing views on a paper's merits, such as its theoretical soundness or experimental validity.
Manually identifying these contradictions across thousands of technical, multi-page reviews is labor-intensive. To address this, researchers have introduced ContraSciView, a large-scale dataset and a corresponding automated system designed to flag these "lock horn" moments, enabling editors to focus on the most contentious aspects of a submission.
Methodology: A Two-Stage Detection System
The authors propose a baseline system that moves beyond simple numerical score discrepancies to analyze the semantic content of reviews. The process involves two distinct stages:
- Aspect Sentiment Model: Utilizing a Multi-Instance Multi-Label Learning Network (MIMLLN), the system decomposes reviews into individual comments. Each comment is categorized into one of eight aspects (e.g., Clarity, Soundness, Substance) and assigned a sentiment (Positive or Negative).
- Contradiction Classifier: The system identifies "Sentiment Disparity Aspect Pairs" (SDAPs)—cases where two reviewers address the same aspect with opposing sentiments. These pairs are processed by a transformer-based classifier (e.g., RoBERTa or XLNet) trained on Natural Language Inference (NLI) tasks to determine if a logical contradiction exists.
The ContraSciView Dataset
To train and evaluate these models, the study introduces ContraSciView, a comprehensive dataset curated from open-review platforms:
- Source: Approximately 8.5k papers from ICLR (2017–2020) and NeurIPS (2016–2019).
- Scale: The dataset comprises roughly 28k review pairs and nearly 50k individual comment pairs.
- Annotation: Doctoral students manually labeled contradictions, achieving a substantial inter-annotator agreement (Cohen’s kappa of 0.62).
Analysis of the dataset reveals that Clarity is the primary "battlefield" for reviewer disagreements, while aspects like Replicability see fewer contradictions, often because they are mentioned less frequently.
Experimental Results
The experiments highlight the importance of domain-specific training. Standard NLI models trained on general text (like Wikipedia) struggle with the technical nuances of peer reviews.
Key findings include:
- Superior Performance: A RoBERTa-large model trained on ContraSciView achieved the highest performance, with an F1-score of 77.55% and an accuracy of 91.50%.
- Dataset Impact: Training on the specialized ContraSciView dataset provided a significant boost over general NLI datasets (e.g., RoBERTa-large F1 improved from 71.14% to 77.55%).
- LLM Comparison: Zero-shot Large Language Models like ChatGPT and Google BARD lagged behind the fine-tuned baseline by 9 to 12 F1 points, respectively, often producing false positives by misinterpreting differing perspectives as logical contradictions.
Limitations and Future Directions
While the current system excels at identifying explicit contradictions, several challenges remain. The model currently struggles with "implicit" contradictions that require deeper contextual understanding. Furthermore, the baseline evaluates comment pairs in isolation; future iterations aim to incorporate the full review context and the paper's content. Finally, the researchers plan to categorize disagreements by severity to help editors prioritize the most critical conflicts.
Summary updated