The lack of theory is painful: Modeling Harshness in Peer Review Comments

Rajeev Verma, Rajarshi Roychoudhury, Tirthankar GhosalAACL/IJCNLP 2022 · OpenReview

TL;DR

This paper tackles the problem of overly harsh and unconstructive peer review comments, which can be a traumatic experience for researchers. The authors introduce the first dataset of peer review comments with real-valued "harshness" scores, created using a novel Best-Worst-Scaling approach, to enable automated moderation and foster more constructive feedback.

What Problem Does This Paper Address?

The peer-review system is the bedrock of scientific validation, but it's increasingly under stress. A significant issue is the "power imbalance" where reviewers, often anonymous, can deliver overly critical or even obtuse feedback without accountability. This can make the process "fraught and traumatic" for authors, particularly early-career researchers. While critical feedback is essential, the way it's delivered can be more damaging than the critique itself. The paper aims to address this by modeling and quantifying "harshness" in review comments, moving beyond simple sentiment analysis to capture the subtle nuances of unconstructive criticism.

Methods

The core of the paper's methodology lies in defining and measuring "review harshness." They break it down into two orthogonal dimensions:

  • Critical Stance: How reviewers express their judgments, using markers like evidentiality (signaling certainty) and presence (using first-person pronouns).
  • Evaluative Focus: The actual content of the comments, identifying problematic practices like "blank assertions," "intellectual laziness" (e.g., overemphasizing novelty), and "policy entrepreneurism" (imposing personal agendas).

To quantify harshness, they employ Best-Worst-Scaling (BWS). Instead of asking annotators to rate comments on an absolute scale, BWS presents annotators with sets of four comments and asks them to identify the "best" (most harsh) and "worst" (least harsh). This comparative approach is argued to be more reliable for subjective qualities like harshness, especially in the formal context of academic writing. They then convert these pairwise comparisons into a continuous, real-valued harshness score.

For dataset creation, they use an Active Learning strategy. This involves starting with a small "seed" dataset and iteratively selecting the most informative (ambiguous) examples for annotation. This helps efficiently build a dataset that captures the subtle, implicit harshness often found in academic reviews, rather than just overtly offensive language.

Datasets

The paper introduces a newly created dataset of 1000 peer-review sentences annotated with real-valued harshness scores. This dataset is a combination of:

  • Sentences from the ShitMyReviewersSay Twitter handle, which provides examples of overtly harsh and disparaging comments.
  • Sentences from the Peer-Review-Analyze dataset (Ghosal et al., 2022), which contains reviews from the ICLR conference and includes sentiment labels.

The authors specifically leverage negative sentiment sentences from Peer Review Analyze and combine them with ShitMyReviewersSay examples to create a more comprehensive dataset that captures a spectrum of harshness, from extreme to subtle.

Key Insights and Ideas

  • Harshness is more than just negativity: The paper highlights that harshness in peer reviews is a distinct phenomenon from simple negative sentiment. It's about how criticism is delivered, not just that it's critical.
  • Subtlety is key in academic harshness: Unlike overt hate speech, harshness in academic reviews is often subtle, embedded in formal language, making it harder to detect with standard NLP tools.
  • Best-Worst-Scaling is effective for subjective tasks: The BWS method proved to be a reliable way to elicit continuous harshness scores, outperforming simpler annotation schemes for this nuanced task.
  • Active Learning is crucial for efficiency: Using active learning significantly reduced the annotation effort required to build a high-quality dataset for modeling subtle harshness.
  • BERT models show promise, but there's room for improvement: While BERT-based models performed best, the fact that HateBERT (a model specifically trained for abusive language) didn't significantly outperform regular BERT suggests that academic review harshness is a unique challenge.

Key Results

  • The paper successfully created a dataset of 1000 peer-review sentences with real-valued harshness scores, ranging from -1.0 (most harsh) to 1.0 (least harsh).
  • The distribution of harshness scores in their dataset is moderately left-skewed, indicating that extremely harsh comments are less common than standard, less critical ones in academic reviews.
  • BERT models achieved the best performance in predicting harshness scores, with an average L1-loss of 1.536 ± 0.112 and an accuracy of 71.23 ± 0.005 in classification.
  • HateBERT did not show a significant performance improvement over regular BERT, suggesting that the nature of harshness in peer reviews differs from general abusive language.
  • Annotation reliability, measured by split-half reliability, was 0.73, indicating good consistency among annotators.

Limitations and Future Work

  • Subjectivity of Harshness: The authors acknowledge that harshness is inherently subjective, and their definition and measurement are based on their interpretation and the annotators' consensus.
  • Model Performance: While BERT models performed well, the authors explicitly state there is "scope for improvement for better predictive models to detect the harshness of the review scores."
  • Future Work: The authors suggest extending the dataset and investigating the impact of reviewer confidence on peer-review text moderation. They also aim to further develop computational predictive models for harshness.

Summary updated

Abstract

The peer-review system has primarily remained the central process of all science communications. However, research has shown that the process manifests a power-imbalance scenario where the reviewer enjoys a position where their comments can be overly critical and wilfully obtuse without being held accountable. This brings into question the sanctity of the peer-review process, turning it into a fraught and traumatic experience for authors. A little more effort to still remain critical but be constructive in the feedback would help foster a progressive outcome from the peer-review process. In this paper, we argue to intervene at the step where this power imbalance actually begins in the system. To this end, we develop the first dataset of peer-review comments with their real-valued harshness scores. We build our dataset by using the popular Best-Worst-Scaling mechanism. We show the utility of our dataset for text moderation in peer reviews to make review reports less hurtful and more welcoming. We release our dataset and associated codes in https://github.com/Tirthankar-Ghosal/moderating-peer-review-harshness. Our research is one step towards helping create constructive peer-review reports.
BibTeX
@inproceedings{verma2022lack,
  author = {Rajeev Verma and Rajarshi Roychoudhury and Tirthankar Ghosal},
  title = {{The lack of theory is painful: Modeling Harshness in Peer Review Comments}},
  booktitle = {AACL/IJCNLP},
  year = {2022},
  url = {https://openreview.net/forum?id=uGooHVwns0},
}
Figure