An Analysis of Tasks and Datasets in Peer Reviewing

Moritz Staudinger, Wojciech Kusa, Florina Piroi, Allan HanburyACL 2024

TL;DR

This paper provides a comprehensive inventory of tasks and datasets related to automating and optimizing academic peer review. It highlights significant challenges with existing datasets, such as poor documentation and lack of standardization, and proposes guidelines for future dataset creation and publication to improve reproducibility and reusability.

What Problem Does This Paper Address?

The academic peer review system, a cornerstone of scientific integrity, is facing increasing strain. A surge in research publications, partly fueled by technological advancements and the rise of Large Language Models (LLMs), has led to an overwhelming number of submissions. This influx, coupled with the voluntary and time-consuming nature of reviewing, results in a growing burden on researchers, potentially leading to lower quality reviews from inexperienced reviewers. To address this, researchers are exploring ways to automate and optimize various stages of the peer review process. However, the field is hampered by a fragmented landscape of datasets, making it difficult to compare results, reproduce studies, and build upon existing work. This paper aims to bring clarity to this landscape by cataloging existing research tasks and datasets, and by offering recommendations for better data practices.

Methods

The authors conducted a systematic review of existing literature and datasets related to AI-assisted peer review. They analyzed 53 datasets, categorizing them based on the research tasks they support. For each dataset, they examined its origin, creation date, and intended use. The paper then synthesizes this information to identify common research tasks, highlight overlaps and discrepancies between datasets, and propose guidelines for future dataset development and publication.

Datasets

The paper categorizes datasets based on the peer review tasks they support. Key task categories and representative datasets include:

  • Review Analysis: Datasets used to understand the impact of reviews on submissions, detect biases, and analyze review quality. Examples include datasets from ICLR, BMC, BMJ, and PLOS Medicine.
  • Reviewer Assignment: Datasets for matching submissions to reviewers based on expertise and interests, and for detecting malicious bidding. Examples include datasets from OpenReview and ACL Rolling Review.
  • Score and Acceptance Prediction: Datasets used to predict review scores and paper acceptance decisions. Examples include datasets from ICLR and various conferences.
  • Revision, Dialogue, Rebuttal Analysis: Datasets focusing on analyzing changes between submission versions, dialogues between authors and reviewers, and rebuttal strategies. Examples include CASIMIR, ARIES, and arXivEdits.
  • Review Generation: Datasets for training models to automatically generate peer reviews. Examples include ReviewRobot, ASAP-Review, and ORB.
  • Metareview Generation: Datasets for generating summaries of reviews and assisting Area Chairs. Examples include MetaWriter, PeerSum, and ORSUM.
  • Quality Analysis: Datasets for assessing review helpfulness, politeness, harshness, and reviewer confidence. Examples include PolitePEER and Verma et al. (2022).
  • Citation Prediction: Datasets linking reviews to publication citations to predict impact. Examples include CiteTracked and Citation-Count-Pred.

The authors note that many datasets are derived from OpenReview, but often with inconsistencies in reporting and filtering.

Key Insights and Ideas

  • Dataset Fragmentation is a Major Bottleneck: The field is characterized by a proliferation of datasets, many created from scratch using APIs like OpenReview. This leads to issues with reproducibility, comparability, and reusability due to inconsistent documentation, filtering, and licensing.
  • Lack of Standardization Hinders Progress: Different conferences use varying review guidelines and scoring scales, making it challenging to combine data from multiple sources.
  • The Need for FAIR Data Practices: The paper strongly advocates for adhering to FAIR principles (Findable, Accessible, Interoperable, Re-usable) for dataset publication, emphasizing the importance of clear documentation, machine-readable metadata, and accessible data.
  • LLMs are Powerful but Not a Panacea: While LLMs are increasingly used for tasks like review generation, the paper suggests that domain knowledge and information extraction techniques are crucial for truly effective and nuanced peer review automation.
  • Interdisciplinary Collaboration is Key: The authors highlight that while NLP researchers are active in this domain, greater involvement from the Semantic Web and Information Retrieval communities could bring valuable expertise.

Key Results

The paper does not present novel quantitative results from experiments but rather synthesizes findings from existing research. Its primary contribution is the comprehensive cataloging and analysis of 53 datasets and their associated tasks. The key "result" is the identification of systemic issues within the current dataset landscape and the formulation of actionable recommendations for improvement.

Limitations and Future Work

The authors acknowledge that their work, while extensive, may not be exhaustive. They identify several limitations with existing datasets:

  • Poor Documentation: Many datasets lack detailed information about their creation process, annotation guidelines, and filtering criteria.
  • Overlapping Data and Inconsistencies: Datasets derived from the same source (e.g., OpenReview) often exhibit discrepancies in reported numbers of submissions or reviews.
  • Unclear Licensing: Many datasets are published without clear licenses, raising ethical concerns about reviewer consent and data usage.
  • Lack of Canonical Splits: Most datasets do not provide predefined splits for training, validation, and testing, hindering model comparability.

Future work suggested includes:

  • Creation of a Taxonomy: Developing a standardized taxonomy of tasks in AI-assisted or automated peer reviewing.
  • Well-Documented, Multi-Domain Datasets: The need for a comprehensive dataset that covers peer reviews from various domains and can be used for multiple subtasks.
  • Encouraging Interdisciplinary Involvement: Fostering collaboration with communities like Semantic Web and Information Retrieval.
  • Future Shared-Tasks: Organizing shared-tasks that leverage existing expertise and encourage the creation of new, well-documented datasets.

Summary updated

BibTeX
@inproceedings{staudinger2024analysis,
  author = {Moritz Staudinger and Wojciech Kusa and Florina Piroi and Allan Hanbury},
  title = {{An Analysis of Tasks and Datasets in Peer Reviewing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure