PeerSum: A Peer Review Dataset for Abstractive Multi-document Summarization

Jianzhong QiarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces PEERSUM, a novel dataset of scientific peer reviews designed to push the boundaries of abstractive multi-document summarization (MDS). Unlike existing datasets, PEERSUM's summaries (meta-reviews) are highly abstractive and capture complex disagreements within the source documents (individual reviews), presenting a significant challenge for current MDS models.

What Problem Does This Paper Address?

Multi-document summarization (MDS) aims to condense information from multiple related documents into a coherent summary. While many applications exist, current MDS datasets often fall short. They frequently feature summaries that are more extractive than abstractive, or the source documents lack complex interrelationships like conflicts. This limits the development of models that can truly synthesize information and handle nuanced disagreements, a skill crucial for tasks like summarizing scientific literature. The authors identify this gap and propose PEERSUM to address the need for a dataset that captures abstractive summarization and complex cross-document relationships.

Methods

The core contribution of this paper is the creation and release of the PEERSUM dataset. The dataset is constructed from peer reviews of scientific publications, where each individual review serves as a source document, and the meta-review acts as the ground truth summary. The authors also explore different variants of the dataset: PEERSUM-R (reviews only), PEERSUM-RC (reviews and comments), and PEERSUM-ALL (reviews, comments, and responses). They evaluate existing state-of-the-art single-document and multi-document summarization models on PEERSUM, using standard metrics like ROUGE-L and BERTScore, and also propose an alternative evaluation strategy based on the review ratings and acceptance outcomes present in the dataset.

Datasets

The primary dataset introduced is PEERSUM, derived from peer reviews of ICLR and NeurIPS conference papers. It contains 10,862 clusters, each comprising multiple reviews (source documents) and a meta-review (summary). The authors also compare PEERSUM against several existing MDS datasets:

  • WCEP: News domain, human-written summaries.
  • Multi-News: News domain, human-written summaries.
  • Multi-XScience: Uses related work of a publication as summary and cited publications as source documents.
  • WikiSum: Uses Wikipedia articles as summaries and cited web articles as source documents.

Key Insights and Ideas

  • Meta-reviews as Abstractive Summaries: The paper highlights that meta-reviews in scientific peer review are inherently abstractive, requiring synthesis and critical evaluation of multiple individual reviews, making them ideal for challenging abstractive MDS.
  • Disagreements Drive Complexity: PEERSUM's unique feature is the presence of disagreements among reviewers. This forces summarization models to not only extract information but also to identify and potentially resolve conflicting viewpoints, a crucial aspect of real-world summarization.
  • Current Models Struggle: The experimental results demonstrate that even advanced MDS models find PEERSUM challenging, indicating that the dataset pushes the limits of current abstractive summarization capabilities.
  • Alternative Evaluation: The inclusion of review ratings and acceptance outcomes offers a novel way to evaluate summarization models, moving beyond purely lexical overlap to assess how well generated summaries capture the overall sentiment and decision-making process.

Key Results

  • PEERSUM summaries are shown to be highly abstractive, with high relevance to their source documents, outperforming many existing datasets in these aspects.
  • Current state-of-the-art MDS models, including BERT-based and transformer-based architectures like PEGASUS, struggle to achieve high performance on PEERSUM, particularly when evaluated using BERTScore. For instance, PEGASUS, while performing best among tested models, still shows room for improvement.
  • Including comments and responses (PEERSUM-RC and PEERSUM-ALL) did not consistently improve summarization performance, suggesting that meta-reviewers primarily rely on the official anonymous reviews.

Limitations and Future Work

The authors acknowledge that while PEERSUM is designed to be abstractive, there might be rare instances where private discussions between meta-reviewers and reviewers could influence the meta-review without being fully reflected in the public reviews. They suggest that future work could explore more sophisticated models that can better handle the complex cross-document relationships and disagreements present in PEERSUM. Additionally, the dataset is designed to grow as more reviews become available, offering a continuously evolving benchmark.

Summary updated

Abstract

We present PeerSum, a new MDS dataset using peer reviews of scientific publications. Our dataset differs from the existing MDS datasets in that our summaries (i.e., the meta-reviews) are highly abstractive and they are real summaries of the source documents (i.e., the reviews) and it also features disagreements among source documents. We found that current state-of-the-art MDS models struggle to generate high-quality summaries for PeerSum, offering new research opportunities.
BibTeX
@misc{qi2022peersum,
  author = {Jianzhong Qi},
  title = {{PeerSum: A Peer Review Dataset for Abstractive Multi-document Summarization}},
  year = {2022},
  eprint = {2203.01769},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.01769},
}
Figure