WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation

Nachshon Cohen, Oren Kalinsky, Yftah Ziser, Alessandro MoschittiACL 2021 · DOI

TL;DR

This paper introduces WikiSum, a new dataset for evaluating text summarization systems. Unlike existing datasets that use academic texts, WikiSum leverages "how-to" articles from WikiHow, making human evaluation significantly easier, faster, and more cost-effective due to simpler language and less reliance on specialized knowledge.

What Problem Does This Paper Address?

Evaluating the quality of automatic text summarization systems is crucial for their development. While metrics like ROUGE exist, they often don't align well with human judgment. Therefore, human evaluation is essential. However, existing datasets for coherent paragraph summarization (where sentences flow together) primarily consist of academic papers. These papers are written for experts, making them difficult for non-specialists to read and understand, thus rendering human evaluation time-consuming, expensive, and requiring annotators with specific domain knowledge. This creates a bottleneck in developing summarization systems that produce truly useful and understandable summaries for a general audience.

Methods

The core contribution of this paper is the creation and validation of the WikiSum dataset. The authors gathered data from WikiHow, a website featuring "how-to" articles. They specifically extracted articles and their corresponding "quick summaries," which are written in plain language and designed to be easily understood by a broad audience.

To validate the dataset's effectiveness for human evaluation, they conducted several analyses:

  • Readability Metrics: They used standard readability scores (FKGL, GFI, SMOG, ARI, CLI) to quantitatively assess how easy the WikiSum documents and summaries are to read compared to academic datasets like arXiv, PubMed, and BigPatent.
  • External Knowledge Assessment: They measured the reliance on specialized knowledge by analyzing the frequency of uncommon words, hypothesizing that words not found in common sources like movie/TV subtitles would require more external knowledge.
  • Human Evaluation Study: They conducted a direct comparison of human evaluation on WikiSum versus PubMed. Annotators were tasked with evaluating summaries, and their feedback was collected on evaluation time, difficulty, perceived qualification, and preference for the dataset.
  • Benchmarking: They evaluated standard summarization models (LEAD-3, TextRank, PEGASUS-LARGE) on WikiSum to establish baseline performance and compare it to existing datasets.

Datasets

  • WikiSum: The newly introduced dataset, comprising 39,775 "how-to" articles and their corresponding "quick summaries" from WikiHow.
  • Existing Datasets for Comparison:
    • arXiv, PubMed, BigPatent: Academic summarization datasets used to highlight the difficulty of human evaluation with expert-level texts.
  • WikiHow (Koupaee and Wang, 2018; Ladhak et al., 2020): Previous datasets derived from WikiHow, but differing in how documents and summaries were constructed (e.g., using step headers vs. full articles).
  • CNN/DM, NYT, Newsroom, XSUM: News domain datasets, often used for summarization research, but typically featuring highlights or single-sentence summaries, not coherent paragraphs.

Key Insights and Ideas

  • Plain Language is Key: The most significant insight is that using simple, everyday language (like in WikiHow articles) dramatically reduces the burden on human annotators, making evaluation faster and more reliable.
  • Domain Knowledge is a Bottleneck: Academic datasets require specialized knowledge, which limits the pool of annotators and increases evaluation costs. WikiSum's broad domain coverage and simple language circumvent this.
  • Readability Directly Impacts Evaluation Efficiency: The study strongly correlates readability scores with human evaluation metrics like time, difficulty, and perceived qualification, demonstrating that simpler texts lead to more efficient and effective evaluations.
  • Unfamiliar Words Signal Difficulty: The presence of uncommon words directly correlates with increased task difficulty and the need for external knowledge, reinforcing the value of WikiSum's accessible vocabulary.
  • Human Preference Matters: Annotators overwhelmingly preferred evaluating summaries from WikiSum over PubMed, providing strong qualitative evidence for its usability.

Key Results

  • Readability: WikiSum documents and summaries scored significantly lower (indicating simpler text) on readability metrics compared to academic datasets like arXiv, PubMed, and BigPatent. For example, the ARI score for WikiSum documents was 7.4, while for PubMed it was 16.74.
  • External Knowledge: WikiSum documents contained a lower ratio of uncommon words, suggesting less reliance on specialized knowledge for comprehension.
  • Human Evaluation:
    • WikiSum annotations took significantly less time (6.8±1.2 minutes) compared to PubMed (10.0±1.2 minutes).
  • Annotators rated WikiSum tasks as less difficult (1.9±0.3) and less tiring (2.2±0.5) than PubMed tasks (3.7±0.3 difficulty, 3.9±0.4 tiring).
  • Annotators felt more qualified to assess WikiSum summaries.
  • In 90% of cases, annotators preferred evaluating WikiSum over PubMed.
  • Model Performance: Models like PEGASUS-LARGE achieved comparable ROUGE scores on WikiSum as they did on academic datasets, indicating that WikiSum is a suitable benchmark for evaluating summarization systems without sacrificing performance comparability.

Limitations and Future Work

  • Dataset Size: While 39,775 samples is substantial, the authors acknowledge that for some downstream tasks, even larger datasets might be beneficial.
  • Comparison Datasets: For the human evaluation, only PubMed was chosen as a comparison for coherent-paragraph summaries due to resource limitations and the need to avoid annotator domain bias. Future work could explore comparisons with other academic datasets.
  • Automatic Metrics: The paper reiterates the challenge of automatic metrics like ROUGE and suggests that further research into metrics that better align with human judgment, especially for simple language datasets, would be valuable.
  • Broader Applications: The authors suggest that the WikiSum dataset could be used for various downstream tasks beyond summarization evaluation, such as studying the impact of language simplicity on model performance or developing better readability assessment tools.

Summary updated

BibTeX
@inproceedings{cohen2021wikisum,
  author = {Nachshon Cohen and Oren Kalinsky and Yftah Ziser and Alessandro Moschitti},
  title = {{WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.28},
}
Figure