WikiSQE: A Large-Scale Dataset for Sentence Quality Estimation in Wikipedia

Kenichiro Ando, Satoshi Sekine, Mamoru KomachiAAAI 2024 · DOI

Introduction

Wikipedia serves as a cornerstone of the modern internet, acting as both a primary information source for the public and a critical training corpus for Large Language Models (LLMs). Despite its utility, the open-editing nature of Wikipedia leads to inconsistent quality. While automated systems effectively catch blatant vandalism, "soft" quality issues—such as subtle bias, vague language, or missing citations—remain difficult to detect. To address this, researchers have introduced WikiSQE, the first large-scale dataset designed for fine-grained sentence quality estimation.

The Challenge of Sentence Quality

Current automated tools excel at identifying formatting errors or profanity, but they often fail to recognize nuanced textual flaws. Wikipedia editors manually flag these issues using "inline templates," such as [citation needed] or [who?]. Prior to WikiSQE, no comprehensive dataset mapped these specific editor-identified problems to individual sentences at scale, hindering the development of AI assistants for quality control.

Methodology and Dataset Construction

The researchers leveraged years of Wikipedia revision history as a massive, distributed annotation effort. By extracting 7 TB of English Wikipedia data, they identified 153 distinct inline cleanup templates. These labels were organized into five primary categories: Citation, Syntactic/Semantic Revision, Information Addition, Disputed Claim, and Other.

Categorization and distribution of quality labels in WikiSQE.

The resulting WikiSQE dataset comprises approximately 3.4 million sentences, providing a robust benchmark for training models to identify specific quality deficits. The dataset and code are publicly available on GitHub.

Key Insights: Perplexity and Expert Knowledge

A significant finding of the study involves the relationship between sentence quality and "perplexity"—a measure of how "natural" a sentence appears to a language model. Sentences requiring syntactic or semantic revision exhibited the highest perplexity, suggesting they are linguistically unusual. Conversely, sentences flagged for "Information Addition" often had low perplexity, indicating that a sentence can be grammatically perfect yet still low-quality due to missing factual details.

The study also posits that Wikipedia editors function as "professional annotators." Their tags reflect a sophisticated grasp of encyclopedic standards that non-expert human annotators often struggle to replicate. In a surprising twist, fine-tuned models outperformed human crowdsourced workers on the majority of tested labels.

Experimental Results

The authors benchmarked WikiSQE using state-of-the-art transformer models, including BERT, RoBERTa, and DeBERTa. These models were tasked with predicting quality labels and were compared against human crowdsourced workers and GPT-4.

Performance comparison of transformer models across quality groups.

Key findings include:

  • Model Superiority: Fine-tuned models achieved F1 scores between 70% and 85%. Notably, the DeBERTa model outperformed non-expert human workers on 10 out of 11 tested labels.
  • Task Difficulty: "Information Addition" was the easiest category to detect (F1 ~85%), likely due to formulaic phrasing. "Citation" issues were the hardest (F1 ~73-74%), as they often require external world knowledge beyond the sentence itself.
  • Fine-tuning vs. Few-shot: While GPT-4 performed well in few-shot settings, it was generally surpassed by smaller, fine-tuned models, highlighting the importance of task-specific training data.

Conclusion and Future Directions

WikiSQE provides a powerful tool for improving the quality of Wikipedia and the datasets used to train LLMs. By filtering corpora for high-quality, unbiased text, WikiSQE can help ensure that AI models learn from reliable information. Future work will look toward expanding the dataset to other languages and improving contextual awareness to address labels that require article-level or external verification.

Summary updated

BibTeX
@inproceedings{ando2024wikisqe,
  author = {Kenichiro Ando and Satoshi Sekine and Mamoru Komachi},
  title = {{WikiSQE: A Large-Scale Dataset for Sentence Quality Estimation in Wikipedia}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29717},
}
Figure