Wikipedia is Not a Dictionary, Delete! Text Classification as a Proxy for Analysing Wiki Deletion Discussions

Hsuvas Borkakoty, Luis Espinosa-AnkeNAACL 2025

TL;DR

This paper tackles the challenge of automatically moderating content on collaborative platforms like Wikipedia by analyzing deletion discussions. They introduce a new dataset and evaluate various language models, finding that predicting deletion outcomes is feasible and that certain models show promise for understanding user stances and policy references within these discussions.

What Problem Does This Paper Address?

Collaborative knowledge hubs like Wikipedia are invaluable resources, but maintaining their quality is a significant challenge due to their sheer size and the constant influx of new content. A crucial aspect of this quality control is the process of "deletion discussions," where community members debate whether an article should be kept, merged, or deleted. Manually moderating these discussions is time-consuming and inefficient. This paper aims to automate parts of this process by using Natural Language Processing (NLP) techniques to understand and predict the outcomes of these deletion discussions, as well as to identify user stances and the underlying policies being referenced. While previous work has touched upon this, there's a notable lack of research on non-English languages and in-depth comparative analyses of different NLP models.

Methods

The core of this research involves applying and evaluating various language models (LMs) for text classification tasks related to deletion discussions. They specifically focus on three tasks:

  • Outcome Prediction: Predicting the final decision (e.g., delete, keep, redirect) of a deletion discussion based on the entire conversation.
  • Stance Detection: Identifying an individual user's stance (e.g., keep, delete, merge, comment) within a single comment.
  • Policy Prediction: Determining which Wikipedia policy (e.g., Notability, Verifiability) a user's comment is implicitly or explicitly referencing.

They experiment with several pre-trained language models, including BERT, RoBERTa, DistilBERT, and Twitter-RoBERTa, in both "fulltext" (seeing all content, including self-assigned tags) and "masked" (hiding self-assigned tags) settings. They also explore the effectiveness of SetFit, a more parameter-efficient fine-tuning approach.

Datasets

The authors construct a novel dataset of deletion discussions from several Wikis across three languages: English (en), Spanish (es), and Greek (gr). This dataset covers multiple platforms, including Wikipedia, Wikidata (entities and properties), Wikinews, and Wikiquote. For stance and policy prediction, they leverage the existing WIKI-STANCE dataset, with some modifications to the policy labels for a more focused analysis.

Key Insights and Ideas

  • Deletion discussions are predictable: The paper demonstrates that it's possible to predict the outcome of deletion discussions with reasonable accuracy, suggesting that the arguments presented often lead to a discernible consensus.
  • Self-produced tags aren't always helpful: Surprisingly, hiding user-generated tags like "keep" or "delete" (in the "masked" setting) sometimes leads to comparable or even better performance for certain models, implying that the explicit tags might not always be the strongest signal compared to the nuanced arguments within the comments themselves.
  • Model performance varies by platform: Smaller, less active platforms like Wikinews and Wikiquote present unique challenges, with models trained on larger datasets sometimes outperforming in-domain models. This highlights the importance of dataset size and diversity.
  • SetFit shows promise for efficiency: The SetFit model, a more parameter-efficient approach, demonstrates competitive performance, especially on smaller datasets, suggesting it could be a practical choice for production environments where computational resources are a concern.
  • Cross-lingual similarities exist, but nuances remain: While models can generalize across platforms and languages to some extent, specific platform characteristics (like Wikiquote's distinct discussion style) and language-specific nuances (like Spanish editors being more explicit) influence performance.

Key Results

  • Outcome Prediction: The best models achieved F1 scores ranging from around 0.4 to 0.76 across different platforms and languages in the fulltext setting. RoBERTa-Large generally performed best.
  • Stance Detection: Weighted F1 scores for stance detection were high, with models achieving around 0.83, comparable to previous work.
  • Policy Prediction: The best model achieved a weighted F1 score of 0.67 for policy prediction, outperforming previous results on a trimmed label set.
  • Cross-platform generalization: Models trained on larger datasets (like Wikipedia) often performed well on smaller datasets (like Wikinews), showing some degree of generalization.
  • Multilingual performance: For Spanish and Greek, models like XLM-R-Large and MBERT showed strong results, with Spanish exhibiting a smaller performance drop in the masked setting, indicating more explicit language use.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited scope of exploration: Their analysis primarily focuses on Wikipedia and doesn't extensively cover all deletion discussions across all years. They also only explore BERT-family models and don't train custom models for sentiment or offensive language detection.
  • Need for more analytical tasks: The proposed tool could be enhanced with more analytical capabilities and model-based activities beyond classification.

Future work could involve:

  • Expanding the analysis to include more LMs and a wider range of analytical tasks.
  • Further investigating the role of tokenization and its impact on model performance.
  • Exploring more robust methods for handling smaller datasets and cross-platform generalization.
  • Investigating the nuances of policy references and potential merging or splitting of policy categories.

Summary updated

BibTeX
@inproceedings{borkakoty2025wikipedia,
  author = {Hsuvas Borkakoty and Luis Espinosa-Anke},
  title = {{Wikipedia is Not a Dictionary, Delete! Text Classification as a Proxy for Analysing Wiki Deletion Discussions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure