A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement

Guillermo Villate-Castillo, Javier Del Ser, Borja Sanz

TL;DR

This paper introduces a novel framework for collaborative content moderation that tackles the inherent subjectivity of toxicity detection by jointly modeling toxicity classification and annotation disagreement. By leveraging multitask learning and conformal prediction, the framework provides more reliable uncertainty estimates, leading to improved model performance, calibration, and a more efficient review process for human moderators.

What Problem Does This Paper Address?

Content moderation, especially for toxic online content, is a complex task due to the subjective nature of language and the varying interpretations of what constitutes toxicity. Traditional approaches often rely on majority voting from human annotators, which can overlook valuable signals of disagreement that highlight ambiguity. Machine learning models trained on such data may struggle with nuanced cases and provide overconfident predictions. This paper addresses the challenge of effectively incorporating human annotation disagreement and model uncertainty into a collaborative content moderation system to improve both accuracy and efficiency.

Methods

The core of the proposed framework is a multitask learning (MTL) architecture. The primary task is toxicity classification, while an auxiliary task is introduced to predict annotation disagreement. This MTL approach uses a DistillBERTbase uncased model as the backbone to extract text embeddings.

To quantify uncertainty and provide reliable predictions, the authors employ Conformal Prediction (CP) techniques. CP is a model-agnostic method that offers guaranteed coverage for predictions, meaning the prediction intervals or sets will contain the true outcome with a specified probability. The framework uses CP for both the classification (toxicity) and regression (annotation disagreement) tasks.

The framework also incorporates a review decision node that flags comments for human review based on two criteria:

  1. If the predicted classification set size is greater than 1 (indicating uncertainty in toxicity prediction).
  2. If the predicted confidence interval for annotation disagreement exceeds a predefined ambiguity threshold ($\gamma$). This threshold allows moderators to control when ambiguity should trigger a review.

Datasets

The primary dataset used in this study is the Jigsaw Unintended Bias in Toxicity Classification dataset. This dataset was chosen for its high-quality annotations and a large number of annotators per comment, which is crucial for capturing annotation disagreement. The authors performed preprocessing, including selecting comments with at least 10 annotators and cleaning the text by removing HTML tags, emojis, and other non-textual elements.

Key Insights and Ideas

  • Annotation disagreement as a signal, not noise: The paper argues that disagreement among annotators is a valuable indicator of content ambiguity and subjectivity, rather than just noise to be discarded.
  • Multitask learning for improved performance: Jointly training for toxicity classification and annotation disagreement prediction leads to better performance and calibration for the primary toxicity detection task.
  • Conformal Prediction for reliable uncertainty: CP provides mathematically guaranteed coverage for predictions, making the uncertainty estimates more trustworthy and useful for flagging ambiguous cases.
  • Adaptive review thresholds: The ability to set an ambiguity threshold ($\gamma$) for annotation disagreement allows moderators to customize the review process based on their tolerance for ambiguity.
  • Enhanced collaborative moderation: The framework aims to create a more efficient collaborative system where models can effectively identify and flag uncertain or ambiguous content, allowing human moderators to focus their expertise on the most challenging cases.

Key Results

The paper presents several key findings across different research questions:

  • RQ1 (Impact on Primary Task Performance): The MTL approach generally improves toxicity classification performance and calibration compared to single-task (STL) models, especially for MSE and BCE regression losses. While RAC showed some benefits, its calibration was poorer.
  • RQ2 (Effects on Uncertainty Quantification): The MTL approach consistently enhances uncertainty estimation for classification tasks, showing better alignment between model uncertainty and annotation disagreement. For regression tasks, improvements were observed in interval size and distance, with the effectiveness depending on the chosen regression loss and uncertainty method.
  • RQ3 (Benefits of Auxiliary Task vs. Independent Models): The MTL framework generally outperforms composite models (CoM) in collaborative content moderation, demonstrating improvements in Review Efficiency (F1 Review Efficiency, CARE) and correlation with annotation disagreement. The MTL approach is also more computationally efficient.

Specific quantitative results include:

  • Classification Performance (Table 2): MTL models achieved higher F1 scores (e.g., MTL+MSE: 72.48) compared to STL CLASS (71.61).
  • Regression Performance (Table 3): MTL models showed consistent improvements in MAE and MSE over STL REG.
  • Uncertainty Metrics (Table 4, 5, 6, 7): MTL models generally showed improved MURE, Marginal Coverage, and $r_{pb}$ scores compared to STL models across various CP methods.

Limitations and Future Work

Limitations: * Dataset Imbalance: The original dataset had a significant class imbalance for toxicity, which was reduced but still present. The framework's performance in highly imbalanced scenarios was not fully explored. * Annotation Quality and Quantity: The framework's effectiveness relies on datasets with sufficient annotator diversity and disagreement. Datasets with minimal disagreement might not fully showcase the framework's potential. * Subjectivity of Toxicity: The inherent subjectivity of toxicity remains a fundamental challenge, complicating dataset creation and annotation. * Contextual Information: While the paper focuses on text, the full context of comments (e.g., prior conversation) is not explicitly modeled.

Future Work: * Exploring other Uncertainty Estimation Techniques: Investigating methods like Evidential Learning, Epistemic Neural Networks, and Bayesian Deep Learning to assess if similar gains extend beyond CP. * Applicability to Other Subjective Tasks: Evaluating the framework's performance on tasks like sentiment analysis, where annotation disagreement is also critical. * Addressing Data Imbalance: Incorporating techniques like oversampling, undersampling, and cost-sensitive loss functions for scenarios with higher degrees of class imbalance. * Real-World Deployment and Collaboration: Testing the framework in real-world moderation workflows and collaborating with industry leaders to refine and adapt it. * Active Learning and Generative Modeling: Exploring the potential of the framework in active learning and generative text modeling, including addressing issues with synthetic data generation and model degradation.

Summary updated

Abstract

Content moderation typically combines the efforts of human moderators and machine learning models. However, these systems often rely on data where significant disagreement occurs during moderation, reflecting the subjective nature of toxicity perception. Rather than dismissing this disagreement as noise, we interpret it as a valuable signal that highlights the inherent ambiguity of the content,an insight missed when only the majority label is considered. In this work, we introduce a novel content moderation framework that emphasizes the importance of capturing annotation disagreement. Our approach uses multitask learning, where toxicity classification serves as the primary task and annotation disagreement is addressed as an auxiliary task. Additionally, we leverage uncertainty estimation techniques, specifically Conformal Prediction, to account for both the ambiguity in comment annotations and the model's inherent uncertainty in predicting toxicity and disagreement.The framework also allows moderators to adjust thresholds for annotation disagreement, offering flexibility in determining when ambiguity should trigger a review. We demonstrate that our joint approach enhances model performance, calibration, and uncertainty estimation, while offering greater parameter efficiency and improving the review process in comparison to single-task methods.
BibTeX
@misc{villatecastillo2024collaborative,
  author = {Guillermo Villate-Castillo and Javier Del Ser and Borja Sanz},
  title = {{A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement}},
  year = {2024},
  eprint = {2411.04090},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.04090},
}
Figure