University at Buffalo at SemEval-2023 Task 11: MASDA–Modelling Annotator Sensibilities through DisAggregation

Michael Sullivan, Mohammed Yasin, Cassandra L. JacobsACL 2023

TL;DR

This paper introduces MASDA, a novel framework for modeling annotator disagreement in text classification tasks. By explicitly representing individual annotator "sensibilities" through embeddings and attention mechanisms, MASDA achieves improved performance over traditional aggregation methods, offering a more nuanced understanding of why annotators might disagree.

What Problem Does This Paper Address?

In many natural language processing tasks, especially those involving subjective or controversial topics, annotator disagreement is a significant challenge. Simply aggregating individual labels (e.g., majority voting) can mask important variations in annotator interpretation, leading to suboptimal model performance. This paper tackles the problem of how to effectively model and leverage these individual differences in annotator judgment, rather than just averaging them out. The goal is to build models that understand why annotators might disagree, leading to more robust and accurate predictions.

Methods

The core of MASDA is its "Annotator Layers" approach. This involves:

  • Annotator Embeddings: Each annotator is represented by a low-dimensional embedding vector. These embeddings capture an annotator's unique "sensibility" or tendency to label text in a certain way.
  • Joint Prediction: The model simultaneously predicts the label for each individual annotator and the final "gold" label. This allows the model to learn from both individual annotator decisions and the consensus.
  • GoldNet Attention: A key innovation is the "GoldNet" module, which uses an attention mechanism. This mechanism learns to weigh the contributions of individual annotator embeddings based on how well they align with the gold label for a given text. This effectively allows the model to learn which annotators are more reliable for specific types of content.
  • Parameter Sharing: To improve efficiency and generalization, certain layers (like the decision layer, DecNet) are shared across all annotator embeddings.

The paper also explores two other approaches:

  • Topic Infusion: This method incorporates the topic of a text as an additional feature, aiming to capture nuances that might influence annotator judgment.
  • Annotator Interaction Modeling: This approach models how annotators might interact or "consult" with each other before making a decision, particularly relevant for conversational data.

Datasets

The models were evaluated on the following datasets:

  • HS-Brexit: A dataset related to hate speech detection.
  • ArMIS: A corpus for Arabic misogyny and sexism detection.
  • MD-Agreement: A dataset for detecting abusive language in conversations.
  • ConvAbuse: Another dataset for detecting abusive language in conversations.

The paper does not introduce new datasets but rather applies its methods to existing ones for SemEval-2023 Task 11 (LeWiDi23).

Key Insights and Ideas

  • Annotator Sensibilities Matter: Explicitly modeling individual annotator characteristics, rather than just their aggregated output, leads to significant performance improvements.
  • Attention is Key to Disagreement: The GoldNet attention mechanism is crucial for learning which annotators are more trustworthy for specific inputs, effectively learning from disagreement.
  • Gradient Flow is Tricky: Isolating the GoldNet during training (preventing gradients from flowing back to annotator embeddings) is important to avoid conflicting learning signals.
  • Topic and Interaction Add Value: Incorporating topic information and modeling annotator interaction can further boost performance, especially for nuanced or conversational data.
  • Dimensionality is Non-Trivial: Finding the optimal dimensionality for annotator embeddings is not straightforward and requires careful tuning.

Key Results

  • The proposed MASDA framework (Approach 1) consistently outperformed baseline models and task organizers' baselines across both HS-Brexit and ArMIS datasets, showing reductions in cross-entropy and improvements in F1-score. For example, on ArMIS, the full model achieved a cross-entropy of 0.548, outperforming the baseline of 0.570.
  • Ablation studies showed that removing the GoldNet or allowing gradients to flow from GoldNet to annotator networks resulted in decreased performance, highlighting the importance of the proposed architecture.
  • Topic Infusion (Approach 2) led to significant gains, with a notable increase of 0.2 in F1-score and 0.14 in cross-entropy on the MD-Agreement dataset.
  • Annotator Interaction Modeling (Approach 3) also demonstrated strong performance, with the "with cross-consultation" model achieving a higher F1-score (0.92) compared to the "without cross-consultation" model (0.63) on the ConvAbuse dataset.

Limitations and Future Work

  • Fixed Annotator Pools: The current architecture relies on a fixed number of annotators for each dataset, which limits its applicability to datasets with a large or variable number of annotators.
  • Optimal Dimensionality: Determining the optimal dimensionality for annotator embeddings is a non-trivial problem that requires further investigation.
  • Generalizability: While the methods show promise, finding an approach that is universally applicable across diverse datasets and annotation scenarios remains a key area for future work.
  • Ethical Considerations: The authors acknowledge that pre-trained models can encode social biases, and their method, by identifying annotator opinions, could potentially be used to exclude certain viewpoints, raising ethical concerns about fairness and impact. Future work could explore mitigating these biases.

Summary updated

BibTeX
@inproceedings{sullivan2023university,
  author = {Michael Sullivan and Mohammed Yasin and Cassandra L. Jacobs},
  title = {{University at Buffalo at SemEval-2023 Task 11: MASDA–Modelling Annotator Sensibilities through DisAggregation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure