Calibrate, Don't Retrain: Decomposing the Value of Annotator Disagreement in Cyberbullying Severity

Arslan Bisharat, Katelyn Skees, Manuel Sandoval Madrigal, Mujtaba Nazari, Ayaan Khan, Deborah L. Hall, Mohammed Abuhamad, Yasin N. SilvaarXiv 2026 · arXiv

TL;DR

In subjective tasks like cyberbullying severity, collapsing multiple annotator judgments into a single majority label discards vital information about human perception. This paper demonstrates that we can recover most of this lost "distributional signal" through simple post-hoc calibration of existing models, rather than expensive retraining, leading to more faithful and potentially fairer content moderation.

What Problem Does This Paper Address?

Machine learning for content moderation typically follows a "gold standard" paradigm: multiple humans label a comment, and their judgments are aggregated into a single majority vote. However, for subjective tasks like grading the severity of cyberbullying, disagreement isn't just noise—it's signal.

A comment that three people call "moderate" and two call "severe" is fundamentally different from one that all five agree is "moderate." Yet, standard training treats them identically. The authors find that in their primary dataset, a staggering 95.9% of comments had at least one dissenting annotator, and only 4.1% were unanimous. By forcing a consensus, we lose the "perspectivism" that reflects how different people perceive harm.

![Fig.

  1. Why a majority vote loses information. (a) A comment labeled mild by 3 annotators, moderate by 1 and severe by 1 becomes the single label mild, and the other 2 judgments are lost. (b) The share of the 9,553 comments in each agreement pattern.](/figures/art_0dbd318577e6446cbd0c87f6cc027edb)

Methods

The researchers investigated whether models can predict the distribution of human judgments (e.g., 60% mild, 40% moderate) instead of just a single label. They built 11 different systems using a DeBERTa-v3-base backbone, varying architectures (flat vs. hierarchical) and training targets.

The core of their approach is Dirichlet calibration. Instead of retraining a model from scratch to output distributions, they took models trained on majority labels and applied a small, 12-parameter map to adjust the output probabilities after training. They compared these "calibrated twins" against models fully retrained on soft label distributions.

Datasets

The study utilized three primary datasets to test generalization:

  • Insta-CTSR: An extension of the University of Colorado Instagram dataset containing ~106k comments. The study focused on 9,553 cyberbullying comments, each labeled by 5 annotators on a 4-level severity scale.
  • EPIC: A binary irony dataset that includes annotator gender, used to test group fairness.
  • ConvAbuse: A 5-level abuse-severity dataset used for replication.
Table 1. The 9,553 study-set comments in Insta-CTSR by majority severity level and 5-annotator label pattern.

Key Insights and Ideas

  • The Accuracy Ceiling: Standard metrics like Macro-F1 "saturate" quickly. Two models might have nearly identical accuracy against the majority label but differ wildly in how well they represent the underlying human disagreement. Accuracy alone is a poor proxy for a model's competence in subjective domains.
  • Calibration is a "Free Lunch": Most of the benefit of distribution-aware modeling comes from fixing overconfidence. Temperature scaling and Dirichlet calibration recovered 50-67% of the "closable gap" in fidelity to human labels without any retraining.
  • The Shared Model Limit: While predicting distributions improves the fit for all demographic groups (e.g., both men and women), it doesn't close the performance gap between them. A single shared model can represent the average disagreement but cannot know which group is "right" for a specific comment without annotator metadata.

Key Results

  • Disagreement is the Norm: Only 4.1% of cyberbullying comments received unanimous severity ratings.
  • Fidelity Gains: Using Brier score (where lower is better) to measure fidelity to human distributions, the "raw" majority-trained model scored 0.294. Simple temperature scaling dropped this to 0.182, and the full label-distribution map reached 0.144. Full retraining only added a tiny further gain to 0.137.
  • Fairness: On the EPIC irony dataset, preserving the distribution improved the Brier score for men's labels by 0.007 and women's by 0.005 compared to the calibrated majority baseline.

Limitations and Future Work

The authors acknowledge several boundaries to their evidence:

  • Era and Language: The data is from Instagram (2013-2015) and is entirely in English.
  • Annotator Identity: The primary dataset lacks annotator demographics, forcing the use of the EPIC dataset for fairness analysis.
  • Human-in-the-loop: While the model represents disagreement better, it’s still an open question whether showing these distributions to human moderators actually helps them make better or faster decisions.
  • Severe Class Performance: Interestingly, the benefits of distribution training were mostly seen in the "mild" and "moderate" classes; for the rarest "severe" cases, the majority-trained model remained competitive.

Future work suggests prompting or tuning "frontier" LLMs with label distributions to see if they can better capture the nuances of human subjectivity.

Summary updated

Abstract

Machine learning models are often trained on labeled data produced by multiple human annotators. When those annotators disagree, their labels are commonly collapsed into a single majority vote, which discards information about the underlying disagreement. We examine the cost of this practice for cyberbullying severity, a subjective judgment on which annotators disagree heavily. In the corpus we study, all 5 annotators assigned the same severity label to only 4.1% of comments. Turning 5 judgments into one label hides this disagreement and obscures how much the annotators differed in their assessments. In this paper, we ask whether a model can preserve this information by predicting the distribution of annotator judgments instead of a single majority label. We find 3 main results. First, accuracy against the majority label cannot tell competent models apart: 2 models with similar accuracy can differ greatly in how well they match the distribution of annotator judgments. Second, predicting the full distribution requires little extra effort. Calibrating an existing model, which adjusts its output probabilities after training, captures most of the improvement without retraining, across 3 datasets covering severity, abuse, and irony, while training on label distributions adds a small further gain. Third, we examine a dataset that includes annotator demographics, where a single shared model has one advantage and one limit. The advantage is that a model predicting the distribution of judgments matches each demographic group's judgments significantly better than a model trained on majority labels, so both groups gain. The limit is that it does not narrow the gap in performance between the groups, because one shared model cannot encode which group is right on a given comment. Preserving annotator disagreement is therefore a low-cost step toward fairer moderation, though not a complete one.
BibTeX
@misc{bisharat2026calibrate,
  author = {Arslan Bisharat and Katelyn Skees and Manuel Sandoval Madrigal and Mujtaba Nazari and Ayaan Khan and Deborah L. Hall and Mohammed Abuhamad and Yasin N. Silva},
  title = {{Calibrate, Don't Retrain: Decomposing the Value of Annotator Disagreement in Cyberbullying Severity}},
  year = {2026},
  eprint = {2611.15010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15010},
}
Figure