Separating Evidence from Severity in Crowd-Labeled Hate Speech

When three crowdworkers label the same social-media post as "offensive," "hate," and "hate," a model trained on their votes cannot tell why they disagreed. One possibility is that someone misread the post. The other is that all three saw the same words and simply judged them with different harshness. The HateXplain data suggests the second case is common. Annotators who disagree on the hate/offensive label often highlight exactly the same tokens as their justification, and that boundary accounts for 42.6% of all disagreement (Majority Vote Silences Minority Values). Every current way of training on these labels mixes the two sources of disagreement. Majority vote and soft labels average them away. Per-annotator output heads try to memorize them. Rationale supervision averages the highlights into one attention target.

We propose a model that splits the decision in two. A shared "evidence" part, trained on crowd highlights through a per-annotator noise model, decides which tokens matter and how severe the post is on a single scale. A tiny per-annotator "threshold" part, with only two numbers per worker, decides where that worker draws the lines between normal, offensive, and hate. The split is itself an interpretability constraint, since the label must flow through a highlighted-evidence severity score. We pair the model with a measurement idea borrowed from information theory, the genie-aided bound. In that technique, giving a decoder carefully chosen side information yields a ceiling on what any decoder could achieve, and choosing which side information to give isolates one source of difficulty from another. Here we train "genie" models that receive, at test time, either other annotators' highlights for the post (where to look) or the target annotator's judgments on other posts (how harshly they judge). Comparing them tells us how much of the roughly 22–28 point accuracy collapse on contested posts can be closed by better evidence-finding and how much is a threshold problem. It also tells us how close the proposed model gets to each ceiling.

Core Claim

On crowd-labeled, rationale-annotated data with structured disagreement, an ordinal model is a model that treats normal < offensive < hate as ordered severity levels. Our version gives that model a shared, rationale-weighted severity score and per-annotator cutpoints. We claim it will predict individual annotators' labels on contested posts better than soft-label training or per-annotator heads, without losing aggregate accuracy, bias, or plausibility relative to BERT-HateXplain. We further claim that a pair of side-information ceilings will show the remaining contested-post error is mostly threshold disagreement rather than evidence error. If that is right, it predicts that better rationale supervision alone, the main lever in prior work, cannot fix the collapse.

Relation to Prior Work

Two papers motivate the proposal by documenting the gap. HateXplain collected 20,148 posts with three workers each, giving labels, target communities, and token highlights. It supervised BERT attention with the averaged, softmax-normalized highlights, improving plausibility and the demographic-bias metric GMB-Subgroup from 0.762 to 0.807. It never modeled any individual annotator. Majority Vote Silences Minority Values compared hard-label, soft-label, and per-annotator-head BERT models. It found the identical-highlight, conflicting-label pattern and the per-annotator head's accuracy collapse to 0.245 on offensive disagreement cases. That paper dropped rationale supervision entirely and offered no way to separate the sources of error. Our proposal sits exactly in the gap between the two papers: it uses the rationales that the second paper discarded to explain the disagreement the first paper averaged over.

Three lines of work supply methods we reuse. The Crowd Layer learns a full confusion matrix per worker on top of a shared network. It reached 83.15% on LabelMe against 76.74% for majority vote. With only three labels per HateXplain post, though, full matrices are badly overparameterized. We replace them with two ordered cutpoints per worker and extend the noise-modeling idea from labels to highlights. GLAD (Whose Vote Should Count More) models correctness through worker ability and item difficulty. Our highlight noise model is a token-level analogue of this. To Aggregate or Not? shows that keeping labels separate beats aggregation when there are few annotators per item, as in CIFAR-10N. We take that as justification for training on individual (annotator, post) pairs.

On the interpretability side, rationale co-training is the lever for accuracy. Do Human Rationales Improve Machine Explanations? reports that rationale supervision lifts movie-review accuracy from 86.00% to 90.00% and roughly doubles human preference for the explanations. e-SNLI shows that explanation targets are harder to fake through shortcuts than labels. Both treat a single rationale as gold. Two papers constrain the design. ERASER warns that plausible highlights need not be faithful, so we must measure faithfulness directly. ERASER also shows that hard extract-then-predict pipelines lose accuracy, so our bottleneck is soft. Saliency Guided Training enforces prediction invariance under masking of non-salient tokens. This pushes toward confident predictions, which is the wrong behavior on posts where disagreement is genuine. We include it as a competitor partly to test that tension.

From information theory we borrow only the logic of bounding. Etkin, Tse and Wang give receivers side information that untangles interference without revealing the message. Soundararajan and Vishwanath give the decoder the sum of two correlated sources so that they become independent. Both obtain bounds that a simple scheme then approaches within a constant gap. Both papers also illustrate the "over-generous genie" failure: if you hand over the target, the bound becomes trivial. We do not prove an information-theoretic theorem. With finite data, extra inputs can hurt a learned model, so our ceilings are estimated ceilings for a fixed model class, and we check that they actually dominate the non-genie models.

Model

Let $h_t(x)$ be the BERT representation of token $t$ in post $x$. Let $a_t(x)\in[0,1]$ be an evidence weight from a token-scoring head. Define a scalar severity $s(x)=w^\top \sum_t a_t(x)\,h_t(x) / \sum_t a_t(x)$. Annotator $j$ is modeled ordinally as

$$P(y^{(j)} \le k \mid x) = \sigma\big(c_k + \delta_{jk} - s(x)\big), \quad k \in \{\text{normal}, \text{offensive}\},$$

where $\sigma$ is the logistic function, $c_k$ are shared ordered cutpoints, and $\delta_{jk}$ are annotator offsets shrunk toward zero by an $L_2$ penalty. The penalty matters most for workers with few labels.

Highlights are modeled as noisy readings of the evidence. Annotator $j$ marks token $t$ with probability $\pi_j a_t(x) + \epsilon_j (1-a_t(x))$, where $\pi_j$ and $\epsilon_j$ are that worker's hit and false-alarm rates. Because the HateXplain recipe treats posts labeled normal as having no rationale, this likelihood is applied only to an annotator's highlights on posts they labeled non-normal.

Training maximizes the joint likelihood of every individual label and every individual highlight. At inference, the aggregate prediction uses the shared cutpoints, and the per-annotator prediction adds $\delta_j$. The only path from text to label runs through the evidence-weighted severity score. That is the structural constraint we expect to help accuracy, by preventing the model from fitting each worker's idiosyncrasies with free capacity.

Research Program

The work has three parts. First, we build the model above and its ablations on BERT-base-uncased with the HateXplain splits. We train on separate (annotator, post) pairs and run five seeds.

Second, we build the genies. Each is the same architecture with one extra input, retrained from scratch on the same training data. The evidence genie receives, as an extra token-level input, the highlights of the other annotators on the post, never the target annotator's own. This leave-one-out choice avoids the over-generous genie, because an annotator's own empty highlight would reveal a normal label. The threshold genie receives the target annotator's identity together with offsets $\delta_j$ fit on all of that annotator's other labeled posts, including test posts other than the current one. This is a transductive estimate of how harshly the person judges, without revealing their current label. The joint genie receives both.

Third, we decompose the contested-post gap. On the subset of test posts where annotators disagree, we compare per-annotator accuracy and log-likelihood across the non-genie model, each single genie, and the joint genie. We further split that subset by highlight agreement, separating posts where disagreeing annotators' highlights have Jaccard similarity near 1.0 from posts where they diverge.

Baselines

The do-nothing reference is BERT trained with cross-entropy on majority-vote labels, which is Model A of Majority Vote Silences Minority Values. That paper reports roughly 80% accuracy on agreed posts and roughly 58% on disagreement posts, with confidence 0.710 on its boundary errors. We retrain it ourselves under our tuning budget rather than copy numbers.

The strongest existing method on aggregate metrics is BERT-HateXplain (HateXplain), which adds attention supervision with averaged highlights. It reports Macro F1 0.687, AUROC 0.851, and GMB-Subgroup 0.807. It is the strongest competitor because it already uses the same rationales to gain both accuracy and fairness. Beating it means matching its Macro F1 and GMB-Subgroup within seed variance while clearly exceeding it on per-annotator log-likelihood and contested-post accuracy, because BERT-HateXplain cannot represent individual annotators at all.

The strongest competitors on individual annotators are the soft-label model (Model B) and the per-annotator-head model (Model C) from the same paper. Model C has reported accuracy 0.245 on offensive disagreement cases and error confidence 0.495. We also include the Crowd Layer with full per-worker confusion matrices. Its 83.15% figure is on LabelMe, and no reported result was found on Lacuna for the Crowd Layer on HateXplain. Finally, we include Saliency Guided Training on top of soft labels as the interpretability-regularizer competitor. No reported result was found on Lacuna for SGT on HateXplain.

All models share the backbone, data splits, maximum sequence length, and a fixed budget of the same number of hyperparameter trials, selected on validation per-annotator log-likelihood. As a check, they are also selected on validation Macro F1. Three ablations isolate the key ingredients. The first removes the rationale bottleneck, keeping ordinal cutpoints on a pooled CLS-token severity. The second replaces the highlight noise model with HateXplain's averaged attention target. The third replaces two cutpoints per worker with a full 3×3 confusion matrix per worker. The proposal's central bet is that the cutpoint ablation is worse on sparse workers and the bottleneck ablation is worse on faithfulness and contested posts.

Why not just…?

Why not just train on soft labels with HateXplain's attention loss added? This is the real threat. It is cheap, it uses both the disagreement distribution and the rationales, and it may match our model on aggregate Macro F1 and calibration. What it cannot do is condition on who is judging. A soft label of [0.67, 0.33, 0] is the same target regardless of whether the minority vote came from a habitually lenient worker. Our test is the held-out per-annotator likelihood on contested posts, where we include soft-label-plus-attention as a baseline. If it matches us there, the per-annotator structure buys nothing, and we would report that the contribution reduces to the genie decomposition.

Why not just use per-annotator heads or the Crowd Layer, which already model individual workers? Model C's reported 0.245 accuracy on offensive disagreement cases suggests that giving each worker free output capacity overfits when each post has three labels and most workers label few posts. This is the sparsity limitation the Crowd Layer also has. Our cutpoint parameterization assumes that workers differ in harshness but not in what they consider evidence. The confusion-matrix ablation tests that assumption directly, stratified by how many posts each worker labeled.

Why not just use inter-annotator agreement as the ceiling instead of training genie models? Agreement statistics, such as the reported Krippendorff's α of 0.46, tell us that the task is hard but not why. They cannot distinguish error a model could remove by finding better evidence from error it could remove by knowing the judge. The genies answer that counterfactual question, and the decomposition is the claim. If both single genies help equally on identical-highlight posts, the evidence/severity split is wrong and we would say so.

Why not just use a hard extract-then-predict pipeline for guaranteed faithfulness? ERASER reports that such pipelines are faithful by construction but consistently less accurate than soft models. On HateXplain they would also inherit the leakage that normal posts carry no rationale. We keep the bottleneck soft and measure faithfulness instead of assuming it.

Proposed Experiments

On HateXplain we report Macro F1, AUROC, and GMB-Subgroup for aggregate predictions. For individual predictions we report per-annotator accuracy and negative log-likelihood, on all test posts and on the disagreement subset. We report error confidence and expected calibration error on contested posts. Following ERASER, we report Token F1 for plausibility, and comprehensiveness and sufficiency for faithfulness of the evidence weights $a_t$. Every figure is averaged over five seeds with paired bootstrap intervals.

The first decisive check is the decomposition itself. On contested posts with near-identical highlights, we predict that the evidence genie closes little of the gap and the threshold genie closes most of it. On contested posts with divergent highlights, we predict the reverse. A result where the evidence genie dominates on identical-highlight posts would falsify the separation hypothesis.

The second check is the gap-to-ceiling. The proposed model should recover a substantially larger fraction of the threshold genie's gain than Model B, Model C, or the Crowd Layer, while staying within seed noise of BERT-HateXplain on Macro F1 and GMB-Subgroup.

The third check is faithfulness. Because labels must pass through the evidence-weighted severity, comprehensiveness should exceed BERT-HateXplain's attention at equal Token F1. If it does not, the bottleneck is decorative and we would drop the interpretability claim.

A final sanity check confirms that each genie model actually outperforms its non-genie counterpart on validation. Without that, the ceilings are not ceilings.