When several people annotate the same sentence for emotion, they disagree in two different ways. They can mark different words as the cause of the feeling, or they can mark the same words and name a different feeling. Current emotion extraction systems cannot tell these two kinds of disagreement apart. They train on one "gold" span and one "gold" label per example, and the few models that do represent disagreement only predict a spread over sentence-level labels. Medical image segmentation has spent a decade on the matching problem, where radiologists trace different outlines of the same tumor, and its central lesson is that you must model a coherent joint distribution over whole outputs, not independent per-pixel guesses.
We propose to carry that lesson to text. The model is an annotator simulator that predicts a joint distribution over (cause span, emotion label) pairs, using a small mixture of "readings" of each sentence. It needs no annotator identities. It lets us split each item's disagreement into boundary uncertainty, label uncertainty, and the coupling between them. The core claim can be tested and could fail: if boundary choice and label choice are coupled across annotators, a joint model will beat factorized soft-label training on held-out annotators. If they are not coupled, the simple factorized model is enough, and knowing that tells dataset designers whether to fix their span guidelines or their category guidelines.
Where the two directions meet, and what is missing
Emotion extraction has moved from sentence labels to structured roles. SRL4E splits an emotional statement into four parts: a cue, an emotion drawn from Plutchik's eight categories, and spans for experiencer, target, and stimulus. It reports 64.3% emotion F1 when heterogeneous corpora are pooled. The same paper finds that long stimulus spans are much harder to extract than cues. Emotion-Cause Pair Extraction couples emotion clauses and cause clauses through interacting prediction heads, reaching 61.28% F1. Both works treat span and emotion as dependent, which motivates the joint view. Both also assume a single correct span per example, and this proposal drops that assumption.
Work on annotator disagreement in NLP has stayed at the level of whole-sentence labels. Soft-label multi-task learning adds a head that matches the vote distribution. It improves POS tagging from 85.73% to 86.76% and stemming from 73.59% to 77.09%, but it predicts one marginal per token or item. S-CNF, a conditional normalizing flow over label-probability vectors, reproduces bimodal crowd splits on MSP-Podcast and HateXplain without annotator IDs. It does not model spans, and its authors list structured outputs as future work. Fleisig et al. condition on annotator demographics, which emotion corpora almost never record.
The most direct evidence for our question comes from HateXplain, a hate-speech dataset in which three crowdworkers per post give a label and token-level rationales (Mathew et al.). The reanalysis in Majority Vote Silences Minority Values finds that 42.6% of disagreements fall at the hate/offensive boundary. Annotators often highlight identical spans (Jaccard similarity 1.0) while choosing different labels. That is one case of "same where, different what." Nobody has measured how common each pattern is in emotion data, or built a model that predicts which pattern a new sentence will produce.
The medical segmentation line supplies both method and caution:
- Coherent samples. The Probabilistic U-Net samples a low-dimensional latent code so that each output is a complete, coherent segmentation. It beats dropout, ensembles, and multi-head models on generalized energy distance (GED), a sample-based distance between predicted and human output sets. We reuse GED, adapted to spans.
- Rater bias versus rater inconsistency. Pionono (κ = 0.758 on Gleason 2019) and the Latent Doctor Model (KL 0.461 on Gleason 2019; entropy correlation with the human panel up to 0.728 on ISUP) separate systematic rater bias from inconsistency. Both require annotator IDs or support sets, which constrains us toward an anonymous design.
- Downstream calibration. DiFF settles boundary disagreement by asking which outline best serves a downstream diagnosis. Emotion has no biopsy-confirmed diagnosis, so we rule this route out rather than port it.
One real difference from images shapes the design. A text span is one-dimensional, so every candidate span in a sentence can be listed and scored exactly. The Probabilistic U-Net needs a continuous latent code because pixel masks cannot be enumerated. We do not need one to make spans coherent. We need a latent only to couple the span with the label.
Core Claim
Let $x$ be a sentence, $s$ the stimulus span an annotator marks (one of the enumerable contiguous spans, plus "none"), and $y$ the emotion label that annotator assigns. We model one annotator's full answer as
$$p(s, y \mid x) = \sum_{k=1}^{K} \pi_k(x)\, p(s \mid x, k)\, p(y \mid x, k),$$
where $k$ indexes one of $K$ latent readings of the sentence and $\pi_k(x)$ is the predicted share of annotators who adopt reading $k$. Span and label are independent within a reading but can be coupled across readings.
Setting $K=1$ gives exactly the factorized soft-label model, with separate span and label distributions. The comparison between $K=1$ and $K>1$ is therefore the ablation that isolates coupling. The model splits each item's total disagreement as
$$H(s, y \mid x) = H(s \mid x) + H(y \mid x) - I(s; y \mid x),$$
where $H(s \mid x)$ is boundary disagreement, $H(y \mid x)$ is label disagreement, and the mutual information $I(s; y \mid x)$ measures how much knowing an annotator's span tells you about their label. The HateXplain pattern corresponds to low $H(s \mid x)$ with high $H(y \mid x)$. A sentence read as "afraid of the storm" by some annotators and "surprised by the news" by others, with different spans, corresponds to high $I(s; y \mid x)$.
We claim three things. First, on emotion data, $I(s; y \mid x)$ is non-negligible for a meaningful fraction of items, concentrated in adjacent pairs such as fear/surprise and anger/disgust. Second, a model trained without annotator IDs can predict this split on unseen sentences. Third, on those items the mixture model gives better held-out joint likelihood and more realistic sampled annotations than any factorized alternative.
Research Program
Data. The file does not show that the SRL4E source corpora keep per-annotator spans, so the program starts by collecting data. We will re-annotate a stratified sample of about 2,000 SRL4E items with five annotators each. Each annotator marks the cue, the stimulus span, and a Plutchik label. The sample will be stratified across tweets, blogs, and literature, and oversampled for the confusable emotion pairs.
HateXplain serves as a second, existing testbed. Its labels are not emotions, but it has per-annotator rationale spans and labels, plus a published finding to reproduce. With only three annotators per post, per-item mutual information is noisy there. We therefore estimate it pooled over bins of items, and use the five-annotator emotion set for per-item claims.
Model. The model has four parts:
- Encoder. A shared RoBERTa-class encoder.
- Span scorer. A span-enumeration head that gives, for each reading $k$, a softmax over all spans up to a length cap, plus "none."
- Label head. A per-reading label head.
- Mixing head. A head that predicts the reading weights $\pi_k(x)$.
We train by maximizing the marginal likelihood of each individual annotator's (span, label) tuple, not the aggregate. This lets the model learn anonymous readings, in the spirit of S-CNF's ID-free simulation.
Pionono's idea of intra-rater inconsistency enters through a per-reading span temperature. Each reading can have a sharp or a diffuse boundary, which separates "annotators agree on the cause but disagree by a word at the edge" from "annotators pick a different cause."
We will choose $K$ between 1 and 6 on validation likelihood. We will also run a continuous-latent variant modeled on the Probabilistic U-Net, as a check on whether discrete readings lose anything.
Analysis. For every item we report the three-term split above and aggregate it by emotion pair and by domain. This answers the question that Majority Vote Silences Minority Values raised for hate speech but not for emotion. We also test whether items the model flags as high-coupling are the ones on which majority-vote models are most confidently wrong.
Baselines
Majority-vote training. The do-nothing reference is a single span extractor and classifier trained on the majority span and label. This is SRL4E's setup, and Model A in the HateXplain disagreement study. It anchors how much any distributional model buys. The file gives these reference numbers:
- BERT-HateXplain reports Macro F1 0.687 and AUROC 0.851 (Mathew et al.).
- Majority-trained models reach about 80% accuracy on agreed posts but about 58% on disagreement posts, with confidence 0.710 when wrong (Majority Vote Silences Minority Values).
- The SRL4E all-in-one model reports 64.3% emotion F1.
We must not lose more than one point of majority-label F1 relative to this baseline, while improving the distributional metrics.
Factorized soft labels. The strongest competitor is factorized soft-label training in the style of Fornaciari et al.. We will pair a soft-label head over enumerated spans with a separate soft-label head over emotions, which is exactly our model at $K=1$. It captures both marginals and differs from us only in coupling, so beating it is the entire claim. We run it with the same encoder, the same span enumeration, the same data, and the same hyperparameter search budget, choosing the better of KL and inverse KL on validation.
Latent-variable alternatives. Three further baselines cover existing latent approaches:
- S-CNF plus independent span head. S-CNF for labels, paired with an independent span soft-label head. It is the strongest existing ID-free label simulator.
- Per-annotator heads. Model C from Majority Vote Silences Minority Values, with reported error confidence 0.495 and 0.245 accuracy on offensive disagreement cases. HateXplain only, since it needs IDs.
- Continuous-latent port. A direct Probabilistic U-Net-style variant with a continuous latent broadcast over tokens.
Sampling-based uncertainty. Deep ensembles and Monte Carlo dropout, meaning dropout left on at test time, are included at a matched number of samples.
Metrics and the gap in reported numbers. All models are compared on the following, with bootstrap confidence intervals:
- Held-out-annotator joint negative log-likelihood.
- Span-adapted GED, using one minus token IoU plus a label-mismatch term as the distance.
- Calibration on disagreement items.
- Correlation between predicted and pooled empirical $I(s; y \mid x)$.
No reported result was found on Lacuna for any multi-rater joint span-and-label distribution metric on emotion data, so these comparisons are new measurements rather than beats of published numbers. Beating the factorized baseline means a statistically significant reduction in joint NLL and GED that is concentrated on high-coupling items, with no loss on low-coupling items.
Why not just…?
Why not just train soft labels for span and emotion separately? This is the real threat, and the design makes it the $K=1$ special case rather than an afterthought. The HateXplain finding that annotators share spans while splitting on labels suggests coupling may be weak there, and in that regime the factorized model is sufficient. The experiment settles the question directly. If the mixture gains nothing on the five-annotator emotion set, we report that emotion disagreement is effectively separable. That is a useful negative result: guideline fixes for boundaries and for categories can then be pursued independently.
Why not just port the Probabilistic U-Net directly? Its continuous latent exists to keep pixel masks coherent, a problem that disappears when spans can be enumerated exactly. A continuous code over tokens adds variational training and sampling cost without a clear benefit. We include it as a baseline and expect it to match, not exceed, the discrete mixture. If it wins, we adopt it, but the decomposition analysis stays the same.
Why not just use an ensemble or dropout to get diversity? In images, Kohl et al. found that ensembles drift toward the frequent mode and drop rare ones, while dropout gives incoherent noise. Ensembles also cost roughly ten times more to train than S-CNF in Wu et al.. More basically, seed diversity reflects model uncertainty, not the spread of human readings. Our matched-sample comparison on held-out-annotator likelihood will show whether that distinction matters for text.
Why not just condition on annotator identity or demographics? The emotion corpora used here record neither. Model C's collapse to 0.245 accuracy on HateXplain boundary cases also shows that per-annotator heads can overfit individual raters. Where IDs exist, on HateXplain, we compare against them directly.
Proposed Experiments
Test 1: does coupling exist? On the new five-annotator SRL4E subset, we compare the mixture model with $K>1$ against the $K=1$ factorized model on held-out-annotator joint NLL and span GED, broken down by emotion pair and domain. The idea is supported only if the gain is significant and concentrated on items with high empirical $I(s; y \mid x)$.
Test 2: can the model predict the split? We bin test items by predicted mutual information and check that the empirical, annotator-pooled mutual information rises monotonically across bins. We run this on both the emotion subset and HateXplain. On HateXplain the model should reproduce the published "same span, different label" pattern as low predicted $H(s \mid x)$ with high $H(y \mid x)$ at the hate/offensive boundary.
Test 3: is the gain robust out of domain? We train on two SRL4E domains and test on the third, following the cross-domain protocol that underlies SRL4E's 64.3% figure. This checks whether the learned readings transfer or merely memorize a domain's annotator pool.
Test 4: does the split explain majority-vote failures? We ask whether high-coupling items account for a disproportionate share of the confident errors of the majority-vote baseline. If they do, the decomposition identifies which items need boundary guidelines and which need category guidelines. If they do not, the practical case for the joint model rests on likelihood gains alone, and we will say so.
References
- SRL4E - Semantic Role Labeling for Emotions: A Unified Evaluation Framework
- Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts
- Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning
- Modelling Variability in Human Annotator Simulation
- When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks
- HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
- Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain
- A Probabilistic U-Net for Segmentation of Ambiguous Images
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle