When people label a short text as positive, negative, or neutral, the "neutral" label covers two different kinds of text. Some neutral texts are plain facts. Others are evaluative in ways that are hard to pin down, so annotators split and a majority vote hides the split. Sentiment models trained on these labels learn to send anything difficult into the neutral or "objective" bucket. This note proposes that much of that hidden disagreement comes from two linguistic properties that standard sentiment pipelines never represent. The first is figurative framing: a metaphor such as "this team is a sinking ship" carries an evaluation without using any sentiment word. The second is temporal openness: describing an event as unfinished or still to come ("the deal is expected to close") leaves its evaluative outcome undecided, while describing the same event as completed ("the deal closed") does not.
We would test this in three steps. We tag both properties as short multi-word spans. We check whether those spans predict which tweets annotators disputed, using a corpus that keeps all five votes per tweet. We then check causally whether rewriting a tweet to remove the metaphor, or to make the event completed, changes how much fresh annotators agree. If the causal test succeeds, neutral-class ambiguity becomes a measurable property of language that can be edited, rather than unexplained noise in the labels.
Core Claim
Our hypothesis is that, among tweets whose majority label is neutral or objective, disputed items contain metaphorical framing spans and temporally open event descriptions more often than unanimous items. We further hypothesize that these spans carry information about the spread of annotator votes that a fine-tuned encoder trained on the same votes does not recover on its own.
The evidence motivating this comes from several papers. Kenyon-Dean et al. (2018) built the McGill Twitter Sentiment Analysis corpus (MTSA) with five annotators per tweet. A logistic-regression classifier on that corpus reaches about 0.80 macro-F1 on unanimous tweets but only 0.53 on tweets where just three of five annotators agree. It scores 0.163 F1 on an explicit COMPLICATED class and defaults to OBJECTIVE on disputed tweets. Andreevskaia and Bergler (2006) showed at the level of word senses that human and system agreement collapses as a word's polarity score approaches zero. Neither paper says what in the text causes the collapse, and that is the question we ask.
The contrast with the closest prior work is direct. Mao and Li (2021) train metaphor tagging jointly with aspect-based sentiment, using a gate that filters what the two tasks share. Their goal is accuracy on clean polar benchmarks (77.6 F1 on the VU Amsterdam metaphor corpus, a 1.26% average gain on aspect sentiment). They do not separate the neutral boundary, do not use annotator disagreement as a target, and do not model time. We use metaphor, together with event boundedness, to explain and predict disagreement, and we test that explanation by intervention.
Research Program
Two span taggers
The first tagger marks metaphorical framing. It is trained on the VU Amsterdam Metaphor Corpus (VUA), whose words were labeled with MIPVU, a procedure that flags a word when its meaning in context contrasts with a more basic, physical sense. We reuse the method of Hall Maudslay et al. (2020), which feeds a classifier the gap between a word's general concreteness and its concreteness in context. That system reached 0.718 F1 on VUA across all parts of speech.
VUA labels single words, not phrases. We therefore group adjacent metaphor-flagged words that share a syntactic head into one span and encode the result with BIO tags. In this scheme each token is labeled as Beginning a span, Inside a span, or Outside any span, so a phrase like "sinking ship" becomes one unit.
The second tagger marks temporally open events. Its output spans cover a verb together with its arguments, labeled as one of three classes: completed and bounded, ongoing, or prospective. Spans are needed here for a linguistic reason: telicity, whether an event has a built-in endpoint, belongs to the whole verb phrase and not the verb alone. "Drank a cup of coffee" is bounded, while "drank coffee" is not.
We train this tagger in two parts. Boundedness comes from the MASC gold telicity data and the InterCorp silver labels of Friedrich and Gateva (2017), who reached 86.7% accuracy by projecting telicity labels from Czech, where aspect is marked in the morphology. Prospective versus past anchoring comes from tense and from temporal expressions, normalized with the compositional operators of Bethard (2013) and the SCATE schema of Laparra et al. (2018).
We expect both taggers to degrade on tweets. VUA systems fall from about 0.74 F1 on academic text to 0.55 on conversation (Leong et al., 2018). To measure this, we hand-annotate 500 MTSA tweets for both span types and report tagger quality on tweets before using the taggers downstream.
Sentiment model
The sentiment model is a RoBERTa-class encoder. It reads the tweet with the span tags supplied as extra input embeddings and is trained to output the full distribution of the five annotator votes, rather than the majority label. Chen et al. (2024) rejected sequence labeling because financial event descriptions are long and discontinuous. That objection does not apply to tweet-length, sentence-local spans. We still report how many gold spans are discontinuous on our tweet sample, so the assumption is checked rather than assumed.
Baselines
The naive reference point is a predictor that always outputs the majority class and the corpus-average vote distribution. Any proposed model must beat it on divergence from the true vote distribution.
The established reported baseline is the logistic regression of Kenyon-Dean et al. (2018), which uses n-grams, GloVe word vectors, and SentiWordNet scores. Its reported results are about 0.80 macro-F1 on unanimous MTSA data, 0.53 on majority data, and 0.163 F1 on COMPLICATED. We rerun it on our splits.
The strongest competitor is a plain fine-tuned encoder of the same size, trained on the same soft vote distributions with the same tuning budget and seeds, but given no span tags. It is the strongest because it sees exactly the same supervision. Katinskaia and Yangarber (2024) also found that transformers already encode verbal aspect and boundedness in their final layers, so this model could in principle learn our signals unaided. We found no reported result on Lacuna for a soft-label fine-tuned encoder on MTSA.
The closest multi-task competitor reimplements the gated metaphor–sentiment bridge of Mao and Li (2021), with VUA as the metaphor tower and MTSA as the sentiment tower. This tests whether simply sharing metaphor training explains any gain.
Two ablations isolate our key ingredients. The first replaces the BIO spans with per-token metaphor and boundedness probabilities, removing span structure while keeping the signal. The second removes one property at a time.
The proposal beats these baselines only if all three of the following hold. It must lower the divergence between predicted and empirical vote distributions on disputed neutral-majority tweets relative to the plain encoder, with bootstrap confidence intervals that exclude zero. It must improve COMPLICATED F1 over 0.163. And it must not lose accuracy on unanimous tweets.
Why not just…?
Why not just fine-tune an encoder on soft labels and let it find these cues? This is the real threat, and the plain encoder baseline tests it directly. If the plain encoder matches our model, we fall back on two checks. First, we probe its final layers for our span labels, following the method of Katinskaia and Yangarber (2024). Second, the minimal-pair experiment below asks whether the plain model's predictions shift the way human votes shift. If the plain model also tracks the human shifts, the modeling claim fails and only the explanatory finding survives.
Why not just prompt a large language model to label metaphor and predict disagreement? Reimann and Scheffler (2025) found that LLMs prompted step by step through MIPVU produce many false positives on conventional metaphors and fail to match supervised models. On a different but related task, Chen et al. (2024) report zero-shot GPT-4 at 36.10 F1 against 67.06 for a fine-tuned model. We nonetheless include a prompted model that directly predicts the vote distribution, as an additional comparison.
Why not use word-level metaphor probabilities instead of BIO spans? This alternative may well work as well as spans for metaphor. That is why it is an explicit ablation. If it ties, we report spans as unnecessary for metaphor and keep them only where telicity compositionality predicts they matter, which is the verb-phrase spans.
Proposed Experiments
The first experiment is correlational. Using MTSA's five agreement tiers, we test whether the rate of metaphor spans and of temporally open spans rises as agreement falls among neutral-majority tweets, controlling for tweet length and for the presence of sentiment-lexicon words.
The second experiment is a transfer test on WT-WT (Conforti et al., 2020), a corpus of tweets about corporate mergers labeled by whether they support, refute, or merely comment on the merger going ahead. There, neural models confuse the stance-neutral Comment class with Support. The best model, SiamNet, reaches about 62–63% macro-F1 against a 75% human upper bound, and accuracy drops from about 78% in healthcare to below 60% in entertainment. Prospective versus completed framing is exactly what should separate commentary on whether a merger will happen from support for it. We test whether adding the temporal-openness spans reduces both the Comment/Support confusion and the cross-industry drop.
The decisive check is a causal one. We write about 600 minimal pairs from disputed neutral-majority MTSA tweets. In each pair we either replace the metaphor with a literal paraphrase or change a prospective or ongoing event description to a completed one, keeping everything else fixed. Each version receives five new annotators under MTSA's protocol. The hypothesis predicts that agreement, measured as the entropy of the vote distribution, rises after rewriting. If rewrites leave agreement unchanged, the claim is false.
As a secondary check, a model that captures these properties should shift its predicted vote distribution in the same direction as the humans on these pairs. We compare the size of this shift between our model and the plain encoder.
References
- Sentiment Analysis: It's Complicated!
- Semantic Tagging at the Sense Level
- Bridging Towers of Multi-task Learning with a Gating Mechanism for Aspect-based Sentiment Analysis and Sequential Metaphor Identification
- Metaphor Detection using Context and Concreteness
- Classification of telicity using cross-linguistic annotation projection
- A Synchronous Context Free Grammar for Time Normalization
- SemEval 2018 Task 6: Parsing Time Normalizations
- A Report on the 2018 VUA Metaphor Detection Shared Task
- EFSA: Towards Event-Level Financial Sentiment Analysis
- Probing the Category of Verbal Aspect in Transformer Language Models
- Using Large Language Models to Perform MIPVU-Inspired Automatic Metaphor Detection
- Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twitter