Teaching a Tagger Where Humans Disagree: Token-Level Ambiguity as an Auxiliary Task for Abstract Word Senses

Language models label concrete words like "car" reliably and abstract words like "justice" or "want" poorly. Part of the reason is that humans also disagree about abstract words, and they do so in a structured way. Annotators who assign coarse semantic categories (supersenses such as "cognition", "relation" or "mental state") split systematically on metaphorical and polysemous uses. Standard training treats that split as noise. This proposal treats it as a signal the model should learn. A shared encoder is trained on two jobs at once. The first is ordinary supersense tagging on a large corpus with one label per word. The second is a small auxiliary task that predicts, token by token, how a panel of human annotators would distribute their labels. The bet is that learning which contexts are ambiguous from a few hundred multiply-annotated sentences reshapes the encoder so that abstract categories are tagged better everywhere, including on the large single-annotated corpus that carries no disagreement information.

What would be new is the granularity and the route. The closest prior work, Supersense tagging with inter-annotator disagreement, turned disagreement into a fixed class-by-class cost matrix inside a structured perceptron. It raised Danish A.MENTAL F1 from 0.0 to 16.53. A class-level matrix says that "cognition" and "communication" are often confused. It cannot say that this use of "grasp" is the contested one and that one is not. We move from class-level costs to token-level ambiguity, and from a single loss to multi-task learning, meaning one network with a shared body and separate output heads trained together. We then test whether this specifically helps the abstract categories where human agreement is lowest.

Why this intersection, and why not the obvious one

The two literatures meet naturally here. On the abstract-concept side, abstract categories lack the predictable grammatical contexts that concrete nouns have. Curran (Supersense Tagging of Unknown Nouns Using Semantic Similarity) and Ciaramita and Johnson (Supersense Tagging of Unknown Nouns in WordNet) showed classifiers struggling to separate abstract classes. Schneider et al. (Coarse Lexical Semantic Annotation with Supersenses: An Arabic Case Study) documented agreement falling on cognitive and general-action categories relative to physical objects. These papers motivate the proposal: the hard cases for models coincide with the hard cases for people. On the multi-task side, auxiliary objectives that adapt alongside a primary task can keep a shared representation from overfitting to noisy single-task labels (Situ et al.). Simultaneous training also avoids the forgetting that sequential intermediate fine-tuning causes (Can You Tell Me How to Get Past Sesame Street?). These papers supply the method.

The most obvious bridge is to jointly train the two Reading Comprehension of Abstract Meaning subtasks (imperceptibility and nonspecificity, from SemEval-2021 Task 4) to close the cross-definition gap of up to 14.4 points. We set it aside because it is uninformative. That benchmark's transfer subtask is defined as training on one definition and testing on the other, so joint training removes the test rather than passing it. The other obvious bridge is to apply gradient-conflict methods such as CAGrad (Conflict-Averse Gradient Descent for Multi-task learning) or RotoGrad (RotoGrad: Gradient Homogenization in Multi-Task Learning) to "concrete versus abstract" heads. That presupposes a conflict nobody has measured in language models. We therefore measure it as a diagnostic and do not build the proposal on it.

Core Claim

A tagger whose shared encoder is also trained to predict token-level annotator label distributions will improve F1 on low-agreement abstract supersenses more than a tagger given the same disagreement data through class-level costs, soft labels on the main head, or distillation. The gain will be concentrated on abstract categories and absent or small on concrete ones. The gain will disappear if the disagreement targets are shuffled across tokens, which shows that it comes from the information in the disagreement and not from the mere presence of an extra head.

Research Program

Model. We use a multilingual pretrained transformer encoder so that English and Danish share one model. It has two heads. The main head is a standard sequence tagger over the 26-plus WordNet supersense classes. It is trained with cross-entropy on SemCor, plus the gold or adjudicated labels of the other corpora where they are used for training. The auxiliary head is a separate softmax over the same label set. It is trained only on tokens with multiple annotations, from the English Twitter sets (Ritter, LOWLANDS) and the multiply-annotated portions of Danish SemDax. Its loss is the KL divergence (a measure of the gap between two probability distributions) from the empirical annotator distribution $p_t$ at token $t$ to the head's prediction $q_t$. The total objective is

$$ \mathcal{L} = \mathcal{L}_{\text{tag}} + \lambda \sum_{t \in A} \mathrm{KL}(p_t \,\|\, q_t), $$

where $A$ is the set of multiply-annotated tokens and $\lambda$ weights the auxiliary loss. Because $A$ is tiny (the file reports 40 to 200 sentences per Twitter set), we draw batches from the two sources at a fixed ratio, the balanced sampling of Yang et al.. This keeps the auxiliary signal from being drowned out. We tune both the ratio and $\lambda$.

The design choice that matters is keeping the two heads separate. The main head stays sharp on hard labels. The auxiliary head pushes the shared encoder to place contested contexts, such as metaphorical uses of verbs, in regions that sit between the classes annotators confuse. That structure then helps the main head on SemCor tokens that never had multiple annotators. Martínez Alonso et al. found that Danish disagreement costs transferred to English. We test the token-level analogue by training the auxiliary head on Danish only and evaluating English abstract categories.

Analysis. We will run three analyses. First, we report per-supersense F1, grouping classes a priori into concrete (e.g. N.ARTIFACT, N.FOOD) and abstract (e.g. N.COGNITION, N.RELATION, V.STATIVE, A.MENTAL). The grouping uses class definitions and is fixed before seeing results. Second, we test whether the auxiliary head's predicted entropy correlates with held-out annotator entropy, using cross-validation over the multiply-annotated sentences. Third, we compute the cosine between auxiliary and main-task gradients on abstract versus concrete tokens. This tells us whether the two tasks conflict, the situation CAGrad and RotoGrad were built for. We would add CAGrad as a variant only if that cosine is reliably negative.

Baselines

The naive reference point is the same encoder fine-tuned with hard-label cross-entropy on the same training data. Any multiply-annotated tokens contribute their adjudicated or majority label. This baseline shows what the disagreement information adds at all.

The strongest existing method is the cost-sensitive approach of Supersense tagging with inter-annotator disagreement. It builds a class disagreement matrix $D$ from the multiply-annotated data and trains with costs $C = 1 - D$. The reported figure is Danish A.MENTAL F1 rising from 0.0 to 16.53 for their structured perceptron with the JUSTSENSE factorization. We will reproduce their perceptron result as a sanity anchor. For fairness we will also port the cost matrix into our encoder as a cost-weighted cross-entropy, with identical data, encoder and tuning budget. No reported result was found on Lacuna for transformer-based supersense tagging on Ritter, LOWLANDS or SemDax, so the ported version is the competitor that counts. Beating it means a statistically reliable gain in macro-F1 over the abstract group, measured across at least five seeds, with concrete-group F1 no worse.

Two further baselines use the same disagreement data differently. The soft-label baseline trains the single main head on annotator distributions wherever they exist and on hard labels elsewhere. The distillation baseline first trains a tagger on the multiply-annotated data with soft targets, then uses it to give SemCor soft pseudo-labels for training the final tagger.

Two ablations isolate the key ingredient. In the shuffled-target control, each token's annotator distribution is replaced by that of a random other token within the same gold class. This keeps the class-level confusion statistics and destroys the token-level information. In the concreteness control, the auxiliary head regresses Brysbaert concreteness ratings instead of disagreement, which tests whether any graded abstractness signal would do. All variants get the same number of tuning trials over learning rate, $\lambda$ and sampling ratio, and the same training steps.

Why not just…?

Why not just train the main head on soft labels? This is the most serious threat. If annotator distributions are informative, putting them directly into the main head is the simplest use of them. We think it will fall short for two reasons. First, soft labels exist for only a few hundred sentences, so a single head trained on a mixture of hard and soft targets gets inconsistent supervision on similar contexts. Second, it pulls the main head's predictions toward hedging on exactly the tokens scored against a single gold label. The soft-label baseline tests this directly. If it matches the auxiliary head, we will report that the separate head is unnecessary and that the contribution reduces to token-level soft supervision.

Why not distill disagreement into SemCor and avoid multi-task training entirely? Distillation would carry token-level ambiguity onto the large corpus, and it may work. Its weakness is that the teacher is trained on 40 to 200 tweets per English set and has to extrapolate to edited prose. The multi-task model instead lets SemCor's own labels correct the encoder while the ambiguity signal shapes it. The distillation baseline settles which approach is better. A tie would argue for the simpler pipeline.

Why not just use a bigger pretrained model? Scale does not remove pretraining biases on abstract words. Untuned masked language models reached only about 60% on abstract cloze questions and chose frequent collocates over contextually correct abstract words (ZJUKLAB at SemEval-2021 Task 4), and large models still showed the 14.4-point cross-definition drop (SemEval-2021 Task 4). We will run the full comparison at two encoder sizes. If the abstract-group gain shrinks to zero at the larger size, that is a real negative finding and we will report it as such.

Why not just add a concreteness axis or concreteness auxiliary? Concreteness is nearly a single direction in static embeddings (On the Geometry of Concreteness). However, shifting along it barely helped separate senses of polysemous words, and abstract words form sprawling neighborhoods rather than shifted clusters (Semantic Similarity Computation for Abstract and Concrete Nouns). A word-level concreteness score is the same for "grasp an idea" and "grasp a rope", whereas annotator disagreement differs between them. The concreteness-auxiliary control measures this gap.

Decisive Checks

We will train on SemCor plus the training portions of the multiply-annotated sets, and evaluate on Ritter-eval, LOWLANDS and the SemDax test data. The primary metric is macro-F1 over the pre-registered abstract supersense group, with concrete-group macro-F1 and overall micro-F1 as guards. The proposal succeeds only if three conditions hold together. The auxiliary-head model must beat the ported cost-matrix, soft-label and distillation baselines on abstract-group F1 across seeds. The shuffled-target control must lose most of that gain. The improvement must be larger on abstract than on concrete classes. The cross-lingual check uses Danish-only disagreement to train the auxiliary head and evaluates English abstract classes. It tests whether the token-level signal transfers the way Martínez Alonso et al.'s class matrices did.

A secondary result could undercut the story. If the auxiliary head's predicted entropy does not correlate with held-out human entropy, any F1 gain cannot be credited to learning ambiguity, and we would treat it as generic regularization. The gradient-cosine diagnostic is reported either way. It is the first measurement of whether "concrete versus abstract" conflict in a shared language encoder is real enough to justify the conflict-averse optimizers that vision and robotics work has relied on.