Teaching a Bi-Encoder's Verb Vector to Know Its Arguments

Word sense disambiguation means picking which dictionary meaning of a word is intended in a given sentence. The strongest practical systems are bi-encoders, also called dual-encoders. One network turns the sentence into a vector for the target word, a second network turns each dictionary definition (gloss) into a vector, and the best-scoring dot product wins. Because the gloss vectors can be computed once and cached, inference costs one forward pass. These models handle rare and unseen senses far better than fixed classifiers. They still fail on verbs, though: enriched bi-encoders report verb F1 below 70% even when overall F1 reaches 82%. Verb senses are distinguished largely by their arguments. "Run a race" and "run a business" differ in what is being run, and a single pooled vector at the verb token has no explicit pressure to encode that.

We propose to supply that pressure with multi-task learning, meaning training one shared network on several objectives at once. During training, the context encoder gets an auxiliary objective: from the target verb's vector alone (the very vector used in the dot product), predict the semantic roles and coarse types of the verb's arguments, using silver labels from an off-the-shelf role labeler run over the training text. At test time the auxiliary head is thrown away. No parser runs, gloss vectors stay cached, and inference cost is unchanged. The claim is that structural knowledge can be pushed into the matching vector through a training-time task rather than through inference-time features. We also claim that where the auxiliary task reads from matters. A head that reads the verb's own pooled vector should help verbs, while a conventional token-level role-labeling head should not.

Core Claim and Relation to Prior Work

The bi-encoder recipe comes from Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders. That paper's model, called BEM, reaches 79.0 F1 on English all-words WSD and 68.9 F1 on zero-shot senses, and it already identifies verbs as a weak point. Improved Word Sense Disambiguation with Enhanced Sense Representations (ESR) enriches the gloss side with synonyms, examples and hypernym glosses and reaches 82.0 F1, but verbs stay below 70%. These two papers motivate the work. Both improve what the gloss encoder sees, and neither changes what the context vector is trained to contain. Our change is on the context side and is complementary, so we stack it on ESR as well as on BEM.

The idea that argument structure fixes a bi-encoder's blind spots comes from Predicate-Argument Based Bi-Encoder for Paraphrase Identification. That paper adds self-attended predicate-argument spans as extra input features and lifts PIT2015 F1 from 52.03 to 59.19. It supplies the intuition but constrains the design, because it needs a parser at inference. We test whether the same benefit can be gained without one.

Evidence that a syntactic auxiliary task helps a semantic main task comes from Transfer Learning for Neural Semantic Parsing. There, co-training with constituency parsing improved ATIS by 2.5%, which motivates the multi-task route.

The existing multi-task bi-encoders take a different approach. UniRetriever shares one dual-encoder across persona, knowledge and response retrieval, and Multilingual Word Sense Disambiguation with Unified Sense Representation co-trains WSD across languages. Both multiply instances of the same matching task. Neither adds a non-matching structural objective that shapes the matching vector. Multi-Task Learning for Metaphor Detection with Graph Convolutional Neural Networks and Word Sense Disambiguation pairs WSD with another task, but it does not use gloss matching at all.

Research Program

The base model is BEM: a BERT-base context encoder and a gloss encoder trained with cross-entropy over each word's candidate senses on SemCor. We run an off-the-shelf semantic role labeler over SemCor once to obtain, for each verb instance, its argument heads, their role labels (agent, patient, and so on), and a coarse type for each head (person, organization, event, physical object), taken from that head's WordNet top-level category. These labels are silver, not gold.

The auxiliary head is a small multi-layer perceptron. It takes the target-verb vector $\mathbf{u}$ and predicts, for each role slot, whether the slot is filled and the coarse type of its filler. The training loss is

$$ \mathcal{L} = \mathcal{L}_{\text{match}} + \lambda \, \mathcal{L}_{\text{arg}}(\mathbf{u}), $$

where $\mathcal{L}_{\text{match}}$ is the usual cross-entropy over gloss dot products and $\mathcal{L}_{\text{arg}}$ is the slot-wise prediction loss. $\lambda$ is a weight tuned on SemEval-2007, the standard development set.

The design choice under test is that $\mathcal{L}_{\text{arg}}$ reads $\mathbf{u}$ and nothing else. Information about a verb's object therefore has to be packed into the same vector that is matched against glosses. Verb glosses often name argument expectations ("to direct or manage a business"), so the hypothesis is that an argument-aware $\mathbf{u}$ aligns better with the right gloss.

Two secondary questions follow. First, silver role labels are noisy and the auxiliary loss may dominate early training. We therefore compare a fixed $\lambda$ against magnitude balancing in the style of RotoGrad and Dual-balancing for multi-task learning. We use only RotoGrad's magnitude component, since its rotation component costs memory quadratic in the feature dimension and the head here is small. Second, we check whether the gain transfers to Wiktionary verbs in FEWS, which tests whether the model learned a general skill rather than SemCor-specific cues.

Baselines

The naive reference point is BEM retrained by us with the same seeds, data, and number of training steps, and with $\lambda = 0$. Its published figure is 79.0 F1 on the English all-words evaluation (BEM). The file gives no exact BEM verb F1, so no reported result was found on Lacuna for the BEM verb breakdown. We report our own reproduction.

The strongest existing method is ESR on RoBERTa-large, which reports 82.0 F1 on all-words WSD and verb F1 below 70% (ESR). We rerun ESR with its enriched glosses at the same backbone and add our auxiliary loss on top, under an identical hyperparameter search budget for both arms. Beating ESR means a verb F1 gain over ESR of at least two points on the concatenated "ALL" set, significant under paired bootstrap resampling, with noun and adjective F1 within half a point of ESR.

Z-Reweighting (Rare and Zero-shot Word Sense Disambiguation using Z-Reweighting) reports 78.6 F1 against its own BEM reproduction at 78.1. It is the strongest single-task alternative aimed at rare senses. We include it at BERT-base to test whether our gains are just rare-sense gains in disguise.

Two ablations isolate the key ingredient. The first is a token-level role-labeling head on the argument tokens, the standard multi-task setup. It uses the same labels and the same $\lambda$ search but does not touch $\mathbf{u}$. The second is a permuted-label control: the same auxiliary head, with role labels shuffled across instances. It separates structural information from the generic regularizing effect of any extra loss.

Why not just…?

Why not just feed predicate-argument spans in as features, as the paraphrase bi-encoder does? This is the most serious threat. Predicate-Argument Based Bi-Encoder for Paraphrase Identification shows that explicit spans help, and doing the same for WSD might match or beat our auxiliary loss. We run it as a full baseline, with spans from the same role labeler concatenated to the verb representation at train and test time. If it wins clearly on verb F1, the honest conclusion is that structure must be supplied at inference. Our method then reduces to a cheaper approximation, and we report how much accuracy it gives up in exchange for removing the parser.

Why not just use a cross-encoder that attends jointly over sentence and gloss? A cross-encoder needs one full forward pass per candidate sense and cannot cache gloss vectors. In this literature it is also not more accurate: GlossBERT reports 77.0 F1 against BEM's 79.0 and its own 78.1 reproduction (Z-Reweighting). Our method keeps the bi-encoder's cost profile exactly.

Why not just add more dictionary data, such as FEWS, to training? Mixing FEWS with SemCor already lifts zero-shot F1 on "ALL" from 68.9 to 74.8 (FEWS). This is a strong, cheap move, but it targets unseen senses, not verb argument confusions. We run BEM and our model both with and without FEWS in training. If our verb gain vanishes once FEWS is added, the auxiliary task is redundant with more data. If the gains add up, the two fix different failures.

Why not just enrich the glosses further? ESR already does this and its verbs stay below 70% (ESR). That result suggests the gloss side is not the bottleneck for verbs, and it is why we build on ESR rather than compete with it.

Proposed Experiments

We train on SemCor and evaluate on Senseval-2, Senseval-3, SemEval-2013, SemEval-2015 and the concatenated "ALL" set, with SemEval-2007 as development. We report F1 by part of speech and by frequency split: most frequent senses, less frequent senses, and zero-shot senses. For transfer we evaluate on FEWS verbs, including its zero-shot verb senses, where the BEM baseline reports 72.77% overall and 66.48% zero-shot accuracy and humans reach about 80% (FEWS). We also evaluate on the verb items of Word-in-Context (WiC), a binary task asking whether a word has the same sense in two sentences, using the same frozen encoder.

Three decisive checks determine whether the claim holds.

The first is a linear probe on frozen $\mathbf{u}$ vectors from held-out SemEval data. It tests whether $\mathbf{u}$ actually encodes argument types better than in BEM. If it does not, any accuracy gain has some other cause.

The second is a minimal-pair diagnostic. We collect test verb instances whose gold senses differ while the verb lemma and syntactic frame match, and we ask whether errors concentrate on pairs whose argument types differ. Our method should reduce exactly those errors.

The third is the token-level ablation and the permuted-label control. If the token-level head matches the target-vector head, the "where the auxiliary task reads from" claim fails. If the permuted-label control matches it, the gain is generic regularization and not structure. We will report whichever outcome occurs.