Many words have several meanings, and dictionary-matching systems for picking the right one still fall back on the most common meaning when the context points elsewhere. Word sense disambiguation (WSD) means choosing which dictionary meaning, or sense, of a word is intended in a sentence. The best current systems use label embeddings: they encode each candidate sense's dictionary definition (its gloss) as a vector and score it against the word in context. This lets them handle senses with few or no training examples. Yet on a set of instances that every strong system gets wrong, those systems still predict the most frequent sense about 63% of the time, even though that sense is correct in only 2% of the cases (Nibbling at the Hard Core of Word Sense Disambiguation).
We think the cause is where the frequency information ends up. When a gloss encoder is trained on a corpus with a heavily skewed, long-tailed distribution of senses, the definition of a common sense is a positive target thousands of times and a rare one only a handful of times. The gloss vector therefore stores two things at once: what the sense means and how often it occurs. We propose to separate them. The model would learn a gloss geometry that is trained not to encode frequency, and it would carry frequency as an explicit prior term. A small learned gate would decide, for each context, how much weight that prior deserves. The two-sentence version: gloss matchers default to common senses because frequency leaks into the gloss vectors, so we train the gloss space to exclude frequency and add frequency back as a separate, context-gated term. If this works, rare-sense accuracy should improve without the loss on common senses that rare-sense data augmentation causes.
Core Claim
The claim has two parts, and each can be tested on its own. The first is diagnostic. In a jointly trained gloss bi-encoder, meaning a model that encodes the context and each gloss separately and scores them by dot product, the training frequency of a sense can be decoded from its gloss embedding alone. This signal would show up as the vector's norm or as its alignment with a shared direction. It would also explain a large share of hard-core errors.
The second part is corrective. The score for sense $s$ in context $x$ would be written as
$$ \text{score}(x,s) = f(x)^\top g(s) + \alpha(x)\,\log \pi(s \mid w), $$
The terms are as follows. $f(x)$ is the context embedding of the target word. $g(s)$ is the gloss embedding. $\pi(s \mid w)$ is the smoothed training frequency of sense $s$ among senses of lemma $w$. $\alpha(x)\ge 0$ is a gate that depends on the context. During training, $g$ is fit with the prior added at a fixed full weight, following the logit-adjusted loss idea. This gives the frequency signal an explicit place to go, so $g$ has no need to absorb it. The gate $\alpha$ is then fit on held-out folds. The purpose is that it learns how often the prior truly helps on unseen data, as opposed to how often it helps on memorized training data. For senses with no training examples, $\pi$ falls back to a smoothed floor, so the prediction rests on the gloss score.
Relation to Prior Work
The method builds on Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders (BEM). BEM jointly trains the context and gloss encoders with a dot-product objective and reports 79.0 F1 overall, 94.1 F1 on most-frequent senses, and 68.9 F1 on unseen senses. We keep that architecture and change what the gloss vector is permitted to encode. BEM's authors themselves point to reweighting or calibration as future work, and none of the later gloss models in our file take that up.
The extractive systems motivate the proposal and also compete with it. ESC: Redesigning WSD with Extractive Sense Comprehension (ESCHER) concatenates all candidate glosses with the sentence and selects a span. It already fights frequency with gloss noise, which injects frequent definitions as negatives. ConSeC: Word Sense Disambiguation as Continuous Sense Comprehension feeds disambiguated neighbouring words back into the input. Both fold frequency into their scores implicitly. Maru et al. show that ESCHER still drops to 54.1 micro-F1 on the rare-sense 42D set. Our approach differs because it makes frequency an explicit, removable quantity; it does not try to push frequency out with noise.
Other long-tail methods constrain the design. Non-Parametric Few-Shot Learning for Word Sense Disambiguation (MetricWSD) trains on balanced episodes, meaning small batches sampled so each sense is equally represented, and gains 6.6% on rare senses. It explicitly omits glosses. Balanced sampling of this kind is a direct alternative to our training-time prior, so we include it as an ablation. Quantum-inspired Non-homologous Representation Constraint Mechanism for Long-tail Senses of Word Sense Disambiguation constrains the geometry of rare-sense representations and pairs it with a separate head-sense recognizer. We aim for a single model with one interpretable prior term, not a hybrid of two systems. Improved Word Sense Disambiguation with Enhanced Sense Representations enriches glosses with synonyms and hypernym definitions. That is orthogonal to our change and could be combined with it.
From outside WSD, CARE: Class-Adaptive Expert Consensus for Reliable Learning with Long-Tailed Noisy Labels supplies the idea of pairing semantic label embeddings with logit adjustment on long-tailed image data. Its adjustment is global and class-level. Ours is gated per context and is specific to each lemma, because the relevant prior for a sense is its frequency among that lemma's senses, not across all labels.
Research Program
The first stage is diagnostic. We would retrain BEM on SemCor, which has roughly 226,000 annotated examples, and probe the resulting gloss embeddings. The main probe is a regression of log training frequency on the gloss vector norm and on its projection onto the mean positive direction. We would also run a gloss-paraphrase swap: rewrite a rare sense's gloss while keeping its meaning, or give a rare sense a frequent sense's wording. If scores follow frequency rather than meaning, that is direct evidence of the leak. Finally, we would measure how much of the most-frequent-sense default on $ALL_{HC}$ disappears when we remove only the frequency-predictive component from $g(s)$ at test time. If frequency is not decodable, or removing it changes nothing, the premise fails and we would stop.
The second stage trains the factored model with the same BERT-base backbone and the same data as BEM. The only change in training is the added $\log\pi$ term. The gate $\alpha(x)$ is a small network with two kinds of input. One is the context embedding. The other is the margin between the top two semantic scores. The intuition is that the prior should matter only when the semantic evidence is close. We fit the gate by cross-fitting: split SemCor into $K$ folds, estimate $\pi$ on $K-1$ folds, and fit $\alpha$ on the remaining fold. This prevents the gate from learning that the prior is always right, which is what it would learn on memorized training data.
The third stage checks that the gains carry over. We would apply the same factoring to an ESCHER-style extractive model by adding the prior to its span scores, which tests whether the finding is specific to bi-encoders. We would also combine the factored model with enriched glosses.
Baselines
The naive reference points are the most-frequent-sense lookup and plain BEM retrained with our seeds and data. Beating the lookup is trivial on $ALL_{HC}$, where it is correct 2% of the time, so it mainly serves to anchor the head-sense numbers.
The strongest bi-encoder competitor is BEM itself, which reports 79.0 F1 on the standard $ALL$ benchmark, 94.1 F1 on most-frequent senses, and 68.9 F1 on unseen senses. It shares our architecture and training data, so any gain isolates the factoring. The strongest system overall is ConSeC at 82.0 F1 on $ALL$ trained on SemCor. On rare senses specifically, the strongest reported number is ESCHER's 54.1 micro-F1 on 42D, from Maru et al. We would rerun ESCHER and ConSeC checkpoints on $ALL_{NEW}$, 42D, and $ALL_{HC}$ rather than rely on legacy $ALL$ numbers.
The most dangerous baseline is post-hoc logit adjustment. It takes trained BEM and subtracts $\tau\log\pi(s\mid w)$ at test time, with $\tau$ tuned on SemEval-2007, the standard development set. No reported result was found on Lacuna for post-hoc frequency adjustment of gloss bi-encoders. We would give it the same development set and the same tuning budget as our gate.
A second ablation uses MetricWSD-style balanced episodic sampling in place of the training-time prior. Two further ablations isolate the gate: a fixed global $\alpha$, and $\alpha=0$ at test time. Beating these baselines means improving 42D and less-frequent-sense F1 over tuned post-hoc adjustment by a margin larger than the seed variance, while keeping most-frequent-sense F1 within about one point of BEM's 94.1.
Why not just…?
Why not just subtract a scaled log-prior from BEM at test time? This is the real threat. If post-hoc adjustment matches us, our contribution shrinks to the diagnostic. The argument against it is that it moves every lemma's decision boundary by the same amount. It therefore trades head senses for tail senses, the same trade-off Maru et al. document for data augmentation. It also cannot undo frequency already stored in $g(s)$. The comparison at matched most-frequent-sense F1 settles the question. If our model is no better on 42D at equal head accuracy, we would report exactly that.
Why not just train on frequency-balanced batches? MetricWSD shows that balanced episodes help rare senses. However, balancing throws away real frequency information that the head senses legitimately need, and it offers no way to restore that information depending on the context. The balanced-sampling ablation tests this directly, and we would look especially at most-frequent-sense F1.
Why not just use richer glosses or a cross-attending model? ESR and ConSeC reach 82.0 F1 on $ALL$. Even so, the gloss-informed systems Maru et al. audit, ESCHER among them, still default to the most frequent sense on $ALL_{HC}$. That suggests better gloss text alone does not fix the frequency bias. We treat this as complementary and test enriched glosses combined with factoring.
Why not just prompt a large language model? In In the LLM era, Word Sense Induction remains unsolved, GPT-4o scores 66.9 F-B³ on SemCor-WSI, below the trivial one-sense-per-word baseline of 73.6. That task is sense induction, not disambiguation, so the evidence is only suggestive, but we would not expect a large language model alone to solve rare senses. Towards Effective Disambiguation for Machine Translation with Large Language Models points the same way: gains came only from retrieving demonstrations of the exact sense, and that retrieval failed for rare senses in 7.5% of cases.
Decisive Checks
We would evaluate on $ALL_{NEW}$, 42D, $ALL_{HC}$, hardEN, FEWS, and the fewer-than-three-example long-tail splits, reporting F1 separately for most-frequent senses, less-frequent senses, and unseen senses. The main number to watch is the rate of most-frequent-sense predictions on $ALL_{HC}$, which starts from roughly 63%. It should fall substantially without most-frequent-sense F1 dropping more than about a point.
hardEN was built adversarially against earlier systems, so any gain there is informative but fragile. We would report it with bootstrap intervals.
The idea is falsified in three cases. The first is if frequency cannot be decoded from gloss vectors. The second is if tuned post-hoc adjustment matches our model at equal head accuracy. The third is if the cross-fitted gate learns a constant, which would mean the context-dependent part adds nothing.
References
- Nibbling at the Hard Core of Word Sense Disambiguation
- Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders
- ESC: Redesigning WSD with Extractive Sense Comprehension
- ConSeC: Word Sense Disambiguation as Continuous Sense Comprehension
- Non-Parametric Few-Shot Learning for Word Sense Disambiguation
- Quantum-inspired Non-homologous Representation Constraint Mechanism for Long-tail Senses of Word Sense Disambiguation
- Improved Word Sense Disambiguation with Enhanced Sense Representations
- CARE: Class-Adaptive Expert Consensus for Reliable Learning with Long-Tailed Noisy Labels
- In the LLM era, Word Sense Induction remains unsolved
- Towards Effective Disambiguation for Machine Translation with Large Language Models