Recurrent Descent Through WordNet for Rare-Sense Disambiguation

Word sense disambiguation means deciding which dictionary meaning of a word is intended in context, for example whether "bank" means a riverside or a financial institution. It is a clear case of the long-tail problem. The largest hand-labeled training set, SemCor, gives most senses a handful of examples or none. Non-Parametric Few-Shot Learning for Word Sense Disambiguation reports that 84% of its annotated words appear fewer than 10 times. Even the strongest gloss-based systems leave a gap of roughly 40 F1 points between a word's most frequent sense and its rarer ones (Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders). A rare sense is rarely isolated, though. In WordNet, the rare sense of "bank" meaning a row of similar objects sits under "arrangement," then "group," and eventually under a broad category, and many other words attest those upper nodes densely.

We propose to predict a sense the way one reads a path through the taxonomy: coarse category first, then progressively finer nodes. A small recurrent network walks down WordNet's hierarchy one level at a time. At each step it conditions on the sentence and on the nodes it has already chosen. Each node is represented by an embedding of its dictionary definition, so a sense with no training examples still gets a representation built from the definitions of its ancestors. The claim is that recurrence over taxonomy depth, rather than over words in a sentence, lets rare leaf senses borrow evidence from well-attested ancestors in a way that flat scoring and additive two-level scoring cannot. We further predict that the benefit appears exactly where a rare sense has frequently attested ancestors, and nowhere else.

Core Claim and Position Relative to Prior Work

The mechanism combines two ideas that have so far been used separately. The first is hierarchical backoff. Hierarchical Semantic Classification adds a score for the specific sense to a score for one of WordNet's 26 broad "supersenses." That raised Senseval-2 accuracy from 70.4% (flat perceptron) to 71.8%. GPLSI uses Basic Level Concepts, which are mid-level nodes found automatically in the hypernym hierarchy, as features to reduce polysemy. Both papers motivate our proposal, since they show that upper levels carry usable signal for sparse leaves. Both are also limited. The first uses exactly two levels combined by a sum. The second treats hierarchy nodes as fixed features of a linear classifier. Neither lets the choice at one level change how the next level is scored.

The second idea is definition-based sense representation, which supplies our method. Incorporating Glosses into Neural Word Sense Disambiguation (GAS) encodes definitions with a bidirectional LSTM, and in its extended variant concatenates hypernym and hyponym definitions into one linear string. The authors acknowledge that this flattens the hierarchy. It reaches 70.6 F1 on the unified benchmark. The bi-encoder BEM embeds contexts and definitions with BERT in a shared space and reaches 79.0 F1 overall and 68.9 F1 on senses never seen in training. We reuse BEM's context and definition encoders unchanged. What we replace is its final step. Instead of one dot product between the context and each candidate's definition, we score a sequence of decisions down the candidates' shared ancestry.

The research file frames a third line as tension rather than method. A Deep Dive into Word Sense Disambiguation with LSTM shows that an LSTM language model with averaged per-sense context vectors cannot represent senses missing from the annotations at all. It falls back to the most frequent sense, and scaling unlabeled text from 1.8B to 100B tokens barely helps. This constrains the design: the gains must come from structure attached to unseen senses, not from more unlabeled text.

We set aside another natural RNN-plus-sparsity idea, which is to modulate LSTM gates by token frequency, inspired by Rare Words Degenerate All Words. A skeptic would rightly answer that subword tokenization and that paper's embedding-level gradient gating already address rare input tokens. They would add that the sense-label long tail, not the token long tail, is what drives the frequent-versus-rare gap.

Model

For a target word in context $c$, let $S$ be its candidate senses in WordNet. Take the union of their hypernym paths and prune it into a small local tree. The root is the lexicographer file, which in WordNet coincides with the supersense. Next come Basic Level Concept nodes where they exist, then the intermediate hypernyms at which candidates diverge, then the leaf synsets. Nodes with a single child are collapsed, so every step is a real choice.

Adjectives and adverbs have no hypernym hierarchy. For them the path degenerates to lexicographer file then synset. For nouns with multiple inheritance we take the shortest path, and as a variant we average over all paths.

A sense $s$ then corresponds to a path $a_1,\dots,a_K$ of nodes, and the model factorizes its probability as

$$p(s \mid c) = \prod_{k=1}^{K} \frac{\exp\big(h_k^\top e(a_k)\big)}{\sum_{a' \in \mathrm{ch}(a_{k-1})} \exp\big(h_k^\top e(a')\big)}, \qquad h_k = \mathrm{GRU}\big(h_{k-1}, [e(a_{k-1}); r(c)]\big).$$

The symbols are as follows. $e(a)$ is BEM's definition encoder applied to node $a$'s gloss. $r(c)$ is BEM's contextual encoding of the target word. $\mathrm{ch}(a_{k-1})$ is the set of children of the previous node that still lead to a candidate sense. The GRU, a gated recurrent unit or small RNN whose state carries what has been decided so far, starts from $h_0 = r(c)$.

Training maximizes the log-probability of the gold path on SemCor with teacher forcing. Every upper-level step therefore receives supervision from all senses beneath it, which is where the statistical sharing happens. At test time we take the highest-probability full path, found by exact search since local trees are small.

Recurrence is the ingredient under test. After choosing "group" over "institution," the state $h_k$ has absorbed that decision and the definition of "group." The next comparison is then made against a context representation already shifted toward that region of meaning. An additive model scores each level independently of the others.

Research Program

The work has three parts. First, we build the local-tree extraction for all SemCor and evaluation lemmas and measure a coverage statistic for each test instance. This statistic is the depth of the deepest ancestor of the gold sense that has at least $m$ training occurrences anywhere in SemCor, with $m=10$ matching the threshold in the MetricWSD analysis. It gives a per-instance prediction of how much borrowing is possible.

Second, we train the recurrent descent model on BEM's encoders, both from the same BERT-base initialization. We also train an all-LSTM variant with GAS-style bidirectional LSTM encoders, to test whether the effect depends on the pretrained transformer.

Third, we analyze where gains land. We stratify by most-frequent sense, less-frequent sense, and zero-shot sense, and cross this with the ancestor-coverage statistic.

Baselines

The naive reference point is the most-frequent-sense heuristic: WordNet's first sense, or SemCor's most frequent one. It matters because it dominates aggregate metrics. On coarse SemEval-2007 Task 7 the first-sense baseline scores 0.78889 F1 against 0.79550 for the full BLC system in GPLSI. Any claimed gain must therefore be reported on the less-frequent and zero-shot strata, not only overall.

The strongest existing method is BEM, which reports 79.0 F1 on the unified benchmark, 68.9 F1 zero-shot, and 94.1 F1 on most frequent senses. We rerun it from its released recipe on the same SemCor training data, with the same BERT-base encoders, epochs, and SemEval-2007 development set for model selection. We give it the same hyperparameter search budget as our model. Our model beats it if it improves less-frequent-sense F1 and zero-shot F1 on the reproduced run while holding most-frequent-sense F1 within noise, across at least three seeds. The sparse dictionary-coding method of Sparsity Makes Sense reports 78.8 F-score and is a strong non-neural competitor. We compare against its reported number and, where its representations can be recomputed, on our frequency strata. GAS_ext at 70.6 F1 is the recurrent reference and pairs with our all-LSTM variant.

Two ablations isolate the key ingredient, and both keep BEM's encoders and our tree. The additive ablation replaces the GRU state with a sum of independent per-level scores $r(c)^\top e(a_k)$. This is a deep generalization of the Hierarchical Multiclass Perceptron and tests whether recurrence matters beyond hierarchical supervision. The shuffled-hierarchy ablation keeps the same tree shapes but randomly reassigns interior nodes' definitions. It tests whether WordNet's actual structure is responsible, as opposed to the extra parameters and steps.

Why not just…?

Why not just add a supersense score to BEM, as the hierarchical perceptron did? This is the real threat. The Senseval-2 gain from two-level additive scoring was 1.4 points (Hierarchical Semantic Classification), and a modern encoder may capture most of what recurrence adds. The additive ablation settles it directly. If it matches recurrent descent on the less-frequent and zero-shot strata, we will report that the benefit comes from multi-level hierarchical supervision, not recurrence, and the contribution shrinks accordingly.

Why not just concatenate hypernym definitions into BEM's gloss input? GAS_ext did exactly this with LSTMs. Its authors note the hierarchy gets flattened, and it reached 70.6 F1, well below gloss bi-encoders. We run the BERT version of this input augmentation as an additional baseline. It tests whether ancestor text alone, without stepwise decisions, supplies the borrowed evidence.

Why not just back off to the most frequent sense, or learn a better metric space for rare senses? Backoff has nothing to say about rare senses by construction. MetricWSD's episodic prototype training reaches 75.1 F1, below BEM, and still needs examples of a sense to build its prototype. Neither approach reaches zero-shot senses, which is where our coverage prediction is sharpest.

Why not just apply post-hoc sparse coding to frozen vectors? At 78.8 F-score this approach is close to BEM without fine-tuning. It binds dimensions to senses through co-occurrence statistics, however, which are absent for unseen senses. Comparing on the zero-shot stratum shows whether it shares the coverage ceiling identified in A Deep Dive into Word Sense Disambiguation with LSTM.

Decisive Checks

Evaluation uses the unified English all-words benchmark, which aggregates Senseval-2, Senseval-3, SemEval-2007, SemEval-2013 and SemEval-2015, with SemEval-2007 as development. We also report SemEval-2007 Task 7 for coarse-grained comparison with GPLSI. Metrics are F1 overall, on most-frequent and less-frequent senses, and on zero-shot senses. We add path-level accuracy at each depth, which shows whether errors occur at coarse or fine levels.

The central check is a dose-response curve. Relative to BEM, the zero-shot and less-frequent-sense gains should rise with the ancestor-coverage statistic. They should be near zero for senses whose whole subtree is rare, and near zero for adjectives and adverbs, whose paths are trivial. A flat curve would falsify the borrowing mechanism even if aggregate F1 improved. Recurrent descent must also beat the additive ablation on these strata, and the shuffled-hierarchy control must collapse to at most BEM's level. If both conditions hold, the result shows that a recurrent walk over a lexical taxonomy is a concrete way to shrink the long-tail gap. If either fails, the experiments still say cleanly whether hierarchy or recurrence carries the signal.