Two-View Pseudo-Labeling for Rare Word Senses

Word sense disambiguation (WSD) is the task of deciding which meaning of an ambiguous word is intended, for example whether bank means a lender or a riverside. Good systems need sense-labeled text, and labeled text is scarce. The largest hand-labeled corpus, SemCor, has about 226,000 annotations. One estimate puts the human effort needed for a robust, domain-independent supervised system at 27 person-years (Ensemble Methods for Unsupervised WSD). Unlabeled text costs almost nothing, so the obvious move is to have a model label raw text and then train on its own guesses. In WSD this backfires in a predictable way. Word senses follow a steep power law, so models already default to the most frequent sense. On hard instances, current systems predict the most frequent sense about 63% of the time even though it is correct only 2% of the time (Nibbling at the Hard Core of Word Sense Disambiguation). Self-generated labels mostly reinforce that habit.

We propose to use unlabeled text for one purpose only: collecting trustworthy examples of the rare and never-seen senses that supervised models get wrong. Each candidate label must be confirmed by two judges that fail in different ways. The first is a neural model that matches the sentence against dictionary definitions; it has learned frequency biases from SemCor. The second is a fixed random walk over the WordNet graph that has no access to sense frequencies at all. We accept a label only when both judges agree, and we cap how many labels each sense can receive so the common senses cannot flood the new training data. The bet is that what limits semi-supervised WSD is which senses the new labels cover, and how reliable those labels are, rather than how many there are.

Core Claim

A pseudo-label for a rare sense is worth far more than one for a frequent sense, and it can be trusted only if a second source of evidence with uncorrelated errors confirms it. A knowledge-graph random walk is a good second source for three reasons. It is weak overall, but it is not biased toward frequent senses. It needs no training. Because it never updates, it cannot drift along with the learner. That last property is what should prevent the rise-then-crash learning curves seen in classical WSD bootstrapping. We predict that this procedure improves the rare-sense and zero-shot challenge sets (42D, hardEN) without losing accuracy on the standard all-words benchmark, whereas plain confidence-based self-training helps neither.

Relation to Prior Work

Four strands of prior work matter here. The first motivates the proposal by showing the gap. Supervised context-definition matchers are the strongest systems in the file: GlossBERT pairs a sentence with a candidate definition and classifies the pair, reaching 77.0 F1 on the concatenated benchmark. Bert_def encodes contexts and definitions separately and matches them, reaching 76.3 F1 and 90.15 F1 on words absent from training. Both are trained only on labeled data. The 54.1 micro-F1 that the best system reaches on 42D, and the 0.0 that all systems score on hardEN by construction (Nibbling at the Hard Core), show that definition matching alone does not solve rare senses. We reuse the matcher as one of our two judges.

The second strand constrains the design. Mihalcea's co-training and self-training study trained Naive Bayes classifiers on two hand-split feature views, local and topical, and let them label raw British National Corpus (BNC) text for each other. It found three problems: unstable learning curves, drift on highly polysemous words, and no gain once the starting classifier exceeded about 75% accuracy. Its two views both learn from the same labels, so their errors are correlated. Our second judge is a non-learned graph, which addresses exactly that. Mihalcea's ceiling for strong starting classifiers is the main threat to this proposal, and the experiments below are built to test it.

The third strand works on semi-supervised graphs for WSD. The example similarity graph approach links labeled and unlabeled sentences by contextual similarity and propagates labels between them. It reached only 73.09 precision on the Senseval-2 English lexical sample, which the authors attribute to the small unlabeled pool. It also never uses the lexical network itself. MuLaN propagates English labels into other languages through multilingual embeddings and uses BabelNet only as a filter afterward. It explicitly cannot produce senses missing from its labeled source. Our proposal differs from both: it is monolingual, it targets senses absent from or rare in SemCor, and it uses the knowledge graph as a voting judge rather than a filter.

The fourth strand supplies the second judge. Personalized PageRank over WordNet lets the context words inject probability into the graph and picks the sense that accumulates the most. Its word-to-word variant reaches 57.9 F1 on Senseval-3. Adding the 551,551 edges of eXtended WordNet, which link concepts through the words in their definitions, matters a great deal (Random Walks for Knowledge-Based WSD). We use it unchanged.

Research Program

We start from a definition-matching model trained on SemCor and run it over every content-word occurrence in the BNC. For each occurrence $x$ of a lemma with candidate senses, the matcher gives a distribution $p_g(s \mid x)$. Personalized PageRank, run with uniform priors over senses, gives a normalized stationary distribution $p_k(s \mid x)$. Let $m_g(x)$ and $m_k(x)$ be each judge's margin, meaning the gap between its top two senses. Let $n_s$ be the number of SemCor examples of sense $s$. The occurrence $x$ is accepted with label $s$ when

$$ s = \arg\max p_g(\cdot \mid x) = \arg\max p_k(\cdot \mid x), \quad m_g(x) > \tau_g, \quad m_k(x) > \tau_k, \quad |A_s| < K, $$

where $A_s$ is the set already accepted for $s$ and $K$ is a per-sense cap. Candidates are processed in increasing order of $n_s$, so senses with few or no SemCor examples fill their quotas first. The most frequent sense of each lemma is held to a much smaller cap. We then retrain the matcher on SemCor plus the accepted set and repeat for two or three rounds. The graph judge stays fixed throughout.

Two analyses accompany the system. The first is a hand audit of accepted labels, stratified by SemCor frequency band, to measure label precision. The second measures how correlated the two judges' errors are on labeled held-out data, again within each frequency band. The method rests on that correlation being low for rare senses.

Baselines

The reference points with no learning are the most frequent sense heuristic and Personalized PageRank alone. The most frequent sense reaches roughly 75% on ambiguous words (The Most Frequent Sense Bias) and should score near zero on hardEN. Personalized PageRank alone scores 57.9 F1 on Senseval-3. These two show what each ingredient achieves without the other.

The strongest supervised competitor is GlossBERT. It reports 77.0 F1 on the concatenated all-words test set. For rare senses, the strongest reported figure is ESCHER at 54.1 micro-F1 on 42D (Nibbling at the Hard Core). We retrain GlossBERT with the same BERT-base backbone, the same SemCor training data, and the same hyperparameter search budget as our system. Our system adds only the pseudo-labeled data. Beating it means a statistically reliable gain on 42D and hardEN with no more than a few tenths of a point lost on the concatenated set. A gain on the concatenated set alone, without rare-sense gains, would not count as support for the claim.

The strongest semi-supervised competitor is single-judge confidence self-training. This is the modern counterpart of Mihalcea: the same matcher, the same BNC pool, the same number of rounds, and a confidence threshold tuned to accept the same total number of labels as our method. We also report Mihalcea's smoothed co-training with its hand-split views on the Senseval-2 noun lexical sample. Its reported precision there is 58.35%, against 53.84% for its base classifier.

Three ablations separate the ingredients. The first keeps two-judge agreement but removes the per-sense caps and the rare-first ordering. The second keeps caps and ordering but uses a single judge. The third replaces the graph judge with a second matcher trained on a different random seed, a correlated judge, to test whether the graph's independence is what matters. A final control trains on SemCor with a loss that discounts frequent senses, using no unlabeled data, to check whether the gain comes from the new data or from simply reweighting the existing labels.

Why not just…?

Why not just reweight rare senses during supervised training? This is the most serious competitor. Counteracting the frequent-sense bias might be enough without any new data. It cannot help senses with zero SemCor examples, since there is nothing to upweight, but it could match our method on rare-but-seen senses. The frequency-discounted control is run under the same budget, and results are reported separately for unseen and rare-but-seen senses. If it matches us in both bands, the claim fails.

Why not just add the WordNet definitions and example sentences as training data? This also targets rare senses directly, and MuLaN already trains on SemCor plus the WordNet gloss corpus (MuLaN). We therefore include those resources in every system's base training data, so any gain we report is measured on top of them. Definition sentences are short and written in an unusual style, while BNC occurrences are natural running text, so we expect them to cover different ground. That expectation is tested rather than assumed.

Why not just ask a large language model to label the raw text? The file's evidence argues against using one as the sole labeler. The best model in RoDEval correctly disambiguates only 58.2% of polysemous words and is confidently wrong about 25% of the time. GPT-4o scores 66.9 F-B³ on SemCor-WSI, below a trivial baseline that puts every occurrence of a word in one cluster, which scores 73.6 (In the LLM era, WSI remains unsolved). A language model could still serve as the second judge. Its errors are probably less correlated with the matcher's than a second seed would be, but its pretraining likely carries the same frequency bias. We run it as a further replacement judge if compute allows, and the error-correlation analysis decides which judge is better.

Why not just use plain self-training? Mihalcea's curves crashed and gave nothing once the starting classifier was strong. The single-judge baseline at a matched label count directly tests whether that negative result still holds with modern encoders.

Proposed Experiments

The main evaluation trains on SemCor, draws unlabeled text from the BNC, and tests on the five unified all-words datasets and their concatenation (Unified Evaluation Framework). We also report the cleaned ALL_NEW version, 42D, and hardEN. All results are broken down by part of speech, with verbs reported separately because verbs are where every system is weakest. They are also broken down into three SemCor frequency bands: unseen, rare, and most frequent sense. A low-resource simulation subsamples SemCor to 10% and 30% to test whether the method's advantage grows as labels get scarcer, as the acquisition-bottleneck argument predicts. A lexical-sample comparison on Senseval-2 lets us compare directly with Mihalcea's reported numbers and with the 73.09 reported by the example-graph method.

Three checks decide whether the idea holds. First, the hand audit must show that accepted rare-sense labels are substantially more precise than labels from single-judge self-training at the same volume. If they are not, the agreement filter adds nothing. Second, on held-out labeled data, the matcher's and the graph's errors must be measurably less correlated in the rare band than the errors of two matcher seeds. If they are equally correlated, the independence argument fails, and Mihalcea's limits should apply to us too. Third, gains must appear on 42D and on the unseen-sense band, beyond what the frequency-discounted control achieves. If the gains appear only on the concatenated set, the method is just more training data rather than a solution to the rare-sense problem.