Is the Low-Resource WSD Gap a Granularity Artifact?

Word sense disambiguation (WSD) means deciding which meaning of a word is intended in context, for example bank as a riverside versus a financial institution. Multilingual models do this far worse outside English. The strongest zero-shot system on the multilingual XL-WSD benchmark is trained only on English and applied directly to other languages, and it falls from the high 70s in English to 47.15 F1 in Basque and 49.97 in Catalan (XL-WSD). Those scores, however, are measured against WordNet's very fine sense distinctions, on which even human annotators agree only about half the time (0.51 agreement, versus 0.81 for a coarse inventory, reported in CSI). It is therefore unclear what the cross-lingual gap actually measures. It may be a failure to carry meaning across languages, or it may be a failure to reproduce lexicographers' hair-splitting in languages where the model has seen little text.

This proposal separates those two explanations. It also asks whether the answer makes useful sense disambiguation possible in languages that have no wordnet at all. We re-score cross-lingual WSD under two coarse inventories. The first is supersenses, the roughly 40 broad WordNet categories such as ARTIFACT, COMMUNICATION, and MOTION. The second is CSI's 45 topical domains, such as LAW&CRIME and FOOD&DRINK. We then break each language's errors into those that cross a coarse boundary and those that stay inside one. Finally, we build a coarse tagger that needs no target-language lexicon and test it on small, newly annotated sets in languages outside BabelNet's coverage.

Core Claim

We claim that the cross-lingual WSD gap should be measured as a relative error ratio, not as a raw F1 difference. Define $r_L = e_L / e_{en}$, where $e_L$ is the error rate in language $L$ and $e_{en}$ is the error rate in English, both computed under the same inventory and on instances that remain ambiguous under that inventory. Coarse-graining lowers error everywhere, so a higher coarse F1 by itself proves nothing.

The substantive question is whether $r_L$ shrinks when we move from fine to coarse labels. If it shrinks, much of the low-resource penalty lies in fine distinctions. In that case a coarse inventory is a legitimate way to deliver usable disambiguation to data-poor languages. If $r_L$ stays flat or grows, the penalty reflects a genuine failure of representation transfer. In that case coarse labels are a cosmetic fix, and effort should go elsewhere, for example into the alignment work of Isomorphic Cross-lingual Embeddings for Low-Resource Languages.

Our prior leans toward partial shrinkage, concentrated in nouns. Pretrained models already separate unrelated meanings almost perfectly in English, reaching 95.24% zero-shot accuracy on homonymy (Analyzing Homonymy Disambiguation). Verbs and abstract categories are already weak in English, at 57.0 F1 for verbs under zero-shot domain prompting (What do Language Models know about word senses?), and agreement on COGNITION was only 49% among Arabic annotators (Coarse Lexical Semantic Annotation with Supersenses). Either outcome is informative.

Relation to Prior Work

Two systems attack multilingual WSD directly, and both stay with the fine-grained BabelNet inventory. XL-WSD builds the 18-language benchmark and compares zero-shot English transfer with training on machine-translated "silver" data. Translate to Disambiguate prompts large models to translate the ambiguous word and then intersects BabelNet senses. When a translation is missing from BabelNet, it falls back to the most frequent sense. Neither system asks how much of the gap survives coarse-graining, and neither works where no BabelNet entries exist.

Our work also differs from the existing coarse-inventory studies. CSI shows that a coarse inventory raises English WSD to 85.9 F1 and suggests cross-lingual use, but does not test it. Ciaramita and Altun's supersense tagger frames coarse disambiguation as sequence labeling, but it relies on WordNet first-sense features, which a lexicon-free language cannot supply. The Arabic supersense study shows that native speakers can annotate coarse labels directly, about 65,000 tokens in roughly 100 hours. It does not attempt automatic transfer. We reuse its annotation guidelines as our method for building test sets.

Towards Multi-Sense Cross-Lingual Alignment is the closest work on the representation side. It learns unsupervised sense clusters and aligns them across languages with dictionaries, but only for high- and medium-resource pairs. Its clusters are not tied to any inventory, so they cannot be scored against human labels. We treat its approach as a competitor rather than as a component.

PerSemCor and the findings of Duh et al. on distribution shift under translation act as constraints on our design. Annotation projection needs a target wordnet linked to English, and translated text is distributionally unlike native text. Both facts argue against a translate-and-project pipeline as our main route.

Research Program

The first step requires no new annotation. XL-WSD gold labels are BabelNet synsets linked to WordNet 3.0. Each gold synset maps deterministically to its supersense through WordNet's lexicographer files, and to a CSI domain through CSI's mapping of 83,000 synsets. This yields coarse gold labels for all 18 XL-WSD languages.

Scoring needs care. Collapsing senses makes many lemmas unambiguous, which inflates scores trivially. We therefore report results on the subset of instances whose candidate senses still span at least two coarse classes, and we report what fraction of each language falls into that subset. On this subset we compute $r_L$ for fine, supersense, and CSI granularity. We also decompose each fine-grained error as either within-class or cross-class, separately for each part of speech.

The second step is a tagger that needs no lexicon. We fine-tune a multilingual encoder, XLM-R Large, to be comparable with XL-WSD. The model receives a sentence and a marked content word and outputs one of the coarse labels. It sees no candidate list, because a language without a wordnet cannot provide one. Training uses English SemCor with gold labels collapsed to coarse classes.

We test two target-language variants. One is pure zero-shot. The other adds a small in-language coarse annotation set, which costs a fixed number of annotator hours. We express this budget in hours rather than examples, because coarse labels are cheaper to produce than fine ones, and this cost difference is the practical advantage we want to measure.

The third step builds evaluation sets outside BabelNet's comfort zone. Following the Arabic guidelines, extended to the 15 verb supersenses, native annotators will label noun and verb supersenses on about 500 sentences each of Nepali and Uyghur. Scaling from the Arabic effort, we estimate 20–30 hours per language. We will double-annotate a fraction of the data to report inter-annotator agreement. That agreement figure is the ceiling against which system scores should be read.

Uyghur also carries the tokenizer fragmentation problem described in Vocabulary Expansion for Low-resource Cross-lingual Transfer. We will report its results separately so that tokenization effects are not mistaken for sense effects.

Baselines

The naive reference point is a most-frequent-coarse-class predictor. For XL-WSD languages it uses the coarse class of each lemma's first BabelNet sense. For Nepali and Uyghur, which have no lexicon, it predicts the overall majority class for each part of speech. The first-sense prior is the right floor because it is notoriously hard to beat. Ciaramita and Altun's first-sense baseline reached 66.47% on SemCor and 64.09% on Senseval-3, against 77.18% and 70.54% for their tagger (Broad-Coverage Sense Disambiguation). A system that fails to clear this floor on the coarse-ambiguous subset counts as a failure.

The strongest existing method is fine-grained XLM-R Large trained on English SemCor and applied zero-shot, as in XL-WSD. XL-WSD reports 65.66 average non-English F1 for this model, with 47.15 for Basque and 49.97 for Catalan. We run it as a coarse baseline by mapping its fine predictions to coarse classes after the fact. We found no result on Lacuna for XL-WSD scored under supersense or CSI labels, so we will reproduce this baseline ourselves. It uses the same encoder, the same English training data, and the same hyperparameter search budget as our tagger. Our trained-coarse tagger must exceed it on the coarse-ambiguous subset, averaged over languages, by more than the variation across seeds. Otherwise the inventory adds nothing at training time.

Translate to Disambiguate is the strongest zero-shot competitor that avoids fine-tuning. Its paper reports beating supervised baselines on recall in 11 of 18 languages, but we found no per-language F1 on Lacuna for C-WLT. We will rerun it with the largest openly available model in the family the paper used, collapse its outputs to coarse classes, and log how often it falls back because a BabelNet entry is missing. It cannot run at all on Nepali or Uyghur words that lack BabelNet entries. We will report that coverage failure rather than impute a score.

Zero-shot NLI domain prompting serves as a second training-free baseline for the CSI condition, run as in What do Language Models know about word senses?. That paper reports 86.8 F1 on adjectives and 57.0 F1 on verbs in English.

Two ablations isolate the key ingredient. The first restricts our tagger's output to candidate coarse classes from BabelNet. This measures how much we lose by doing without a lexicon. The second replaces the in-language coarse annotations with an equal number of annotator-hours of fine-grained annotation. We can run it only for XL-WSD languages with gold fine labels, where we simulate the time cost using the agreement-adjusted annotation rates reported in the cited studies. This measures whether coarse supervision is worth more per hour.

Why not just…?

Why not just machine-translate SemCor and train on silver data? XL-WSD already found that zero-shot English transfer often beats language-specific silver training. PerSemCor-style projection also needs a target wordnet linked to English, which Nepali and Uyghur lack in usable form. We still include coarse-collapsed silver training for the XL-WSD languages, so that the comparison is direct rather than assumed.

Why not just collapse a fine-grained model's predictions after the fact? This is the real threat. If post-hoc collapsing matches coarse training, then the "method" half of this proposal reduces to the diagnostic half. We do not argue around this. The trained-versus-collapsed comparison is our headline ablation, and a tie would be reported as a finding. Two things could break a tie. The first is coarse supervision in the target language, which the fine model cannot use. The second is the lexicon-free setting, where a fine model has no candidate senses to choose among.

Why not just prompt a large model to translate the word (C-WLT)? It is strong where BabelNet coverage is good, and our rerun may show it winning on Spanish-like languages. However, it is defined in terms of BabelNet entries. The missing-translation fallback that its authors describe becomes the common case in our new languages, and our coverage logs will quantify this.

Why not just learn unsupervised sense clusters, as in multi-sense alignment? Unsupervised clusters cannot be scored against human labels or reused as a shared tag set. The low-resource alignment literature also shows that small target corpora give unstable vector spaces: VecMap reaches 54.8% bilingual-lexicon precision and an eigenvalue dissimilarity of 156.8 on low-resource pairs (Isomorphic Cross-lingual Embeddings). We therefore do not expect clusters induced from Nepali or Uyghur text to be reliable. We leave this as an argument rather than a run, because the competing system cannot be evaluated against our labels.

Decisive Checks

The first check is the fine-to-coarse change in $r_L$, broken down by language and part of speech, with bootstrap confidence intervals. A reduction in the median ratio across XL-WSD languages supports the granularity-artifact reading. A flat ratio refutes it. We expect nouns and verbs to diverge, and we will report them separately.

The second check is the error decomposition. If most of Basque's extra errors relative to English cross supersense boundaries, the gap is representational. The third check is whether supersenses or CSI domains give the lower ratio. Topical domains may transfer better because surrounding content words cue them, whereas supersenses depend more on the word itself. The fourth check is the trained-coarse versus post-hoc-collapsed comparison on the coarse-ambiguous subset. The fifth is Nepali and Uyghur accuracy measured against their own inter-annotator agreement, together with the hours-matched comparison of coarse and fine supervision.

A positive result would establish coarse inventories as a measured and reusable option for lexicon-free languages. A negative result would retire the "just use coarser labels" intuition with evidence.