ConSeC: Word Sense Disambiguation as Continuous Sense Comprehension

Edoardo Barba, Luigi Procopio, Roberto NavigliEMNLP 2021 · DOI

Word Sense Disambiguation (WSD) is the fundamental task of determining the correct meaning of a word in a specific context. While modern systems have moved toward using Transformer-based architectures, they often treat the disambiguation of each word as an independent classification event. This paper, presented at EMNLP 2021, introduces ConSeC (CONtinuous SEnse Comprehension), a model that breaks this "independence assumption" by framing WSD as an iterative comprehension process.

The Problem: The Independence Assumption

Traditional WSD systems assume that the latent context provided by a Transformer (like BERT) is sufficient to distinguish between senses. However, humans often rely on the explicit meanings of surrounding words to narrow down possibilities. For instance, knowing that "crane" refers to a bird in a sentence makes it significantly easier to disambiguate a nearby word like "marsh." Furthermore, treating senses as arbitrary ID labels makes it difficult for models to handle rare or unseen meanings.

Methods: Continuous Sense Comprehension

ConSeC introduces three primary innovations to address these limitations:

  1. WSD as Text Extraction: Instead of classifying a word into a fixed category, ConSeC is presented with a target word and a list of candidate definitions (glosses). The model must "extract" the span corresponding to the correct definition, forcing it to actually "read" and understand the semantic content of the senses.
  2. The Feedback Loop: Disambiguation occurs in a specific order, typically from the least ambiguous words to the most polysemous. Once a word is disambiguated, its chosen definition is fed back into the model as explicit context for the next word in the sequence.
  3. Relative Position Overriding: To integrate these definitions without disrupting the original sentence flow, the authors manipulate the positional encoding matrix (using the DeBERTa architecture). This allows a target word to "see" the definitions of its neighbors as if they were adjacent in the text.
Illustration of the relative position overriding mechanism and context definitions.

To prevent the model from being overwhelmed by too many definitions, Normalized Pointwise Mutual Information (NPMI) is used to select only the most relevant nearby words to provide context definitions.

Key Results

ConSeC was evaluated on standard English benchmarks and a recent cross-lingual benchmark covering eight languages.

  • State-of-the-Art Performance: ConSeC achieved an 82.0 F1 score on the English "ALL" benchmark when trained on SemCor, outperforming the previous state-of-the-art (ESCHER) by a statistically significant 1.3 points.
  • Data Augmentation: When supplemented with WordNet glosses and examples (WNGE), the model's performance reached 83.2 F1.
  • Cross-Lingual Success: The model set new records across all eight tested languages, demonstrating the robustness of the "comprehension" approach across different linguistic structures.
Performance comparison of ConSeC against state-of-the-art WSD models across standard benchmarks.

Analysis and Insights

The success of ConSeC highlights that the "independence assumption" serves as a performance ceiling for WSD. Explicitly providing definitions of nearby words yields a massive boost, suggesting that latent context alone does not capture all necessary semantic nuances.

Additionally, ConSeC performs exceptionally well on Least Frequent Senses (LFS). Because the model reads definitions rather than memorizing labels, it generalizes better to rare meanings that appear infrequently in training data. Even under "adversarial" conditions—where the feedback loop is provided with incorrect definitions—the model remains robust, weighing the sentence context against the provided definitions.

Limitations and Future Work

While highly effective, the iterative nature of the feedback loop makes ConSeC slower than parallel disambiguation models. Future work may focus on making the dependency selection process fully learnable or developing a single, massively multilingual model to replace the current monolingual approach.

Summary updated

BibTeX
@inproceedings{barba2021consec,
  author = {Edoardo Barba and Luigi Procopio and Roberto Navigli},
  title = {{ConSeC: Word Sense Disambiguation as Continuous Sense Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.112},
}
Figure