A Comparison of Knowledge-based Algorithms for Graded Word Sense Assignment

Annemarie Friedrich, Nikos Engonopoulos, Stefan Thater, Manfred PinkalCOLING 2012

TL;DR

This paper explores how well existing knowledge-based word sense disambiguation (WSD) algorithms can be adapted for "graded" word sense assignment (GWSA), where a word can evoke multiple senses to varying degrees. They found that while these unsupervised methods show promise and correlate with human judgments, their performance differs significantly, suggesting that standard WSD evaluation metrics might not fully capture GWSA capabilities.

What Problem Does This Paper Address?

Traditional Word Sense Disambiguation (WSD) assumes a word in context maps to exactly one sense from a predefined list. This often oversimplifies word meaning, especially when a word genuinely evokes multiple senses simultaneously, each to a different extent. This is particularly true for fine-grained sense inventories like WordNet. Graded Word Sense Assignment (GWSA) offers a more natural way to represent word meaning by allowing multiple senses to apply with varying degrees of applicability. However, evaluating systems for this more nuanced task requires new approaches beyond simple classification. This paper investigates how well existing knowledge-based WSD systems, which don't require labeled training data, can be adapted to this GWSA task and proposes a new evaluation metric to better assess their performance.

Methods

The authors adapt three knowledge-based WSD algorithms for GWSA:

  • Vector Space-based WSD (VSM): This method represents words as vectors in a high-dimensional space based on co-occurrence statistics with other words in a large corpus. For GWSA, it computes contextualized vectors for the target word and compares them to vectors derived from WordNet sense paraphrases, using the similarity scores to rank senses.
  • Topic Model-based WSD: This approach uses Latent Dirichlet Allocation (LDA) to model text corpora as mixtures of latent topics. It estimates topic distributions for the context of a target word and for its sense paraphrases, using the cosine similarity between these topic distributions as a score for ranking senses.
  • Graph-based WSD (PageRank): This method constructs a graph where nodes represent word senses and edges represent relationships between them (e.g., based on semantic similarity of content words in the context). It then applies the PageRank algorithm to score the nodes, and these scores are used to rank the senses.

Crucially, these methods are "knowledge-lean," relying primarily on semantic lexicons like WordNet and large unlabeled text corpora, making them unsupervised.

Datasets

The paper evaluates the proposed methods on two GWSA datasets:

  • WSsim-1: Introduced by Erk and McCarthy (2009), this dataset consists of 430 sentences for 11 lemmas, annotated by three human annotators on a scale of 1-5 for sense applicability.
  • WSsim-2: Introduced by Erk et al. (2012), this dataset contains 260 sentences for 26 lemmas, with similar annotation procedures to WSsim-1.

These datasets differ from traditional WSD datasets by providing graded applicability scores rather than a single sense assignment.

Key Insights and Ideas

  • GWSA is a valuable lens for WSD evaluation: The paper argues that GWSA, with its graded annotations, offers a more context-sensitive and nuanced way to evaluate WSD systems, revealing differences in their ranking behavior that standard classification metrics might miss.
  • Knowledge-based systems show promise for GWSA: Despite being unsupervised, the tested knowledge-based systems demonstrate positive correlations with human judgments for graded sense assignment, indicating their potential for this task.
  • Different WSD approaches have distinct strengths for GWSA: The VSM approach, which focuses on contextualized word meaning, appears to be better at capturing subtle distinctions between senses compared to graph-based or topic-model-based methods.
  • Standard WSD success doesn't guarantee GWSA success: The PageRank-based system, which performs well on traditional WSD tasks, shows a different ranking behavior in GWSA, suggesting that algorithms optimized for picking a single "best" sense might not be as effective at ranking multiple applicable senses.
  • Human annotators exhibit varying agreement: The analysis of human judgments reveals that annotators agree more on extreme applicability scores (very applicable or not applicable) and tend to use intermediate scores more interchangeably.

Key Results

  • All three knowledge-based systems showed positive correlations with human judgments on both WSsim-1 and WSsim-2 datasets, as measured by Spearman's rank correlation coefficient (ρ).
  • The VSM system generally outperformed the Topic Models and PageRank systems across various metrics, including correlation and the newly proposed Adjusted Accuracy.
  • The VSM system achieved an Adjusted Accuracy of up to 76% (for a threshold of t=2) on WSsim-2, while the PageRank system showed a sharp increase in performance only at a high threshold (t=5), suggesting it's better suited for standard WSD.
  • The paper introduces "Adjusted Accuracy" as a new metric that accounts for chance agreement, providing a more robust evaluation of system performance on GWSA by treating the graded annotations as a gold standard for coarse-grained WSD.

Limitations and Future Work

  • Human annotator agreement: The paper acknowledges that human annotator agreement is not perfect, making it challenging to definitively quantify the "actual degree of applicability."
  • System tuning: The authors note that most algorithms were not specifically tuned for the GWSA task, with the exception of setting parameters for the PageRank algorithm. This suggests that further tuning could potentially improve performance.
  • Supervised vs. Unsupervised: While the knowledge-based systems show promise, they do not reach the performance of the supervised approach presented by Erk and McCarthy (2009). Future work could explore ways to bridge this gap or further improve unsupervised methods.
  • New evaluation paradigm: The paper advocates for the use of GWSA data with correlation and accuracy analyses as a way to gain deeper insights into WSD system performance beyond traditional classification, suggesting this approach could be valuable for future WSD research.

Summary updated

BibTeX
@inproceedings{friedrich2012comparison,
  author = {Annemarie Friedrich and Nikos Engonopoulos and Stefan Thater and Manfred Pinkal},
  title = {{A Comparison of Knowledge-based Algorithms for Graded Word Sense Assignment}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure