Evaluation Metrics for the Lexical Substitution Task

Sanaz Jabbari, Mark Hepple, Louise GuthrieNAACL 2010

TL;DR

This paper critiques the evaluation metrics used in the SemEval-2007 English Lexical Substitution (ELS07) task, arguing they don't adequately guide system development. The authors propose new metrics that better assess a system's ability to generate diverse, high-quality word substitutes, offering a more nuanced view of performance beyond just finding the single "best" answer.

What Problem Does This Paper Address?

The English Lexical Substitution (ELS) task challenges systems to replace a target word in a sentence with a suitable synonym, often requiring both word sense disambiguation and synonym generation. The ELS07 shared task used specific evaluation metrics, but the authors of this paper argue these metrics have shortcomings. Specifically, they point out that existing metrics can be too "brittle" (only applicable to certain types of system outputs), can lose valuable information by not crediting partially correct or good-but-not-optimal answers, and don't fully encourage systems to develop robust strategies for identifying a range of good substitutes. This hinders the development of systems that can effectively provide multiple, contextually appropriate alternatives, which is crucial for many real-world applications.

Methods

The paper proposes new evaluation metrics to address the identified shortcomings. For assessing a system's "best guess" answer, they introduce best1(i), which assigns a score of 1 to a guess matching the most frequent human substitute and proportionally less credit for less frequent but still correct substitutes. Crucially, best1(i) applies to all test items, unlike the original "mode" metric which was limited to items with a unique most frequent human answer.

For evaluating a broader set of candidates (the "coverage" task), they propose metrics based on recall and precision, but crucially incorporate the frequency with which human annotators proposed each candidate. This "count-weighting" ensures that substitutes proposed by more annotators are given more importance. They also introduce a weighted recall (R(i)) and a weighted precision (P(i)) that accounts for incorrect answers, and an F-score derived from these.

Datasets

The paper uses the ELS07 dataset for evaluation. This dataset comprises around 2000 sentences, each with a target word that is sense-ambiguous and has at least one synonym. For each target word, five human annotators provided up to three substitutes. The dataset includes nouns, verbs, adjectives, and adverbs. The authors also mention using outputs from several systems (bow, lm, cmlc, and KU) that participated in or were developed around the ELS07 task.

Key Insights and Ideas

  • Beyond the "Best" Answer: The paper strongly advocates for evaluating systems on their ability to provide a range of good substitutes, not just the single most likely one. This is essential for applications where flexibility and multiple options are beneficial.
  • Count-Weighting is Crucial: Simply counting correct answers isn't enough. The frequency with which human annotators agree on a substitute is a strong indicator of its quality and should be incorporated into evaluation metrics.
  • Avoiding Brittleness: New metrics are designed to be applicable across a wider range of system outputs, making them more robust and informative than metrics that only work under specific conditions (like the original "mode" metric).
  • Encouraging Boundary Strategies: The proposed coverage metrics implicitly encourage systems to develop strategies for distinguishing between good and bad candidates, a vital skill for practical utility.

Key Results

The paper doesn't present a single, overarching quantitative result like "System X achieved Y% accuracy." Instead, it focuses on demonstrating the utility of its proposed metrics by applying them to existing system outputs.

  • The best1(i) metric shows that systems performing optimally (returning the most frequent human answer) can score significantly higher with the new metric compared to the old one (e.g., scoring 1 instead of 0.3 for a specific example).
  • The proposed coverage metrics, when applied to system outputs, reveal rankings of systems that align with their underlying candidate ranking quality. For instance, the system KU, known for its strong performance, consistently ranks high across the new metrics.
  • Table 3 shows F-scores for different systems using a simple "top n" boundary strategy, demonstrating how performance varies with this strategy and highlighting the potential for improvement with optimal boundary selection.

Limitations and Future Work

  • Focus on Single Words: The authors explicitly state they only consider the case of substituting for single words and do not address multi-word substitutions.
  • Further Metric Development: While proposing new metrics, the paper acknowledges that further work might be needed. For example, they mention a related task in Jabbari et al. (2010) that directly addresses the ability of systems to achieve good ranking of substitution candidates, suggesting that task-specific performance metrics might be complemented by component-ability metrics.
  • Optimal Boundary Strategy: The paper shows the potential of an "optimal boundary strategy" for coverage but notes that systems are often developed for the "out of ten" (oot) task, not coverage, implying a need for systems to explicitly develop and be evaluated on their boundary-setting capabilities.

Summary updated

BibTeX
@inproceedings{jabbari2010evaluation,
  author = {Sanaz Jabbari and Mark Hepple and Louise Guthrie},
  title = {{Evaluation Metrics for the Lexical Substitution Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure