Information Content Measures of Semantic Similarity Perform Better Without Sense-Tagged Text

Ted PedersenNAACL 2010

TL;DR

This paper demonstrates that semantic similarity measures derived from Information Content (IC) perform best when using a large, unannotated corpus to estimate concept frequencies, rather than relying on smaller, manually sense-tagged corpora. This finding is significant because it suggests that achieving high semantic similarity performance doesn't require the expensive and time-consuming process of creating sense-tagged datasets.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), understanding the semantic similarity between words or concepts is crucial for many tasks like question answering, information retrieval, and machine translation. A common approach uses WordNet, a lexical database that organizes words into synsets (sets of synonyms representing concepts) and defines hierarchical relationships between them. Early methods relied on the "path length" between concepts in this hierarchy, but this can be inaccurate, especially for very specific concepts.

To improve accuracy, researchers proposed augmenting WordNet with "Information Content" (IC). IC quantifies how specific a concept is, with more specific concepts having higher IC. This IC is typically derived from frequency counts of concepts in a corpus of text. The challenge has been determining the best type of corpus to use: should it be a large, manually annotated corpus where each word's sense is explicitly identified (sense-tagged), or a larger, raw corpus where word senses are not explicitly distinguished? Creating sense-tagged corpora is expensive and labor-intensive, making it a bottleneck for research. This paper investigates whether using a large, unannotated corpus can yield better results than a smaller, sense-tagged one.

Methods

The paper compares several Information Content-based semantic similarity measures: Resnik (res), Lin (lin), and Jiang & Conrath (jcn). These measures all use the Information Content of concepts and their least common subsumer (LCS) in WordNet.

The core of the methodology involves deriving Information Content values from different corpora:

  • Sense-tagged SemCor: A manually sense-tagged subset of the Brown Corpus (approx. 226,000 sense-tagged words).
  • Untagged SemCor (semcor-raw): The same SemCor corpus, but treated as plain text, meaning all possible senses of a word contribute to its count.
  • English GigaWord Corpus: A massive collection of newspaper text (over 1.7 billion words). The authors use various subsets of this corpus, including portions of the "xie" (Xinhua News Agency) data, and also combine different sources (afe, apw, nyt).
  • addlonly: A baseline method that assigns a count of 1 to every synset and its ancestors, effectively simulating add-1 smoothing without any text.

The performance of these IC measures is evaluated by calculating their Spearman's rank correlation with human judgments on established semantic similarity/relatedness benchmarks.

Datasets

  • WordNet: Used as the underlying lexical resource for concepts and their hierarchy. The paper uses version 3.0.
  • SemCor: A manually sense-tagged subset of the Brown Corpus, used for deriving IC from sense-tagged text.
  • English GigaWord Corpus (1st edition): A large collection of newspaper text used for deriving IC from unannotated text. Subsets like "xie" (Xinhua News Agency) were specifically utilized.
  • Gold Standards for Evaluation:
    • RG (Rubenstein and Goodenough, 1965): A collection of 65 noun pairs.
  • MC (Miller and Charles, 1991): A collection of 30 noun pairs (a subset of RG).
  • WS (WordSimilarity-353, Finkelstein et al., 2002): A collection of 353 pairs, scored for relatedness rather than strict similarity.

Key Insights and Ideas

  • Coverage Trumps Annotation: The most striking finding is that increasing the coverage of concepts in WordNet (i.e., ensuring more concepts have associated frequency counts) is more important for IC-based similarity measures than having perfectly accurate sense annotations.
  • Unannotated Text is Sufficient (and Better): Using large amounts of unannotated text to derive IC counts leads to better performance than using smaller, sense-tagged corpora. This is because unannotated text, despite its ambiguity, provides counts for a much wider range of concepts.
  • The "addlonly" Baseline: Even a simple baseline that assigns counts to all concepts (simulating add-1 smoothing) performs remarkably well, outperforming the sense-tagged SemCor results and approaching the performance of IC measures derived from large unannotated corpora. This further emphasizes the importance of broad coverage.
  • Implications for Resource Creation: The results suggest that researchers can achieve high-quality semantic similarity without the prohibitive cost of creating large, sense-tagged datasets.

Key Results

  • Information Content measures derived from the untagged version of SemCor (semcor-raw) consistently outperformed those derived from the sense-tagged SemCor (semcor) across all three measures (jcn, lin, res) and evaluation benchmarks (WS, MC, RG).
  • As the size of the unannotated GigaWord corpus subsets increased (e.g., using more months or years of "xie" data), the correlation with human judgments generally improved.
  • The IC measures derived from larger unannotated corpora (like the full "xie" portion of GigaWord) achieved correlations that were competitive with, and sometimes surpassed, established path-based measures (like lesk and vector, as seen in Table 1).
  • The "addlonly" baseline achieved a correlation of 1.00 with the WS benchmark, significantly outperforming the semcor results and demonstrating the power of broad concept coverage.

Limitations and Future Work

  • Relatedness vs. Similarity: The paper notes that the WS benchmark is scored for "relatedness," which is a broader notion than "similarity." While IC measures performed well, other measures like vector and lesk (which are more akin to relatedness measures) performed even better on WS. This suggests a distinction in what these measures capture.
  • Future Work: The authors imply that further exploration into optimizing corpus selection and size for IC derivation could yield even better results. The success of the "addlonly" baseline also opens avenues for research into simpler, yet effective, methods for assigning concept counts. The paper's findings strongly advocate for leveraging large, readily available unannotated text corpora for building robust semantic similarity systems.

Summary updated

BibTeX
@inproceedings{pedersen2010information,
  author = {Ted Pedersen},
  title = {{Information Content Measures of Semantic Similarity Perform Better Without Sense-Tagged Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure