From Global to Local Similarities: A Graph-Based Contextualization Method using Distributional Thesauri

Martin Riedl, Chris BiemannEMNLP 2013

TL;DR

This paper introduces a novel graph-based method to contextualize word similarities, moving beyond global "bag-of-words" approaches. By representing distributional thesauri as graphs and leveraging dependency parses, they generate context-specific word expansions that significantly outperform previous methods on lexical substitution tasks.

What Problem Does This Paper Address?

Traditional methods for understanding word meaning often rely on "distributional semantics," the idea that words appearing in similar contexts have similar meanings. This has led to the creation of "distributional thesauri" (DTs), which list globally similar words for any given term. However, these DTs are static and don't account for how a word's meaning can shift depending on its specific usage in a sentence. For example, "cold" can refer to temperature or illness, and a global DT might not distinguish between these. This paper tackles the challenge of creating a dynamic, context-aware system that can generate relevant word substitutions or expansions based on the immediate linguistic environment.

Methods

The core of the proposed method involves two main stages:

  1. Holing System: This system takes dependency parses of sentences and splits them into pairs of (term, context feature). A "context feature" is essentially the surrounding words and their grammatical relationships, with a placeholder (a "hole") indicating where the target term was. This allows the system to represent both the term and its specific context.
  2. Distributional Similarity and Contextualization:
    • Distributional Similarity (DS): The paper first builds a "distributional thesaurus" (DT) using a graph-based approach. They construct a bipartite graph of terms and context features, then calculate a "significance score" (using Lexicographer's Mutual Information - LMI) for each term-context pair. This is followed by creating a second-order graph that captures the similarity between terms based on shared significant context features. This DS graph represents global similarities.
  3. Contextual Similarity (CS): To contextualize, they take the global DS graph and adapt it to the specific sentence. They identify context features relevant to the target word in the sentence and build a new graph. This graph includes the target word, its globally similar terms (from the DS graph), and the relevant context features. A ranking score is then calculated for each candidate expansion using a harmonic mean of its compatibility with the context features, effectively prioritizing words that fit the specific sentence.

What's novel is the explicit use of dependency parses to define context features and the graph-based representation for both global similarity and contextualization, moving away from purely statistical co-occurrence counts and towards a more structured understanding of linguistic relationships.

Datasets

  • Training/DT Construction: The paper mentions using newspaper corpora (120 million sentences) and the Gigaword corpus for building the distributional thesaurus.
  • Evaluation: The primary dataset used for evaluation is the LexSub dataset from the SemEval 2007 Lexical Substitution task. This dataset consists of trial sentences and official test sentences for evaluating word substitution.

Key Insights and Ideas

  • Context is King: The most significant insight is that global word similarity is insufficient. The meaning and appropriate substitution of a word are heavily dependent on its immediate context, and this method effectively captures that dependency.
  • Dependency Parses as Rich Context: Using dependency parses, rather than just simple word co-occurrence, provides a more structured and informative representation of context, leading to better contextualization.
  • Graph-Based Approach for Scalability: Representing similarities and contexts as graphs, and using techniques like LMI and harmonic mean for ranking, offers a scalable and efficient way to handle large amounts of text and compute contextualized similarities.
  • Unsupervised and Knowledge-Free: The method is unsupervised and doesn't rely on pre-existing lexical resources for candidate generation, making it more generalizable.

Key Results

  • Substantial Improvements: The proposed contextualization method showed substantial improvements over a strong non-contextualized baseline (the DT itself) on the LexSub dataset across all parts of speech.
  • Strong Performance on Verbs: Notably, the largest relative improvements were observed on verbs, which are often challenging for semantic analysis.
  • Coverage: The method demonstrated competitive coverage, reaching up to 55.89% coverage for the top 200 similar words from the DT, indicating its potential to provide a wide range of relevant expansions.
  • Dependency Depth Matters: For adverbs, performance improved significantly when considering targets with at least two or three dependencies, highlighting the importance of richer contextual information.

Limitations and Future Work

  • Adverb Performance: The paper notes a slight decline in precision for adverbs with only one dependency, suggesting that richer contextual information is crucial for this part of speech.
  • Dependency on Parser: The method's performance is inherently tied to the quality of the dependency parser used.
  • Future Work: The authors suggest extending the holing system to incorporate other types of contexts, such as n-gram contexts, and exploring more advanced methods for all-words simultaneous expansion, potentially leading to probabilistic models of lexical expansion. They also aim to adapt more advanced methods for contextualization, such as those that yield all-words simultaneous expansion.

Summary updated

BibTeX
@inproceedings{riedl2013from,
  author = {Martin Riedl and Chris Biemann},
  title = {{From Global to Local Similarities: A Graph-Based Contextualization Method using Distributional Thesauri}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
}
Figure