A Semantic Network Approach to Measuring Relatedness

Brian HarringtonCOLING 2010

TL;DR

This paper introduces ASKNet, a novel system that automatically builds a large-scale semantic network from natural language text. By using spreading activation on this network, it can measure the relatedness between words with accuracy comparable to state-of-the-art systems, even when using a relatively small web corpus. This approach offers a more human-like way to understand word relationships and demonstrates the power of general-purpose information integration tools.

What Problem Does This Paper Address?

Determining how closely two words are related (semantic relatedness) is a fundamental task in Natural Language Processing (NLP) with applications in areas like word sense disambiguation, information retrieval, and text summarization. While semantic similarity (e.g., cars and bicycles) is a subset of relatedness, many applications require a broader understanding of how concepts connect (e.g., money and bank). Existing automated methods for measuring relatedness typically rely on either the structure of curated lexical resources like WordNet or co-occurrence statistics from massive text corpora. However, WordNet's hierarchical structure can sometimes misrepresent relatedness, and building semantic networks from extremely large corpora becomes computationally prohibitive, leading to diminishing returns. This paper aims to bridge this gap by proposing an automated approach that creates a rich semantic network from readily available text and uses it to measure relatedness effectively.

Methods

The core of the proposed approach is the ASKNet system, which automatically constructs semantic networks. ASKNet leverages existing NLP tools, specifically the C&C parser and the Boxer semantic analysis program, to extract syntactic and semantic information from text. This information is then integrated into a large-scale network using a modified spreading activation update algorithm. This algorithm maps named entities and general objects from incoming text fragments to existing nodes in the network, iteratively refining mapping scores based on semantic information within the network.

Once the semantic network is built, spreading activation is employed to measure semantic relatedness between two terms. Activation is initiated at one term's node and allowed to propagate through the network. The amount of activation received by the target term's node serves as a measure of their relatedness. This process is performed in both directions (term A to term B, and term B to term A), and the sum of the activation received is used as the final relatedness score. This method is designed to capture not just direct connections but also indirect relationships through multiple paths in the network.

Datasets

The paper evaluates its approach using three established datasets of human judgments on semantic relatedness:

  • Hodgson dataset: Consists of 65 word pairs, categorized by lexical relations (e.g., antonymy, synonymy). No explicit scores were provided, so the paper used unrelated pairs generated by McDonald and Brew (2004) for comparison.
  • RG (Rubenstein and Goodenough) dataset: Contains 65 word pairs, with human judges rating them based on synonymy.
  • ws-353 (Wordsimilarity-353) dataset: A larger dataset of 353 word pairs, where human judges ranked relatedness on a scale of 1 to 10.

For each dataset, the authors created a custom corpus by querying Google for each word in the word pairs and collecting the top 5 search results. This approach aimed to create a corpus that was representative of general language use and required minimal human intervention.

Key Insights and Ideas

  • Human-like Relatedness: The spreading activation mechanism on a semantic network is presented as being more analogous to how humans intuitively understand word relationships, moving beyond purely statistical or hierarchical methods.
  • General-Purpose Tool: ASKNet is designed as a general tool for information collection and integration, not specifically for semantic relatedness tasks. The fact that it performs well on this task highlights its versatility and potential for broader NLP applications.
  • Scalability Advantage: By building networks from web corpora, the approach circumvents the limitations of manually curated resources like WordNet and the computational burden of processing extremely large, pre-existing corpora.
  • Indirect Relationships Matter: The method effectively captures relatedness through indirect paths in the network, which is crucial for understanding nuanced connections between concepts that might not be directly linked in simpler models.

Key Results

The paper reports strong performance across the evaluated datasets:

  • Hodgson dataset: An ANOVA test showed a significant effect for relatedness, with the network scores for related words being significantly higher than for unrelated words. The effect size (η²) was 0.411, indicating that approximately 41% of the variance in scores could be explained by the relatedness of the word pairs.
  • ws-353 and RG datasets: Rank correlation scores (Spearman's rank correlation) showed that the ASKNet approach achieved results on par with the best performing existing systems. For the ws-353 dataset, the correlation was 0.62, and for the RG dataset, it was 0.86. This performance was achieved despite using a significantly smaller corpus compared to some of the top-performing systems.
  • Efficiency: The largest network, built for the ws-353 dataset, took just over 24 hours to complete, including parsing and semantic analysis, demonstrating the feasibility of automated network creation.

Limitations and Future Work

  • Lower Precision than Manual Creation: The authors acknowledge that automatically creating networks results in lower precision compared to manual creation, but this is offset by the significant gains in scalability and speed.
  • Unrelated Pair Generation: For the Hodgson dataset, unrelated pairs were generated, and the authors note that some of these might have had subtle relationships, which could affect the strictness of the comparison.
  • Future Work: While not explicitly detailed, the success of ASKNet as a general information integration tool suggests potential for its application in a wider range of NLP tasks beyond semantic relatedness. The paper implies that further exploration of its capabilities in other domains would be valuable.

Summary updated

BibTeX
@inproceedings{harrington2010semantic,
  author = {Brian Harrington},
  title = {{A Semantic Network Approach to Measuring Relatedness}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
}
Figure