An Open-source Framework for Multi-level Semantic Similarity Measurement

Mohammad Taher Pilehvar, Roberto NavigliNAACL 2015

TL;DR

This paper introduces ADW, an open-source Java framework for measuring semantic similarity across various linguistic levels, from word senses to full texts. It leverages a novel alignment-based disambiguation technique combined with Personalized PageRank on WordNet, offering a unified, state-of-the-art approach that is both comprehensive and efficient.

What Problem Does This Paper Address?

Measuring semantic similarity – how alike two pieces of text are in meaning – is fundamental to many Natural Language Processing (NLP) tasks, including information retrieval, question answering, and machine translation evaluation. While numerous methods exist, they often fall into two camps: corpus-based (relying on statistical patterns in large text collections) or knowledge-based (using structured knowledge like WordNet). Existing knowledge-based methods often struggle with applying to diverse linguistic items (words, phrases, sentences) uniformly, require extensive tuning, or are limited to comparing items of the same type (e.g., only word senses). This paper addresses the need for a single, robust, and versatile framework that can handle multi-level semantic similarity measurement effectively.

Methods

The core of the ADW framework is a two-phase approach:

  1. Alignment-Based Disambiguation: Before measuring similarity, ADW disambiguates linguistic items. For a pair of items, it uses an alignment technique to determine the most likely sense for each word in the context of the other item. This is achieved by measuring the similarity of each potential sense of a word to all senses of words in the other item, selecting the sense that yields maximal similarity. This process is repeated for all words in both items.
  2. Personalized PageRank (PPR) on WordNet: After disambiguation, ADW models the linguistic items as high-dimensional "semantic signatures." This is done by applying the PPR algorithm on the WordNet graph. The algorithm is initialized from the disambiguated senses of the words in the item. The resulting stationary distribution of PPR, where dimensions correspond to WordNet synsets, forms the semantic signature.

Finally, the similarity between two items is computed as the similarity of their respective semantic signatures. The framework offers four signature comparison techniques: Jensen-Shannon divergence, Kullback-Leibler divergence, cosine similarity, and Weighted Overlap.

To enhance efficiency, especially for longer texts, ADW pre-computes semantic signatures for individual WordNet synsets and words. It also provides a method for offline computation of signatures for textual items, avoiding repeated PPR calculations.

Datasets

The paper primarily uses WordNet 3.0 as its knowledge base. For evaluation, it employs two standard benchmarks:

  • RG-65 dataset: Used for similarity judgment correlation, measuring how well ADW's similarity scores correlate with human judgments.
  • TOEFL dataset: Used for synonym recognition, evaluating ADW's ability to identify correct synonyms from a list of candidates.

The authors also mention the use of the Princeton Annotated Gloss Corpus to enrich the WordNet graph.

Key Insights and Ideas

  • Unified Multi-Level Measurement: The most significant contribution is the ability to measure semantic similarity across different linguistic granularities (word senses, words, phrases, sentences) using a single, consistent framework. This overcomes the limitations of methods that are specialized for only one level.
  • Efficiency through Pre-computation: Pre-computing semantic signatures for WordNet synsets and words drastically speeds up similarity calculations, making the approach practical for large-scale applications. The paper highlights that compressed signatures (top 5000 dimensions) achieve performance comparable to full signatures.
  • Alignment-Based Disambiguation as a Core Component: Integrating an explicit disambiguation step before similarity calculation is crucial for accurate results, especially when dealing with polysemous words.
  • Weighted Overlap's Effectiveness: The Weighted Overlap signature comparison method is shown to provide improvements over traditional cosine similarity, particularly in tasks requiring rank-based comparisons.

Key Results

The paper reports state-of-the-art performance on multiple evaluation benchmarks. Specifically:

  • On the RG-65 dataset, ADW achieved high correlation scores (Spearman ρ and Pearson r) using both full and truncated (top 5000 dimensions) vectors, with Weighted Overlap generally outperforming cosine similarity.
  • On the TOEFL dataset, ADW achieved high accuracy (96.3% with truncated vectors), matching the performance of full vectors.
  • The compressed semantic signatures (top 5000 dimensions) significantly reduce space requirements (over 15 times) while maintaining high performance.

Limitations and Future Work

  • Limitations: The paper acknowledges that pre-computing signatures for all possible textual items is infeasible. The current approach relies on WordNet as the knowledge source, which has its own inherent limitations in coverage and accuracy.
  • Future Work: The authors provide a Java API and a Web interface, encouraging integration into other NLP systems. They also release the source code and pre-computed signatures, facilitating further research and development in semantic similarity measurement. The framework's ability to handle arbitrary linguistic items suggests potential for applications beyond those explicitly tested.

Summary updated

BibTeX
@inproceedings{pilehvar2015open,
  author = {Mohammad Taher Pilehvar and Roberto Navigli},
  title = {{An Open-source Framework for Multi-level Semantic Similarity Measurement}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure