TL;DR
This paper introduces a novel method for computing semantic relatedness using Wikipedia's category network, demonstrating its effectiveness and competitiveness against WordNet on various benchmarks. The authors show that Wikipedia-derived knowledge can significantly improve NLP tasks like coreference resolution, highlighting its potential as a valuable, large-scale, and continuously updated knowledge base.
What Problem Does This Paper Address?
The paper tackles the challenge of acquiring and utilizing world and domain knowledge for Natural Language Processing (NLP) tasks. While advances in NLP have been significant, many high-level linguistic tasks (like question answering and textual entailment) critically depend on this knowledge. Existing knowledge bases, such as WordNet, often suffer from limited coverage and scalability issues. The authors propose using Wikipedia, a vast and collaboratively edited online encyclopedia, as a rich source of semantic information to overcome these limitations.
Methods
The core of the paper's methodology involves treating Wikipedia's category system as a semantic network. The process includes:
- Page Retrieval and Disambiguation: Given a pair of words, the system retrieves the corresponding Wikipedia pages. A novel disambiguation strategy is employed where pages help disambiguate each other by maximizing relatedness.
- Category Network Extraction: The categories assigned to the retrieved pages are extracted.
- Path Computation: Paths between category pairs are computed within the Wikipedia category network, using a depth-limited search to manage the graph's complexity.
- Relatedness Measure Computation: Standard path-based and information-content-based semantic relatedness measures are adapted and applied to the Wikipedia category network. Additionally, a text overlap measure (Lesk) is adapted using the first paragraph of Wikipedia articles.
Datasets
The authors evaluate their approach on several standard English datasets for semantic relatedness:
- Miller and Charles (M&C): A list of 30 noun pairs.
- Rubenstein and Goodenough (R&G): A synonymity list of 65 word pairs (a subset of M&C).
- WordSimilarity-353 Test Collection (353-TC): A larger dataset, used both in its entirety and as a test subset.
- KLEB: A subset of the reader-based lexical cohesion dataset.
For an extrinsic evaluation, they use the ACE 2003 dataset for coreference resolution.
For German experiments, they use a German translation of the Rubenstein and Goodenough (R&G) dataset.
Key Insights and Ideas
- Wikipedia as a Semantic Network: The paper convincingly demonstrates that Wikipedia's category structure can be effectively leveraged as a semantic network for computing relatedness, offering broader coverage than traditional resources like WordNet.
- Wikipedia Outperforms WordNet on Specific Datasets: While WordNet excels on datasets designed for similarity, Wikipedia shows superior performance on datasets specifically targeting semantic relatedness, suggesting a better fit for tasks requiring broader conceptual connections.
- Coreference Resolution Improvement: Integrating Wikipedia-derived semantic features significantly boosts the performance of a machine learning-based coreference resolution system, validating its practical utility in complex NLP tasks.
- Robustness Across Wikipedia Versions: The methodology proves robust, showing consistent performance across different versions of the English Wikipedia, even with its quasi-exponential growth.
- Cross-Lingual Applicability: The inherent structural consistency of Wikipedia across languages allows for straightforward application of the developed methods to other languages, as demonstrated with German.
Key Results
- Intrinsic Evaluation: Wikipedia-based relatedness measures correlate well with human judgments, often outperforming a Google count baseline. On datasets like 353-TC and KLEB, Wikipedia measures outperformed WordNet measures, particularly for relatedness rather than similarity.
- Extrinsic Evaluation (Coreference Resolution): Adding Wikipedia semantic features to a coreference resolution system led to performance improvements, comparable to or sometimes exceeding those gained from WordNet features. This was particularly evident for common nouns.
- German Experiments: Wikipedia performed competitively with GermaNet on the German R&G dataset, showing slightly better performance when considering all pairs and a larger coverage.
Limitations and Future Work
- Limitations of Wikipedia Categorization: The authors acknowledge that Wikipedia's category system is not a fully-fledged ontology, as relations are not semantically typed. This limits the ability to distinguish between similarity and relatedness directly from the categories.
- Dataset Size and Arbitrariness: The available datasets for evaluating semantic relatedness are noted as being small and somewhat arbitrarily assembled, making fair comparisons challenging.
- Future Work: The authors plan to focus on making semantic relations between concepts explicit within the Wikipedia category network to derive semantic similarity measures. They also aim to explore how to leverage the rich information in Wikipedia's article texts more effectively. The potential for using collaboratively created folksonomies like Wikipedia in NLP applications, similar to hand-crafted taxonomies, is highlighted as a key takeaway.
Summary updated