Exploring the use of word embeddings and random walks on Wikipedia for the CogAlex shared task

Josu Goikoetxea, Eneko Agirre, Aitor SoroaCogALex @ COLING 2014 · OpenReview

TL;DR

This paper explores how well different computational methods can predict a missing word given a set of related words, a task common in cognitive linguistics. They found that while word embeddings performed best on the specific CogAlex dataset, traditional knowledge-based methods like random walks on Wikipedia and WordNet struggled due to the dataset's mixed word types (not just nouns) and its nature as a thesaurus rather than a dictionary.

What Problem Does This Paper Address?

The core problem this paper tackles is measuring semantic similarity and relatedness between words, a fundamental challenge in natural language processing (NLP) and cognitive science. This capability has applications in areas like textual entailment, word sense disambiguation, and information retrieval. The authors specifically investigate how well different computational approaches can solve a task where the goal is to predict a missing word given a set of semantically related words, as presented in the CogAlex shared task. They aim to compare methods that rely on pre-existing knowledge bases (like WordNet and Wikipedia) with those that learn word meanings from large text corpora (word embeddings).

Methods

The researchers tested three main approaches:

  • Word Embeddings: They utilized the Mikolov model (specifically the Skip-gram variant) trained on a massive Google News corpus. This model represents words as dense, low-dimensional vectors where similar words have similar vector representations. The "distance" program from the word2vec tool was adapted to find the closest words in the vocabulary to a calculated vector representing the input set.
  • Random Walks on WordNet: This method treats WordNet (a lexical database) as a graph and uses random walks to determine semantic relatedness.
  • Random Walks on Wikipedia: Similar to the WordNet approach, this method models Wikipedia as a graph, where articles are nodes and links are edges. They employed PageRank and Personalized PageRank algorithms to rank the importance of articles and infer relatedness. To build the Wikipedia graph, they created a dictionary linking mentions to Wikipedia articles using titles, redirects, disambiguation pages, and anchor text.

Datasets

  • Google News Corpus: Used for training the word embeddings (approximately 100 billion words).
  • Edinburgh Association Thesaurus (CogAlex dataset): This was the primary dataset used for the shared task, serving as both training and test data. The authors note a key difference: unlike datasets like WordSim353 which primarily contain singular nouns, the CogAlex dataset included a mix of word types (around 50% nouns) and grammatical words, and words in plural.
  • WordSim353: Used for comparison during development, this dataset consists of noun-relatedness pairs.

Key Insights and Ideas

  • Dataset Mismatch is Crucial: The most significant insight is how the nature of the CogAlex dataset (a thesaurus with mixed word types) differed from typical semantic similarity benchmarks (like WordSim353, which are often noun-focused). This mismatch heavily influenced the performance of the algorithms.
  • Embeddings Excel on Mixed Data: Word embeddings, which learn from broad corpora, proved more robust to the varied word types in the CogAlex dataset compared to knowledge-based methods.
  • Knowledge Bases Struggle with Non-Nouns: Random walks on WordNet and Wikipedia performed poorly because these resources are heavily noun-centric. They couldn't effectively capture relationships for grammatical words or plurals, nor could they easily distinguish between lexicalizations (e.g., noun vs. adjective) if the dataset didn't explicitly provide them.
  • Task Definition Ambiguity: The authors point out that the task's examples suggested a dictionary-like definition recovery, but the actual data was a thesaurus, leading to a disconnect in expectations.

Key Results

  • Development Phase: WordNet and Wikipedia random walk methods performed poorly on the CogAlex development data. Word embeddings achieved an accuracy of 14.1% on the training data.
  • Comparison with WordSim353: In contrast, on the WordSim353 dataset (which is noun-focused), random walks on Wikipedia achieved a Spearman correlation of 72.8%, WordNet 68.5%, and embeddings 71.0%, showing the knowledge-based methods' strength on more structured, noun-centric data.
  • Test Results: Based on the development performance, the team submitted their word embedding approach. They achieved 16.35% accuracy, ranking fourth out of twelve submissions for the CogAlex shared task.

Limitations and Future Work

  • Limitations:
    • The primary limitation identified is the mismatch between the nature of the CogAlex dataset (thesaurus, mixed word types) and the strengths of the knowledge-based methods (WordNet, Wikipedia) which are more suited for noun-centric, structured knowledge.
  • The random walk algorithm couldn't capture implicit semantic relationships (e.g., "absent" and "minded" being related in "absent minded") if they weren't explicitly encoded as links or co-occurrences.
  • The task definition was somewhat ambiguous, leading to a potential misinterpretation of the data's nature.
  • Future Work: Not explicitly discussed in the paper, but implied by the findings, future work could involve:
  • Developing knowledge-based methods that are more adaptable to mixed word types and thesaurus-like structures.
  • Integrating word embeddings with knowledge bases in a more sophisticated way to leverage the strengths of both.
  • Clarifying task definitions and dataset characteristics in shared tasks to ensure fair evaluation of different approaches.

Summary updated

BibTeX
@inproceedings{goikoetxea2014exploring,
  author = {Josu Goikoetxea and Eneko Agirre and Aitor Soroa},
  title = {{Exploring the use of word embeddings and random walks on Wikipedia for the CogAlex shared task}},
  booktitle = {CogALex @ COLING},
  year = {2014},
  url = {https://openreview.net/forum?id=JchFQQ4IIt7},
}
Figure