Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations

Philip Blair, Yuval Merhav, Joel BarryICLR 2017 · OpenReview

Evaluating how well a computer "understands" the meaning of words is a notoriously slippery task. For years, researchers have relied on human-annotated datasets for word similarity or analogy tasks. However, these datasets are expensive to build, often subjective, and rarely available in languages other than English. Furthermore, there is a known "correlation gap": performing well on these intrinsic tests doesn't always mean the model will succeed at actual downstream tasks like classifying the sentiment of a movie review.

In their ICLR 2017 paper, Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations, Philip Blair and his colleagues at Basis Technology propose a clever solution: let the data build the test. By leveraging the structured graph of Wikidata, they created a language-agnostic pipeline to automatically generate "outlier detection" tasks—the linguistic equivalent of "one of these things is not like the others."

The Methodology: Graph-Based Outlier Detection

The authors treat Wikidata as a massive semantic graph. In this graph, nodes are entities (like "Chicago Bulls") and edges represent relationships like "instance of" (a basketball team) or "subclass of" (a sports team). The core assumption is that semantic similarity is inversely proportional to graph distance.

To build the evaluation sets, the pipeline follows a structured process:

  1. Forming Clusters: The system picks a "class" (e.g., Association Football Club) and selects its most popular members (e.g., FC Barcelona, Manchester United) based on Wikipedia sitelink counts.
  2. Generating Outliers: To test the "granularity" of an embedding, the system selects outliers at three distinct levels of difficulty:
  3. $O_1$ (Hard): A "sibling" class (e.g., an American Football Team like the New England Patriots).
  4. $O_2$ (Medium): A "cousin" class (e.g., an Ice Hockey Team like the Anaheim Ducks).
  5. $O_3$ (Easy): A random entity at least 7 steps away in the graph.
Graph illustrating the generation of clusters and outliers of varying difficulty levels using Wikidata relationships.

Statistical heuristics are then used to filter out "noisy" clusters (e.g., entities that only differ by a year) and ensure the labels don't have too much syntactic overlap, which would make the task trivial for a machine.

The WikiSem500 Dataset

This automated pipeline bypasses the "human bottleneck," allowing for the creation of massive, diverse datasets that cover niche topics humans might not naturally think of. The primary contribution of this paper is WikiSem500, a benchmark dataset containing roughly 500 test groups for each of five languages: English, Spanish, German, Chinese, and Japanese. In total, it provides over 13,000 test cases.

Key Results and Insights

The authors evaluated several heavy-hitting models on the English portion of WikiSem500, including Google News word2vec (CBOW), GloVe, and Skip-Gram models trained on various corpora.

Table showing the performance of GloVe, CBOW, and Skip-Gram models on the outlier detection task across different training corpora.
  • Model Performance: Skip-Gram and CBOW models trained on the Wikipedia+Gigaword corpus were the top performers, reaching roughly 56-57% accuracy.
  • The Google News Surprise: The famous Google News vectors performed significantly worse than expected (around 22-24% accuracy). The authors hypothesize this is due to the news corpus lacking the broad, encyclopedic coverage found in Wikipedia.
  • Human Baseline: Humans achieved an overall precision of 68.9%. Interestingly, humans struggled with the broad domain knowledge required (e.g., knowing specific obscure book titles), highlighting exactly why an automated, knowledge-base-backed system is useful.
  • The Intrinsic-Extrinsic Link: One of the most significant findings is that performance on WikiSem500 correlates very strongly (Pearson’s $r > 0.97$) with downstream sentiment analysis performance. This suggests that outlier detection is a much better proxy for "real-world" utility than traditional similarity scores.

Limitations and Future Work

While highly effective, the approach has limitations. The current method is purely semantic; it can tell you "apple" doesn't belong with "car" and "truck," but it doesn't test if a model understands syntactic relationships (like "running" being the present participle of "run"). The authors suggest integrating WordNet or Wiktionary to add syntactic clusters in the future.

Additionally, because the test is built from Wikidata, models trained on Wikipedia might have an inherent, unfair advantage. Finally, handling "Out-of-Vocabulary" (OOV) words remains a challenge, as discarding entities or using sentinel vectors both introduce trade-offs in evaluation fairness.

Overall, this work marks a significant step toward more robust, scalable, and multilingual evaluation of the representations powering modern NLP systems.

Summary updated

Abstract

We propose a language-agnostic way of automatically generating sets of semantically similar clusters of entities along with sets of "outlier" elements, which may then be used to perform an intrinsic evaluation of word embeddings in the outlier detection task. We used our methodology to create a gold-standard dataset, which we call WikiSem500, and evaluated multiple state-of-the-art embeddings. The results show a correlation between performance on this dataset and performance on sentiment analysis.
BibTeX
@inproceedings{blair2017automated,
  author = {Philip Blair and Yuval Merhav and Joel Barry},
  title = {{Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
  url = {https://openreview.net/forum?id=SkuqA_cgx},
}
Figure