A comparative evaluation of off-the-shelf distributed semantic representations for modelling behavioural data

Francisco Pereira, Samuel J. Gershman, Samuel Ritter, Matthew BotvinickCognitive 2016 · DOI

TL;DR

This paper rigorously compares various pre-trained word embedding models to see how well they predict human behavior in semantic tasks. They found that models trained on massive datasets, like GloVe and word2vec, generally perform best, and that simple vector similarity measures are surprisingly effective for many tasks.

What Problem Does This Paper Address?

Understanding how humans represent and process word meanings is a fundamental goal in cognitive science and linguistics. One powerful approach is to use "distributed semantic representations" (DSRs), which represent words as vectors in a high-dimensional space, where proximity in this space reflects semantic similarity. These vector representations are often learned automatically from large text corpora. Researchers have increasingly used these DSRs to model human behavior in semantic tasks (like word association or similarity judgments) and even to decode brain activity (fMRI). However, with a growing number of DSRs available "off-the-shelf," it's unclear which ones are best suited for modeling human data. This paper aims to provide a comprehensive evaluation of these off-the-shelf DSRs to guide researchers in choosing the most effective representations for their work.

Methods

The core of the paper involves evaluating a range of popular, publicly available word embedding models. The authors focus on DSRs derived from distributional semantics, meaning they are learned from word co-occurrence patterns in text. They categorize these methods into "context-counting" and "context-predicting" approaches, and also distinguish between "local" and "global" context windows.

For evaluation, they employ a standard methodology:

  1. Generate Rankings: For each task, they create a "true ranking" based on human judgments or behavioral data.
  2. Predict Rankings: They then compute the similarity between word vectors from each DSR model and generate a "predicted ranking."
  3. Measure Accuracy: The accuracy of each DSR is measured by the correlation (typically Spearman rank correlation) between the true and predicted rankings.

They also explore the role of different vector similarity measures (cosine similarity, correlation, and Euclidean distance), ultimately finding cosine similarity and correlation to be generally more robust.

Datasets

The paper utilizes several established datasets for evaluating semantic tasks:

  • Word Association: Nelson, McEvoy, and Schreiber (2004) free association norms.
  • Similarity and Relatedness Judgments:
    • MEN dataset
    • SimLex-999 dataset (focusing on similarity rather than relatedness)
    • WordSim-353 dataset
  • Synonyms and Analogy Problems:
    • TOEFL dataset (retired synonym items)
    • Google Analogy dataset (semantic and syntactic analogies)

The paper does not introduce new benchmark datasets but rather leverages existing ones to provide a standardized comparison.

Key Insights and Ideas

  • Massive Corpora Reign Supreme: DSRs trained on extremely large text corpora (billions of words) consistently outperform those trained on smaller datasets, regardless of the specific model architecture. This suggests that exposure to a vast amount of linguistic data is crucial for capturing nuanced semantic information.
  • Simple Similarity is Powerful: Basic vector similarity measures (especially cosine similarity) are remarkably effective for predicting human semantic judgments across a wide range of tasks. This challenges the need for overly complex prediction models when using off-the-shelf embeddings.
  • Task Matters: While overall performance is correlated across tasks, some DSRs perform better on specific types of tasks. For instance, models with sparse and positive vectors (like NNSE) might be less suitable for analogy tasks where fine-grained distinctions are needed.
  • Dimensionality vs. Corpus Size: Increasing the dimensionality of word vectors doesn't always lead to better performance beyond a certain point. Corpus size appears to be a more critical factor for achieving high predictive accuracy.
  • Off-the-Shelf is Often Good Enough: For many researchers, using readily available, pre-trained DSRs is a practical and effective approach, saving significant computational resources and time compared to training custom models.

Key Results

The paper presents detailed results in tables and figures, but here are some highlights:

  • Top Performers: GloVe and word2vec (specifically the context-predicting version) consistently ranked among the top performers across most tasks.
  • Corpus Size Impact: Table 2 shows a strong positive correlation (around 0.6-0.75) between the log of corpus size and performance across tasks.
  • Dimensionality Impact: The correlation between dimensionality and performance was weaker (around 0.4-0.55), with some models showing diminishing returns.
  • Task-Specific Performance: While cosine similarity and correlation generally performed well, Euclidean distance was less effective, particularly for tasks requiring fine distinctions.
  • Inter-Task Correlation: Table 1 demonstrates a high correlation (often > 0.9) between the performance of different DSRs across various tasks, suggesting that models that perform well on one task tend to perform well on others.

Limitations and Future Work

  • Off-the-Shelf Focus: The study is limited to readily available DSRs. The authors acknowledge that custom-trained models on specialized corpora might yield better results for specific domains.
  • Vector Similarity Assumption: The reliance on simple vector similarity measures might not capture all aspects of human semantic processing, especially for more complex tasks or when subtle semantic distinctions are crucial.
  • Interpretability: While models like NNSE offer more interpretable (sparse, positive) vectors, their performance can be lower. The authors suggest that future work could explore methods that balance interpretability with predictive power.
  • Future Directions: The authors propose moving beyond word-level representations to concept-level or sentence/passage-level embeddings. They also suggest exploring "metric learning" to dynamically weight dimensions based on task or individual differences, potentially leading to more nuanced predictions. Furthermore, they highlight the importance of modeling the process of judgment rather than just the final output.

Summary updated

Abstract

In this paper we carry out an extensive comparison of many off-the-shelf distributed semantic vectors representations of words, for the purpose of making predictions about behavioural results or human annotations of data. In doing this comparison we also provide a guide for how vector similarity computations can be used to make such predictions, and introduce many resources available both in terms of datasets and of vector representations. Finally, we discuss the shortcomings of this approach and future research directions that might address them.
BibTeX
@inproceedings{pereira2016comparative,
  author = {Francisco Pereira and Samuel J. Gershman and Samuel Ritter and Matthew Botvinick},
  title = {{A comparative evaluation of off-the-shelf distributed semantic representations for modelling behavioural data}},
  booktitle = {Cognitive},
  year = {2016},
  doi = {10.1080/02643294.2016.1176907},
}
Figure