TL;DR
This paper proposes a novel statistical perspective on word embeddings, treating them as observations from random variables rather than just geometric vectors. This leads to surprisingly effective and efficient methods for measuring semantic similarity between sentences using correlation coefficients, outperforming many complex deep learning models on standard benchmarks.
What Problem Does This Paper Address?
The field of Natural Language Processing (NLP) has seen tremendous progress with the advent of word embeddings, which capture semantic relationships between words. However, while these embeddings are widely used, the methods for comparing sets of embeddings (representing sentences or phrases) have largely relied on geometric interpretations, like cosine similarity. This paper questions whether these geometric approaches are optimal and explores an alternative: viewing word embeddings through a statistical lens. The core problem is finding robust and efficient ways to measure semantic similarity between sentences by comparing their constituent word embeddings, moving beyond traditional geometric measures.
Methods
The authors propose two main families of methods:
- Pooling-based approaches: These methods first aggregate word embeddings within a sentence into a single vector using operations like mean, max, or min pooling. Then, standard univariate correlation coefficients (like Pearson's r, Spearman's rho, or Kendall's tau) are applied to compare these pooled sentence vectors. A key contribution here is the empirical validation that robust correlation coefficients (like Spearman's rho) and specific pooling strategies (like max-pooling) are crucial for handling the non-normal distributions and outliers often present in word embeddings.
- Pooling-free approaches: This more advanced approach avoids pooling altogether. Instead, it directly compares sets of word embeddings using multivariate correlation operators within Reproducing Kernel Hilbert Spaces (RKHS). The paper introduces a novel application of the Centered Kernel Alignment (CKA) as a generalization of squared cosine similarity and Pearson correlation for sets of word vectors. CKA is presented as a powerful tool that makes no parametric assumptions about the underlying distributions of the embeddings.
Datasets
The primary dataset used for evaluation is the Semantic Textual Similarity (STS) dataset series 2012-2016. This dataset consists of sentence pairs with human-annotated similarity scores on a scale of 0 to 5. The authors also mention using word embeddings trained on Common Crawl (600B tokens), specifically fastText, as it has shown consistently strong performance in previous evaluations. Other word embedding models like GloVe and word2vec are also used for comparison.
Key Insights and Ideas
- Statistical Perspective is Powerful: Treating word embeddings as observations from scalar random variables, rather than just geometric vectors, unlocks new avenues for similarity measurement. This statistical viewpoint is empirically validated to be highly effective.
- Outliers Matter: Word embeddings, especially from models like GloVe and fastText, often exhibit non-normal distributions with significant outliers. These outliers can severely degrade the performance of standard correlation coefficients and geometric measures.
- Robustness is Key: Employing robust correlation coefficients (like Spearman's rho) or outlier-handling techniques (like max-pooling or winsorization) is crucial for achieving high performance on STS tasks, particularly with non-normal embedding distributions.
- CKA as a Generalization: Centered Kernel Alignment (CKA) offers a principled way to generalize squared Pearson correlation and cosine similarity to sets of word vectors, providing strong empirical results without making strong distributional assumptions.
- Simplicity Can Be Superior: The proposed methods, especially pooling-based approaches with robust correlations, are significantly simpler and faster to implement than many complex deep learning architectures, yet achieve competitive or superior performance.
Key Results
- Pooling-based approaches using robust correlation coefficients (e.g., MaxPool Spearman) achieve results that outperform many recently proposed methods on standard STS benchmarks, while being much faster and simpler to implement.
- Max-pooling, when combined with Spearman's rho, consistently outperforms mean-pooling across various word embedding models and STS tasks.
- CKA, when applied to sets of word embeddings, also demonstrates very strong empirical results, comparable to or exceeding other state-of-the-art methods.
- The paper provides empirical evidence that deep learning-based sentence representation methods do not always outperform simpler set-based compositions of word vectors, especially when appropriate similarity measures are used.
- Table 1 shows that MaxPool Spearman with fastText embeddings achieves scores comparable to or better than many deep learning approaches, with scores in the high 70s for some STS tasks.
Limitations and Future Work
- Error Analysis: The authors acknowledge that even their best-performing methods still make errors, particularly in cases of antonymy, negation, and when sentences have identical meanings but different vocabulary. They suggest that these errors are often rooted in the inherent limitations of word embeddings themselves, such as conflating semantic similarity with conceptual relatedness or failing to distinguish synonyms from antonyms.
- Future Directions: The paper suggests that addressing the weaknesses of word embeddings themselves could further improve sentence-level similarity systems. They also hope their work encourages further exploration of the statistical perspective on word embeddings and entirely novel interpretations of textual representations. The authors also note that while CKA is computationally more expensive than pooling-based methods, it can offer performance boosts on certain tasks, making it a worthwhile consideration depending on the application.
Summary updated