Measuring semantic textual similarity (STS) in an unsupervised manner remains a core challenge in natural language processing. While alignment-based methods like Word Mover's Distance (WMD) offer high interpretability by showing exactly how words map between sentences, they have historically underperformed compared to simple vector averaging techniques. This paper introduces Word Rotator's Distance (WRD), a novel metric that bridges this performance gap by decoupling word vectors into their norm (importance) and direction (meaning).
The Limitation of Euclidean Distance in WMD
Word Mover's Distance (WMD) calculates the minimum cost of transforming one sentence into another by treating words as points in a vector space and using the Euclidean distance between them as the transportation cost. However, the authors argue that Euclidean distance is an inappropriate metric because it conflates two distinct properties of a word vector: its length (norm) and its orientation (angle).
The authors hypothesize that the norm of a word vector serves as a highly effective proxy for word importance—content words naturally have larger norms than function words—while the angle represents its semantic content. WMD's reliance on Euclidean distance leads to suboptimal alignments because it penalizes words with different importance levels even if their meanings are identical. For instance, a word like "pho" might have a very different vector length than "noodle" despite being semantically closer than "snack." WMD might incorrectly align "noodle" with "snack" simply because their Euclidean distance is smaller, confusing the alignment algorithm.
Word Rotator's Distance (WRD)
To address these flaws, Word Rotator's Distance (WRD) treats sentences as distributions of words on a unit hypersphere. It utilizes the Earth Mover's Distance (EMD) framework but redefines the parameters to respect the natural decomposition of word vectors:
- Probability Mass (Weight): The importance of a word is mapped to its vector norm. This ensures that content-heavy words contribute more to the overall similarity score than frequent, less informative words.
- Transportation Cost (Distance): The semantic dissimilarity between words is measured using cosine distance (the angle between vectors) rather than Euclidean distance.
By "rotating" words along the surface of the hypersphere, WRD finds an optimal alignment that preserves the semantic nuances captured by vector directions while respecting the relative importance indicated by vector lengths. This approach allows the model to align "noodle" with "pho" regardless of their differing norms, as long as their angular distance is small.
Enhancing Representations with Vector Converters
The authors further enhance WRD by introducing a Vector Converter (VC). They observed that many powerful sentence embedding techniques (such as SIF weighting) essentially apply linear transformations to "denoise" word vectors before summing them. These transformations often involve removing the first principal component or scaling vectors based on frequency.
The VC extracts these mathematical transformations and applies them directly to individual word vectors before they are fed into the WRD algorithm. This process effectively "grows" the norms and adjusts the directions to be even better proxies for importance and dissimilarity. By applying these transformations at the word level, WRD maintains the interpretability of alignment-based methods while capturing the performance gains typically associated with sophisticated sentence-level models.
Key Results and Insights
The combination of WRD and the Vector Converter yields significant performance gains across a wide array of datasets, including SemEval STS tasks, the STS Benchmark (STS-B), and SICK-R.
- Consistent Outperformance: On the STS-B dataset using fastText embeddings, standard WMD scores a Pearson correlation of 52.31. The proposed WRD improves this to 62.10, and adding the Vector Converter pushes the score to a remarkable 74.94.
- Validation of the Norm-Importance Hypothesis: Ablation studies confirmed that treating all words as equally important (ignoring the norm) leads to a sharp decline in performance. This validates the theory that vector length is a crucial signal for semantic weighting.
- Robustness Across Embeddings: WRD consistently outperformed WMD and other alignment methods like DynaMax and BERTScore across different embedding types, including GloVe and word2vec.
Conclusion
Word Rotator's Distance demonstrates that the interpretability of alignment-based methods can be successfully combined with high performance. By recognizing that word importance and meaning are encoded in different geometric properties of word vectors, WRD provides a more robust and theoretically grounded approach to unsupervised semantic textual similarity. Future work may involve adapting this framework to contextualized embeddings like BERT, which currently require fine-tuning to achieve competitive results on unsupervised STS tasks.
Summary updated