Introduction
Presented at ICML 2019, this research offers a statistically grounded alternative to the "cosine similarity" heuristic commonly used in Natural Language Processing (NLP). By framing sentence similarity as a model comparison problem, the authors provide a framework that is both theoretically robust and highly practical for real-time applications.
The core innovation lies in treating the comparison of two sentences as a statistical test: do these groups of word embeddings likely originate from the same generative distribution, or two distinct ones? This approach achieves state-of-the-art results in "online" settings—where data arrives sequentially—and demonstrates that word embedding magnitude, often discarded in directional metrics, is a critical signal for semantic meaning.
Problem Statement
In Semantic Textual Similarity (STS), the goal is to quantify how closely the meanings of two text snippets correspond. The standard approach involves representing words as vectors (embeddings), averaging them into a "sentence vector," and calculating the cosine similarity.
However, this approach suffers from two major flaws:
- Heuristic Nature: Cosine similarity is a geometric rule of thumb rather than a formal statistical model. It fails to explicitly account for the underlying distribution of words or sentence complexity.
- The "Online" Gap: Top-performing methods like SIF (Smooth Inverse Frequency) require a "look-ahead" at the entire dataset to perform Principal Component Analysis (PCA). In real-world scenarios, such as a chatbot responding to a single user query, the entire dataset is not available beforehand.
Proposed Methodology
The authors propose a probabilistic framework where similarity is calculated via Model Comparison. Given two sets of word embeddings, $D_1$ and $D_2$, the framework contrasts two hypotheses:
- Model 1 ($M_1$): Both sentences are generated from the same underlying semantic distribution (they are "the same").
- Model 2 ($M_2$): Each sentence is generated from its own distinct distribution (they are "different").
Instead of a standard Bayesian approach, which can be sensitive to priors and computationally intensive, the authors utilize Information Criteria (IC)—specifically the Akaike Information Criterion (AIC) and the Takeuchi Information Criterion (TIC). These metrics reward models that fit the data well while penalizing complexity, effectively preventing overfitting.
Two specific generative models were tested:
- von Mises-Fisher (vMF): A distribution for data on a hypersphere, focusing only on vector direction.
- Gaussian (Diagonal): A distribution that accounts for both the direction and the magnitude of the vectors.
Results and Discussion
The experimental results yielded several key insights:
- Magnitude is Meaningful: The Gaussian model, which considers vector length, consistently outperformed the vMF model. This suggests that the "intensity" or frequency information encoded in word vector length is vital for understanding sentence similarity.
- Online Performance: In "online" scenarios where PCA is unavailable, the proposed method outperformed standard weighted averaging (MWV) and Word Mover’s Distance (WMD). It effectively matched the performance of the more complex SIF method while remaining applicable to real-time stream processing.
- Efficiency: With a complexity of $O(nd)$, the method is computationally efficient. The diagonal Gaussian AIC method was approximately 33% faster than the SIF+PCA baseline during evaluation, making it suitable for high-throughput systems.
- Statistical Justification: The derived similarity measure naturally mirrors Tversky’s contrast model from cognitive science, defining similarity by contrasting common features against distinctive ones.
Limitations and Future Work
While robust, the current model assumes that words in a sentence are "independent and identically distributed" (i.e., a bag-of-words model), thus ignoring word order. Additionally, the authors assumed a unimodal distribution for each sentence; exploring multi-modal distributions (multiple semantic clusters per sentence) remains a promising area for future research. Finally, for very short sentences, statistical estimates can become unstable, requiring "padding" with common punctuation or stop words to stabilize the calculations.
Summary updated