Abstract
Modern word embeddings derived from distributional statistics often fail to distinguish between semantic similarity and conceptual association. Specifically, antonyms frequently appear in similar contexts, leading models to assign them high cosine similarity. We introduce counter-fitting, a lightweight post-processing technique that leverages external linguistic constraints to refine word vector spaces. By simultaneously attracting synonyms and repelling antonyms, counter-fitting achieves a new state-of-the-art on the SimLex-999 benchmark and demonstrates significant improvements in downstream dialogue state tracking tasks.
Introduction: The Distributional Dilemma
The "distributional hypothesis"—the idea that words with similar meanings appear in similar contexts—is the cornerstone of modern NLP. However, this hypothesis has a fundamental limitation: antonyms like hot and cold or expensive and cheap are often distributionally indistinguishable. In standard vector spaces like GloVe, these opposites are clustered together.
This conflation of similarity and association is problematic for practical applications. In a Dialogue State Tracking (DST) system, if a user requests a "cheap" restaurant and the system treats "expensive" as a synonym, the interaction fails. This paper proposes a method to bridge the gap between mere association and true semantic similarity.
Methodology: The Counter-fitting Algorithm
Counter-fitting is an efficient post-processing step that "nudges" pre-trained vectors to satisfy known linguistic constraints. The optimization objective consists of three terms:
- Antonym Repel (AR): Increases the distance between antonymous pairs until they reach a threshold of orthogonality.
- Synonym Attract (SA): Decreases the distance between synonymous pairs.
- Vector Space Preservation (VSP): Ensures that the refined vectors remain close to their original distributional positions, preserving the broad semantic knowledge captured during initial training.
The process is computationally inexpensive, typically converging in under two minutes on standard hardware using Stochastic Gradient Descent (SGD).
Experimental Setup
The authors evaluated counter-fitting using:
- Base Vectors: GloVe and Paragram-SL999.
- Constraints: Synonym and antonym pairs from PPDB 2.0 and WordNet.
- Benchmarks: SimLex-999 (semantic similarity) and DSTC 2/3 (dialogue tracking).
Results and Discussion
Quantitative Performance
Counter-fitting significantly outperforms baseline models and previous state-of-the-art methods on the SimLex-999 dataset. While the human inter-annotator agreement was previously considered a "ceiling" (0.67), counter-fitted Paragram vectors reached a Spearman correlation of 0.74.
Qualitative Shift
The impact of counter-fitting is most visible in the nearest neighbors of words. Before refinement, the neighbors for "expensive" included its antonym "cheaper." After counter-fitting, the space is reorganized to prioritize true synonyms.
Dialogue System Improvements
In downstream DST tasks, counter-fitted vectors improved tracking accuracy. For instance, in the "Tourist Information" domain, accuracy rose from 60.5% to 62.8%. This suggests that injecting domain-specific ontologies into the vector space can make dialogue systems more robust to varied user phrasing.
Conclusion
Counter-fitting provides a simple yet powerful way to inject external knowledge into distributional word vectors. By explicitly modeling antonymy and synonymy, it overcomes the inherent limitations of the distributional hypothesis. While the method depends on the quality of external lexicons, its efficiency and performance gains make it a valuable tool for tasks requiring high-precision semantic similarity. Future work may explore balancing similarity with relatedness for tasks where conceptual association remains critical.
Summary updated