Abstract
Retrofitting distributional word vectors with structured semantic knowledge is a standard technique to improve embedding quality. However, previous research noted that certain high-quality resources, like FrameNet, appeared to degrade performance on similarity benchmarks. This paper demonstrates that this perceived degradation is often an artifact of the evaluation metric used. By comparing Spearman rank correlation with Root-Mean-Square Error (RMSE), we show that retrofitting consistently improves semantic accuracy for the words it touches, even when global correlation scores suggest otherwise.
Introduction
Distributional word embeddings like GloVe and word2vec capture semantic relationships from raw text but often lack the precision of curated knowledge bases. Retrofitting addresses this by adjusting vectors to be closer to their neighbors in a semantic graph (e.g., WordNet). Despite the intuitive benefit, some studies found that resources like FrameNet (FN) negatively impacted performance. This paper investigates whether these "failures" are due to the resources themselves or the metrics used to evaluate them.
Methods
The study re-evaluates the retrofitting objective, which balances distributional fidelity with semantic graph proximity. The author contrasts two evaluation approaches:
- Spearman’s Rank Correlation: The standard metric, which measures the monotonic relationship between model scores and human judgments.
- Root-Mean-Square Error (RMSE): A distance-based metric that measures the absolute error between predicted and human scores.
Furthermore, the analysis distinguishes between "all pairs" in a benchmark and "pairs in resource"—those specifically present in the semantic database used for retrofitting. The datasets include word2vec and GloVe vectors retrofitted with WordNet (WN+), PPDB, and FrameNet (FN), including a novel FN-ANNO variant derived from FrameNet annotations.
Results: The Metric Trap
The most significant finding is the discrepancy between correlation and RMSE. Spearman correlation is sensitive to global rankings; if a resource has low coverage, the unchanged scores of missing words can distort the overall rank, masking local improvements.
As seen in Figures 1 and 2, the FN-ANNO resource appears detrimental when measured by correlation but highly effective when measured by RMSE. This "polarity flip" suggests that FrameNet provides high-quality semantic information that moves individual word pairs closer to human truth, even if it disrupts the global ranking of the entire dataset. This occurs because FrameNet groups words by conceptual structure (like "Attack"), which can include diverse words that create "scatter" in global rankings while still improving local accuracy.
Discussion and Conclusion
The perceived failure of certain semantic resources is largely a measurement error. While FrameNet is heterogeneous and often has lower coverage than WordNet, it consistently improves the vectors of the words it includes. RMSE provides a more sensitive lens for detecting these improvements.
However, low coverage remains a practical challenge for downstream tasks. Future work should explore larger, more consistently annotated resources like PropBank to achieve both high-quality semantic shifts and the broad coverage necessary to satisfy traditional metrics. Researchers are encouraged to look beyond global correlation and inspect improvements at the word-pair level to truly understand the impact of semantic retrofitting.
Summary updated