Abstract
Traditional evaluations of word embeddings rely on symmetric similarity measures, which fail to capture the inherent directional nature of human language and cognition. This paper introduces a Bayesian framework to quantify asymmetry in both static (e.g., GloVe) and contextual (e.g., BERT) embeddings. Our analysis reveals a "BERT Paradox": while contextual models significantly outperform static models in capturing directional relationships and prototypicality, they perform poorly on traditional symmetric similarity benchmarks. We validate our findings using large-scale human association datasets, identifying the Small World of Words (SWOW) as the most robust ground truth for modern asymmetry research.
Introduction
In Natural Language Processing (NLP), the quality of word embeddings—mathematical representations of words—is typically measured by similarity. However, similarity is a symmetric metric: the distance from "dog" to "puppy" is identical to the distance from "puppy" to "dog." Human cognition, as noted by psychologist Amos Tversky, is fundamentally asymmetric; we naturally state that "an ellipse is like a circle" because the circle serves as a prototypical reference, while the reverse statement feels less natural.
Standard evaluation methods overlook this geometric property of language. Furthermore, measuring asymmetry in contextual models like BERT is challenging because a word's representation shifts based on its surrounding sentence. This research bridges the gap by proposing a theoretically sound method to measure asymmetry in both static and dynamic embedding spaces.
Methodology
The authors define the Log Asymmetry Ratio (LAR) to compare the conditional probability of word $b$ given word $a$ ($P(b|a)$) against the reverse ($P(a|b)$).
For static embeddings such as GloVe and word2vec, probabilities are estimated using vector projection and softmax. For contextual embeddings like BERT, a novel Bayesian approach is employed. Since BERT lacks a single fixed vector for a word, the authors sample thousands of real-world contexts from Wikipedia where two words co-occur. BERT is then used as a Masked Language Model to predict the likelihood of one word appearing given the presence of the other in that specific context. Averaging these results across contexts provides an "unbiased" estimate of the general relationship between words.
Datasets and Ground Truth Validation
The study utilizes three major human "word evocation" datasets: the Edinburgh Association Thesaurus (EAT), Florida Association Norms (FA), and the Small World of Words (SWOW).
As shown in Table 1, the SWOW dataset is the most extensive and robust, showing the highest counts and strong correlations with other human datasets. The authors use Correlation on Asymmetry (CAM) to measure how well a model's LAR aligns with this human evocation data.
Results and Analysis
The BERT Paradox
A key finding is the divergence between asymmetry and similarity performance. While BERT-large and RoBERTa-large achieve the highest correlation with human asymmetry data, they perform poorly—sometimes near-randomly—on traditional similarity benchmarks like SimLex-999. For instance, fastText achieves a .45 correlation on SimLex-999, while BERT-large manages only .16. This suggests that BERT learns how words interact and relate in context (directional logic) rather than fixed "dictionary" definitions.
Relation-Specific Performance
Models exhibit varying degrees of success depending on the type of semantic relationship.
As illustrated in Figure 1, models are generally effective at capturing directional logic for "isA" (e.g., robin is a bird) and "atLocation" (e.g., stars are in the sky). However, they struggle with "hasProperty" or "capableOf" relations. This discrepancy may stem from "reporting bias," where obvious properties (e.g., "the sun is yellow") are rarely explicitly stated in text.
Discussion and Conclusion
The ability of BERT to capture asymmetry implies it successfully learns the hierarchical nature of language from raw text. Larger model capacity appears to be a significant factor in "memorizing" these nuanced lexical semantics.
However, the Bayesian method for contextual embeddings is computationally intensive, requiring the processing of thousands of paragraphs per word pair. Future work may explore how fine-tuning affects a model's sense of asymmetry and whether multi-modal models can better capture "hasProperty" relations that are often omitted in written corpora.
Summary updated