Consider the words coffee, cup, and mug. If you were asked to group them by how "connected" they are, you might say all three belong together in a kitchen. But if you were asked to find a replacement for a broken cup, only mug would do.
This distinction highlights a fundamental divide in computational linguistics:
- Semantic Similarity (Taxonomic/Functional): Words that share defining physical or functional features and can often substitute for one another in a sentence (e.g., cup and mug).
- Semantic Relatedness (Thematic/Topical): Words that are associated through context, usage, or co-occurrence within a specific domain, but are not interchangeable (e.g., coffee and cup).
For years, standard Natural Language Processing (NLP) models treated these two concepts as interchangeable. However, as highlighted by Agirre et al. (2009) and Hill et al. (2015), failing to differentiate between strict similarity and broader topical association leads to imprecise semantic modeling. A model that conflates the two might predict that "fuel" is a great synonym for "vehicle" simply because they frequently appear together (Nguyen 0001 et al., 2018).
To build models that truly understand language, researchers have developed techniques to isolate genuine similarity from mere association.
The Root of Conflation: The Distributional Hypothesis
Why do standard word embeddings (like Word2Vec or GloVe) struggle with this distinction? The answer lies in the Distributional Hypothesis: the idea that words occurring in similar contexts share similar meanings.
Imagine a sliding window moving across a massive collection of text. It encounters sentences like:
- "He poured hot coffee into the ceramic cup."
- "He poured hot tea into the ceramic mug."
Because coffee, tea, cup, and mug all appear in close proximity to words like poured, hot, and ceramic, a standard bag-of-words or linear window model assigns them all high cosine similarity in the vector space. As Kiela (2015) points out, both relationships share similar linguistic contexts, creating vector spaces where topical associates and true synonyms are squashed together.
Mechanism 1: Shifting from Linear Windows to Syntactic Dependencies
One elegant way to fix this without relying on external dictionaries is to change how a model defines "context." Instead of looking at a flat, linear window of neighboring words, we can look at the structural rules of grammar.
Levy et al. (2014) demonstrated that replacing linear sliding windows with syntactic dependency-based contexts fundamentally alters what word vectors represent.
Linear Window (±2 words):
[He poured] -> COFFEE -> [into the]
Syntactic Dependency:
poured (verb) --dobj--> COFFEE (noun)
poured (verb) --prep_into--> CUP (noun)
In a dependency graph, coffee serves as the direct object (dobj) of the verb poured, while cup serves as the object of a prepositional phrase (prep_into). They play entirely different grammatical and semantic roles. Conversely, cup and mug will both consistently appear as the object of prep_into(poured).
By conditioning embeddings on these structural relationships, the model filters out topical noise. As supported by MacAvaney et al. (2018), using enhanced dependency graphs forces the vector space to capture functional similarity (interchangeable roles) rather than broad topical relatedness (same domain).
Mechanism 2: Injecting External Knowledge and Constraints
Another approach is to guide distributional models using human-curated linguistic structures, such as WordNet or FrameNet. If the data explicitly states that cup and mug are co-hyponyms (siblings in a taxonomic tree), we can force the vector space to reflect that geometry.
Counter-Fitting
Mrkšić et al. (2016) introduced a post-processing technique called counter-fitting. It takes an existing, topically-conflated vector space and applies an optimization function with three competing constraints:
- Synonym Attraction: Pull vectors of known synonyms (e.g., cup, mug) closer together.
- Antonym Repulsion: Push vectors of opposite meanings (e.g., hot, cold) farther apart—solving a classic failure mode where antonyms score highly because they share contexts.
- Vector Preservation: Ensure the vectors do not drift too far from their original positions, preserving useful distributional information.
Non-Distributional and Joint Models
Instead of fixing broken spaces, some methods build clean spaces from scratch. Faruqui et al. (2015) bypassed text corpora entirely to build sparse, binary vectors directly from lexical resources, ensuring that similarity scores strictly track taxonomic definitions. Similarly, the DASI framework proposed by Dai et al. (2020) uses an autoencoder to jointly learn from dictionary definitions and explicit semantic relations, successfully isolating genuine similarity from co-occurrence patterns.
Mechanism 3: Unlocking the Hidden Geometry of Word Embeddings
Surprisingly, standard architectures like Word2Vec Skip-gram actually learn the distinction between similarity and relatedness internally—we just historically threw half the answer away.
When training a Skip-gram model, the network maintains two distinct vector spaces: a Target matrix ($W$) and a Context matrix ($C$). When the model calculates the probability of a context word given a target word, it computes the dot product $W_{target} \cdot C_{context}$. Normally, after training, engineers discard the context matrix $C$ and use $W$ for downstream tasks.
Torabi Asr et al. (2018) demonstrated that these two matrices map directly to different cognitive processes:
- Target-Target Comparison ($W \cdot W$): Measures taxonomic similarity. It evaluates how similarly two words behave when they act as targets, capturing whether they share the same physical or functional properties.
- Target-Context Comparison ($W \cdot C$): Measures thematic relatedness. It evaluates how strongly one word triggers or predicts the presence of another, capturing asymmetric human associations (like stork evoking baby).
Why the Distinction Matters: Downstream Performance
Isolating these two properties isn't just an academic exercise; it directly impacts how well AI performs real-world tasks.
As Chiu (2016) demonstrated, older intrinsic benchmarks (like WordSim-353) conflated similarity and relatedness, making them poor predictors of how well an embedding would perform in actual applications. Benchmarks that isolate genuine similarity—such as SimLex-999, SimVerb-3500, and Multi-SimLex—provide a much cleaner signal for tasks requiring precise substitution.
| Task Type | Preferred Property | Example Application | Why it Works |
|---|---|---|---|
| Substitution-Based | Genuine Similarity | Lexical Substitution, Thesaurus Construction | Requires words that preserve grammatical flow and exact meaning when swapped (Mihalcea et al., 2009). |
| Expansion-Based | Topical Relatedness / Association | Document Classification, Information Retrieval | Benefits from finding any word within the same thematic domain to broaden search scope (Kiela, 2015). |
By moving away from a one-size-fits-all view of "meaning" and choosing the right mechanism—whether through syntactic contexts, external constraints, or targeted matrix comparisons—modern language models can choose whether to group items by what they are or what they are with.
Key Foundational Work
A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches (2009) — contributed 1 concept elements to this direction.
Comparing Wikipedia and German Wordnet by Evaluating Semantic Relatedness on Multiple Datasets (2007) — contributed 1 concept elements to this direction.
Combining Lexical Resources for Contextual Synonym Expansion (2009) — contributed 1 concept elements to this direction.