In natural language processing, we often use the word "similarity" as a catch-all for any two concepts that feel close to one another. However, researchers have long argued that this umbrella term masks a fundamental distinction between two different ways words relate: semantic similarity and topical relatedness.
Understanding this distinction is not just a pedantic exercise in linguistics; it is the key to moving beyond "black box" word embeddings toward models that truly understand functional equivalence and logical necessity.
The Coffee and Cup Problem
Imagine three words: coffee, cup, and mug.
To a standard distributional model like Word2Vec or GloVe, all three are highly "similar" because they frequently appear in the same contexts. However, as Felix Hill, Roi Reichart, and Anna Korhonen demonstrate in their work on SimLex-999, humans perceive these relationships very differently:
- Semantic Similarity (Taxonomic): Cup and mug are similar. they share functional properties and can often be substituted for one another. They are members of the same category.
- Topical Relatedness (Associative): Coffee and cup are related. They are frequently found together in the world, but they are not the same kind of thing. You cannot drink a cup, and you cannot pour coffee into coffee.
The "Distributional Hypothesis"—the idea that words occurring in similar contexts have similar meanings—tends to conflate these two. Because coffee and cup appear in the same sentences, models map them to nearby points in vector space. This conflation leads to "false positives" in tasks like automated thesaurus construction or medical text mining, where a model might treat a symptom and its treatment as synonyms simply because they are topically linked Pesaranghader et al..
Why Distributional Models Struggle
The core of the problem lies in the training objective. Most language models are trained to predict a word based on its neighbors. This naturally captures topical association. As Manaal Faruqui and Chris Dyer point out in their work on non-distributional representations, these models lack the "grounding" provided by human-curated taxonomies like WordNet.
Without explicit structural information, a model cannot tell if two words appear together because they are synonyms (similarity) or because they are part of the same event (relatedness). This is particularly evident in verbs. Daniela Gerz and colleagues introduced SimVerb-3500 to highlight how models struggle with the nuanced syntactic and semantic differences between actions, often grouping all "restaurant-related" verbs together regardless of their actual meaning.
Rewiring Models for Genuine Similarity
To fix this, researchers have moved toward "injecting" structured knowledge into neural architectures. The goal is to "rewire" the vector space so that distance reflects taxonomic identity rather than just co-occurrence.
- Lexical Constraints: Ivan Vulić and his team developed LexFit, a lightweight fine-tuning method that uses synonym pairs to pull similar words closer together while pushing unrelated but topically associated words apart.
- Hybrid Embeddings: Systems like QLUT combine the statistical strengths of Word2Vec with graph-based traversals of knowledge bases like BabelNet. By looking at the "path" between concepts in a hierarchy, the model can verify if a relationship is truly taxonomic.
- Information Content: Building on foundational work by Philip Resnik, researchers use Information Content (IC) to measure how much specific information two concepts share within a taxonomy. This allows a model to mathematically determine that a "car" is more similar to a "bicycle" than to "gasoline," even if "car" and "gasoline" appear together more often in text.
The Impact on Reasoning and Bias
The distinction between similarity and relatedness isn't just about better dictionaries; it affects how AI reasons. Hongyin Luo and James R. Glass found that when models rely on simple semantic similarity (co-occurrence), they often inherit social biases. For example, a model might associate certain professions with specific genders simply because those words appear together frequently in training data.
By reframing the problem as textual entailment—asking if one sentence logically follows from another rather than if they are just similar—they were able to mitigate these biases Luo & Glass 2023. This shift moves the model's focus from "what words are associated with this topic?" to "what does this sentence actually mean?"
Conclusion
As we move toward more complex applications like table union search or online debate analysis, the failure to distinguish between similarity and relatedness becomes a critical bottleneck. A model that cannot tell the difference between a counterargument (related but opposite) and a supporting point (similar and aligned) cannot effectively navigate human discourse. By prioritizing "genuine similarity" benchmarks like SimLex-999, the field is moving toward a more precise, structurally grounded understanding of language.