How can a computer understand what a word means? Unlike humans, who map words to rich sensory experiences and cultural contexts, a machine requires a mathematical representation. The breakthrough idea that made modern natural language processing possible is that meaning can be modeled geometrically: by translating words into coordinates in a high-dimensional space, we can measure the "distance" between concepts using linear algebra.
This approach is grounded in the distributional hypothesis: words that appear in similar contexts tend to share similar meanings. By tracking which words frequently appear near one another, we can construct a geometric space where spatial proximity serves as a reliable proxy for semantic relatedness.
From Context to Coordinates
To build an intuitive understanding of this space, imagine tracking how often words appear next to terms like "dock," "captain," or "runway." A word like "ship" will frequently co-occur with "dock" and "captain," while "airplane" will co-occur with "runway." If we treat these co-occurrence counts as coordinates on a graph, "ship" and "boat" will be plotted close to each other because their contextual profiles match.
In early foundational work, the authors of Word Space demonstrated that these automatically induced distributed representations could successfully identify synonyms and handle lexical relationships without any human-coded hierarchies. Instead of relying on rigid, hand-crafted dictionaries, the system allows semantic relationships to emerge naturally from raw text statistics.
The Geometry of Similarity: Why the Angle Matters
Once words are mapped into a high-dimensional vector space, how do we calculate how close two meanings are? The most intuitive metric might be the straight-line (Euclidean) distance between two points. However, Euclidean distance suffers from a major flaw in text analysis: it is highly sensitive to word frequency. If the word "automobile" appears 10,000 times in a dataset and "car" appears only 100 times, their vectors will have vastly different lengths, placing them far apart in the space even though they are exact synonyms.
To solve this, researchers use cosine similarity. Rather than measuring the distance between the tips of two vectors, cosine similarity measures the angle between them.
$$\text{Similarity} = \cos(\theta) = \frac{A \cdot B}{\|A\| \|B\|}$$
If two vectors point in exactly the same direction, the cosine of their angle is 1, indicating perfect semantic alignment, regardless of how long the vectors are. This mechanism isolates the direction of the context vector—which represents the flavor of the meaning—from its magnitude, which merely represents frequency.
This geometric metric has proven remarkably versatile across diverse domains:
- Speech Recognition: Stephen Cox applied cosine similarity in A semantically-based confidence measure for speech recognition to evaluate whether a decoded word fits the broader semantic theme of an utterance, providing an independent sanity check on acoustic models.
- Disambiguation: In Acronym Disambiguation Using Word Embedding, Chao Li et al. demonstrated that comparing the cosine similarity between a vague acronym's context vector and candidate expansions can accurately resolve ambiguities (such as determining whether "CMU" means Carnegie Mellon or Central Michigan) without manual feature engineering.
- Quantifying Cultural Bias: Because vector spaces mirror human language usage, they also capture societal biases. In Based on billions of words on the internet, people = men, April H. Bailey et al. used cosine similarity to show that the generic concept of "people" is consistently positioned closer to male-associated terms than female-associated terms across massive internet corpora, exposing a systemic linguistic bias.
Compositionality: Combining Vectors
Individual words are the building blocks of language, but meaning truly happens when words combine into phrases and sentences. If "red" is a vector and "apple" is a vector, how do we represent a "red apple"?
The simplest approach is vector addition or averaging. In An Empirical Study on Compositionality in Compound Nouns, Diana McCarthy et al. explored this concept of semantic transparency. They found that for literal compound nouns, an additive model that combines the constituent vectors yields a predicted phrase vector that sits very close to the actual observed usage vector of the compound.
When a phrase is highly literal (compositional), its parts pull the compound vector toward their respective locations in the semantic space. Conversely, for idiomatic phrases (like "couch potato"), the constituent vectors point in directions completely unrelated to the actual usage of the phrase, causing a drop in similarity that alerts the system to a non-literal meaning.
Mapping Shifts and Contextual Diversity
Language is not static; a single word can shift its meaning depending on who is speaking or the context in which it appears. Modern vector spaces capture this fluidity by analyzing the geometric dispersion of vectors.
Instead of assigning a single fixed vector to a word, contemporary models generate contextualized embeddings where each instance of a word is mapped to a unique point based on its immediate surroundings. This allows researchers to measure contextual diversity as a geometric property.
In A New Formulation of Zipf's Meaning-Frequency Law through Contextual Diversity, Ryo Nagata demonstrated that the semantic variety of a word can be quantified by measuring the geometric dispersion—the spread or variance—of these instance vectors in the high-dimensional space. Words with many meanings form a wide, scattered cloud of vectors, while words with highly specific meanings cluster into a tight, dense point.
Similarly, ideological differences leave distinct geometric traces. In One Word, Two Sides: Traces of Stance in Contextualized Word Representations, Aina Garí Soler et al. showed that when opposing political groups use the same word, the vectors generated from within each group cluster together, but show a measurable divergence when compared across groups. By calculating the average cosine similarity between these ideological sub-clusters, the framework can mathematically isolate how different viewpoints subtly warp the internal meaning of shared vocabulary.