Consider two words: thing and stethoscopy. If you look at a massive corpus of text, where does the word thing appear? It can appear next to verbs, adjectives, and nouns of every imaginable category. Its contextual distribution is vast, varied, and highly unpredictable. If you try to guess the surrounding words given thing, your probability distribution is flat. In information theory, this flat distribution represents high entropy.
Conversely, where does stethoscopy appear? It is tightly constrained to medical environments, surrounded by a narrow vocabulary like diagnostic, heart, physician, or examination. Its context distribution is highly peaked and predictable—yielding low entropy.
This fundamental asymmetry forms the backbone of a powerful insight in computational linguistics: semantic specificity is encoded in the statistical geometry of context. By using information-theoretic tools like Shannon entropy and Kullback-Leibler (KL) divergence, researchers can mathematically quantify how general or specific a linguistic unit is based entirely on the company it keeps.
The Mechanics: Measuring Distributional Distortion
How do we mathematically capture this intuition? One foundational approach is to measure how much a specific word's local context distribution deviates from the background language. Herbelot et al. formalized this by calculating the KL divergence between a term's local context probability and the global corpus distribution. Highly specific terms heavily distort the expected background distribution of language, while generic words blend seamlessly into it.
Alternatively, we can look directly at the Shannon entropy ($H$) of the surrounding words. As Rambelli et al. demonstrate, contextual entropy serves as a direct proxy for categorical specificity, independent of whether a concept is concrete or abstract. When a word's context distribution is "peaked," its entropy is low; when it is "flat" or noisy, its entropy is high—a property Lee et al. leverage to quantify the stability and noisiness of medical concept embeddings.
Instead of discrete counts, this can even be modeled geometrically. The authors of CatE (Discriminative Topic Mining via Category-Name Guided Text Embedding) use directional statistics via the von Mises-Fisher distribution on spherical embeddings, where a high concentration parameter represents a highly specific, localized semantic footprint, and a low concentration represents a general term that spans multiple contexts.
High Specificity (e.g., "stethoscopy") Low Specificity (e.g., "thing")
▲ ▲
│ █ │
│ █ │
Prob │ █ │ █ █ █ █ █ █ █ █ █ █
│ █ █ █ │ █ █ █ █ █ █ █ █ █ █
└────────────► └────────────────────►
Context Words Context Words
(Low Entropy / Peaked) (High Entropy / Flat)
Uncovering Hierarchies and Hypernymy
Why does this matter? One of its most elegant applications is solving the problem of asymmetric semantic relations, such as hypernymy (the "is-a" relationship). Traditional vector similarity (like cosine distance) is symmetric: dog is just as close to animal as animal is to dog. But animal is a broad hypernym, while dog is a specific hyponym.
To discover the direction of the hierarchy, Santus et al. introduced SLQS, an entropy-based measure. By identifying the most strongly associated contexts of a word and calculating their median entropy, they showed that hypernyms consistently exhibit higher context entropy than their narrower hyponyms. This insight allows supervised systems like ROOT13, developed by Santus et al., to accurately separate hypernyms from co-hyponyms (like dog and cat) by evaluating context breadth.
This principle scales from individual words to entire sentences. By representing sentences as probability distributions rather than static vectors, Gimpel et al. showed that the entropy of the resulting distribution naturally encodes entailment: a general sentence has higher entropy, while a precise, specific sentence has a tighter, lower-entropy representation.
Isolating Technical Terminology and Keywords
Because specialized domains rely on precise vocabularies, entropy metrics naturally excel at isolating technical terminology from common language. This realization dates back to foundational work by Blois et al., who demonstrated that calculating the entropy of word distributions across distinct medical categories cleanly separates low-entropy technical terms from high-entropy general vocabulary.
Modern applications apply this to personalize models or detect linguistic anomalies:
- Keyword Detection: Yeh et al. cluster word embeddings and use cluster entropy to automatically identify user-specific keywords for speech recognition. Words with low entropy across clusters are highly topical, whereas high-entropy words are generic fillers.
- Clinical Assessment: Rouhizadeh et al. utilize distributional specificity scores to flag unexpected or atypical word choices in the language of children with autism.
- Translation Analysis: In translation studies, Yung et al. use the entropy of semantic senses to prove the "explicitation hypothesis," showing that translators systematically replace high-entropy, ambiguous connectives (like and) with low-entropy, highly specific ones (like furthermore).
Context Informativeness and Linguistic Productivity
The relationship between entropy and specificity also operates in reverse: we can measure how much a context constrains a target word. When a sentence provides a highly restrictive, low-entropy environment, it is incredibly informative.
Kabbach et al. leverage this to achieve "nonce learning"—training high-quality word representations from only a single example. By using entropy to filter out generic mentions and focus exclusively on highly informative contexts, models can learn rapidly without catastrophic forgetting or vector space drift.
Finally, this framework explains the structural flexibility of multi-word expressions (MWEs) and grammatical constructions, a concept known as productivity. Lenci et al. use relative entropy to measure the degrees of freedom within linguistic constructions. If a structural slot has low entropy, it is a fixed, non-productive idiom (like kick the bucket); if it has high entropy, it is highly productive. Tracking this over time, Alves et al. demonstrate that multi-word expressions in scientific English have steadily increased in syntagmatic productivity, adapting to broader, more diverse contexts over the centuries.
Even abstract grammatical classes reveal themselves through this lens. As Bhat et al. show, treating word embeddings as probabilistic feature sets allows the entropy of those sets to cleanly differentiate high-entropy function words (like prepositions and articles) from low-entropy content words.
Key Foundational Work
Measuring semantic content in distributional vectors (2013) — contributed 2 concept elements to this direction.
Word and Object in Disease Descriptions (1980) — contributed 2 concept elements to this direction.
Distributional semantic models for the evaluation of disordered language (2013) — contributed 1 concept elements to this direction.