Standard word embeddings are built on the distributional hypothesis: words that appear in similar contexts share similar meanings. While this allows models to learn that "coffee" and "cup" are related, it creates a fundamental "similarity vs. relatedness" problem. In a distributional space, "hot" and "cold" often appear in identical contexts (e.g., "the water is ___"), leading the model to treat antonyms as synonyms.
Semantic Specialization (or "retrofitting") is a post-processing paradigm that reshapes these distributional spaces using external linguistic knowledge. Instead of training new vectors from scratch, researchers "warp" the existing space to satisfy specific lexical constraints.
The Attract-Repel Mechanism
The core intuition behind specialization is a multi-objective optimization problem. As pioneered by Nikola Mrkšić, Ivan Vulić, and colleagues in their ATTRACT-REPEL framework, the process involves three forces:
- Attract: Pull known synonyms (e.g., happy and cheerful) closer together.
- Repel: Push known antonyms (e.g., happy and sad) further apart.
- Regularize: Ensure the vectors don't move too far from their original positions, preserving the broad distributional knowledge captured during initial training.
By applying these constraints, the model learns to distinguish between true semantic similarity and mere topical association. This is particularly effective for tasks like Dialogue State Tracking, where confusing "confirm" with "reject" can break a conversation.
Handling Morphology and Structure
Specialization isn't limited to simple synonym lists. In Morph-fitting, Ivan Vulić and Roi Reichart demonstrated that simple language-specific rules can be used to specialize vectors for morphologically rich languages. By pulling inflectional variants (e.g., run, runs, running) together and pushing derivational antonyms (e.g., appear, disappear) apart, they improved performance without needing expensive, hand-curated lexicons.
This structural refinement extends to complex linguistic categories. Vulić and Anna Korhonen showed that English-language resources like VerbNet—which groups verbs by grammatical behavior—can be used to induce verb classes in other languages. The English "teacher" language provides the semantic constraints that reshape the target language's vector space.
Cross-Lingual and Multi-Source Injection
One of the most powerful aspects of specialization is its ability to bridge languages. By applying constraints across a joint vector space, high-resource languages like English can "teach" semantic nuances to lower-resource languages.
Furthermore, specialization can incorporate diverse data types. The DASI method, proposed by Yinpei Dai and Zhijian Ou, combines dictionary definitions with lexical relations. They found that refining existing vectors via an autoencoder is significantly more efficient than training from scratch, as it leverages the "latent" knowledge already present in large-scale corpora while injecting precise semantic boundaries.
From Static Vectors to Language Models
While early work focused on static embeddings like Word2Vec or GloVe, the field has evolved to specialize Transformer-based models. LexFit, developed by Goran Glavaš and Edoardo Ponti, applies these same lexical constraints to fine-tune models like BERT.
By "rewiring" the model's parameters to prioritize semantic similarity over topical relatedness, LexFit allows pretrained models to act as high-quality, "decontextualized" word embedders. This demonstrates that the principles of semantic specialization—attraction, repulsion, and regularization—remain a vital tool for aligning neural representations with human linguistic intuition.