Abstract
Contextualised word embeddings from Neural Language Models (NLMs) like BERT represent words by considering both the target word and its surrounding context. In contrast, static word embeddings like GloVe offer memory- and compute-efficient representations but fail to distinguish between different word senses. We propose Context Derived Embeddings of Senses (CDES), a method that extracts sense-related information from contextualised embeddings and injects it into static embeddings to create sense-specific static vectors. Experimental results on multiple Word Sense Disambiguation (WSD) and sense discrimination benchmarks show that CDES achieves performance comparable to state-of-the-art sense embeddings while maintaining the efficiency of static models.
Introduction: The Embedding Trade-off
In Natural Language Processing (NLP), researchers typically choose between two paradigms of word representation:
- Static Embeddings (e.g., GloVe): These assign a single fixed vector to each word. While fast and memory-efficient, they are "sense-agnostic," meaning they cannot distinguish between a "bank" as a financial institution and a "bank" as a river edge.
- Contextualised Embeddings (e.g., BERT): These generate dynamic vectors based on the surrounding sentence. They capture nuance brilliantly but are computationally expensive, requiring the full model to run during inference.
The challenge lies in creating sense-specific static embeddings: multiple fixed vectors for a single word (one for each meaning) that are as accurate as BERT but as lightweight as GloVe. Previous attempts often struggled with data sparsity, as there is rarely enough sense-tagged text to train high-quality vectors for every possible meaning of every word.
Methodology: The CDES Framework
CDES treats the creation of sense embeddings as a form of model distillation, using a complex "teacher" (BERT) to train a simpler "student" (sense-specific static vectors).
Sense-Specific Projection
For a word $u$ and its $i$-th sense $s_i$, the model learns a diagonal projection matrix $A_i$. This matrix transforms the generic static embedding $g(u)$ into a sense-specific vector $s_i(u)$. The authors found that using diagonal matrices rather than full matrices significantly reduced parameters and prevented overfitting without sacrificing accuracy.
The Objective Function and Non-linearity
The system is trained to minimize the squared L2 distance between the transformed static vector and the contextualised embedding from BERT. Crucially, the authors found that the relationship between these spaces is complex; using a non-linear activation function (specifically GELU) to align the two spaces significantly improved performance over simple linear mapping.
Context Aggregation and Filtering
To combat data sparsity, CDES incorporates extra information from dictionary definitions (glosses) in WordNet and clusters sentences from Wikipedia to find representative contexts for rare senses. Additionally, because contextualised embeddings contain "noise" (grammar, gender, etc.) unrelated to word sense, CDES uses a learnable filter matrix ($W$) to strip away this extra info, focusing purely on semantic sense.
Evaluation and Key Results
The model was evaluated using standard benchmarks, including the SemCor 3.0 corpus for training and a suite of "all-words" WSD tasks (Senseval-2/3, SemEval-07/13/15). It was also tested on the Words in Context (WiC) dataset to measure its ability to discriminate between subtle shifts in meaning.
Performance Highlights
- WSD Accuracy: CDES reported the best performance in 4 out of 6 benchmarks compared to other sense-embedding methods, matching or beating models with much more complex architectures.
- Sense Discrimination: It outperformed existing static sense embeddings like LMMS and LessLex on the WiC task.
Visual Proof
t-SNE visualizations comparing GloVe, ARES, and CDES. CDES creates significantly tighter and more distinct clusters for different senses of the word "bank," clearly separating financial (red) and geographic (blue) meanings. Source, p. 7
As shown in the t-SNE visualizations above, CDES produces much cleaner separation between senses than traditional methods. For the word "bank," it successfully clusters financial terms like "payment" and "savings" away from geographic terms like "coast" and "riverbank."
Discussion and Future Work
CDES demonstrates that BERT's contextual knowledge can be successfully "distilled" into efficient static vectors. However, the method currently faces limitations, including a reliance on English-only data and predefined sense inventories like WordNet. If a word gains a new meaning not present in the dictionary, the model cannot capture it without manual updates. Future research will focus on extending CDES to multilingual sense embeddings and reducing the reliance on expensive sense-annotated data.
Summary updated