Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy

Danushka Bollegala, Yi Zhou

Abstract

Contextualised word embeddings from Neural Language Models (NLMs) like BERT represent words by considering both the target word and its surrounding context. In contrast, static word embeddings like GloVe offer memory- and compute-efficient representations but fail to distinguish between different word senses. We propose Context Derived Embeddings of Senses (CDES), a method that extracts sense-related information from contextualised embeddings and injects it into static embeddings to create sense-specific static vectors. Experimental results on multiple Word Sense Disambiguation (WSD) and sense discrimination benchmarks show that CDES achieves performance comparable to state-of-the-art sense embeddings while maintaining the efficiency of static models.

Introduction: The Embedding Trade-off

In Natural Language Processing (NLP), researchers typically choose between two paradigms of word representation:

  1. Static Embeddings (e.g., GloVe): These assign a single fixed vector to each word. While fast and memory-efficient, they are "sense-agnostic," meaning they cannot distinguish between a "bank" as a financial institution and a "bank" as a river edge.
  2. Contextualised Embeddings (e.g., BERT): These generate dynamic vectors based on the surrounding sentence. They capture nuance brilliantly but are computationally expensive, requiring the full model to run during inference.

The challenge lies in creating sense-specific static embeddings: multiple fixed vectors for a single word (one for each meaning) that are as accurate as BERT but as lightweight as GloVe. Previous attempts often struggled with data sparsity, as there is rarely enough sense-tagged text to train high-quality vectors for every possible meaning of every word.

Methodology: The CDES Framework

CDES treats the creation of sense embeddings as a form of model distillation, using a complex "teacher" (BERT) to train a simpler "student" (sense-specific static vectors).

The CDES architecture: A generic static embedding is projected into a sense-specific space and aligned with a filtered contextualized embedding using a non-linear mapping.

Sense-Specific Projection

For a word $u$ and its $i$-th sense $s_i$, the model learns a diagonal projection matrix $A_i$. This matrix transforms the generic static embedding $g(u)$ into a sense-specific vector $s_i(u)$. The authors found that using diagonal matrices rather than full matrices significantly reduced parameters and prevented overfitting without sacrificing accuracy.

The Objective Function and Non-linearity

The system is trained to minimize the squared L2 distance between the transformed static vector and the contextualised embedding from BERT. Crucially, the authors found that the relationship between these spaces is complex; using a non-linear activation function (specifically GELU) to align the two spaces significantly improved performance over simple linear mapping.

Context Aggregation and Filtering

To combat data sparsity, CDES incorporates extra information from dictionary definitions (glosses) in WordNet and clusters sentences from Wikipedia to find representative contexts for rare senses. Additionally, because contextualised embeddings contain "noise" (grammar, gender, etc.) unrelated to word sense, CDES uses a learnable filter matrix ($W$) to strip away this extra info, focusing purely on semantic sense.

Evaluation and Key Results

The model was evaluated using standard benchmarks, including the SemCor 3.0 corpus for training and a suite of "all-words" WSD tasks (Senseval-2/3, SemEval-07/13/15). It was also tested on the Words in Context (WiC) dataset to measure its ability to discriminate between subtle shifts in meaning.

Performance Highlights

  • WSD Accuracy: CDES reported the best performance in 4 out of 6 benchmarks compared to other sense-embedding methods, matching or beating models with much more complex architectures.
  • Sense Discrimination: It outperformed existing static sense embeddings like LMMS and LessLex on the WiC task.

Visual Proof

t-SNE visualizations comparing GloVe, ARES, and CDES. CDES creates significantly tighter and more distinct clusters for different senses of the word "bank," clearly separating financial (red) and geographic (blue) meanings. Source, p. 7

t-SNE visualizations comparing GloVe, ARES, and CDES. CDES creates significantly tighter and more distinct clusters for different senses of the word "bank," clearly separating financial (red) and geographic (blue) meanings.

As shown in the t-SNE visualizations above, CDES produces much cleaner separation between senses than traditional methods. For the word "bank," it successfully clusters financial terms like "payment" and "savings" away from geographic terms like "coast" and "riverbank."

Discussion and Future Work

CDES demonstrates that BERT's contextual knowledge can be successfully "distilled" into efficient static vectors. However, the method currently faces limitations, including a reliance on English-only data and predefined sense inventories like WordNet. If a word gains a new meaning not present in the dictionary, the model cannot capture it without manual updates. Future research will focus on extending CDES to multilingual sense embeddings and reducing the reliance on expensive sense-annotated data.

Summary updated

BibTeX
@inproceedings{bollegala2021learning,
  author = {Danushka Bollegala and Yi Zhou},
  title = {{Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy}},
  booktitle = {Paclic},
  year = {2021},
}
Figure