TL;DR
This paper tackles the challenge of Word Sense Induction (WSI) by proposing a novel method to remove "nuisance" linguistic information (morphosyntactic features) from contextualized word embeddings. By training a "forget-gate" using an adversarial framework, they demonstrate that these "cleaned" representations lead to significantly improved WSI performance, suggesting that these features, while informative for language models, can actually hinder sense disambiguation.
What Problem Does This Paper Address?
Word Sense Induction (WSI) is the task of identifying distinct senses of a word within a corpus, often without pre-defined sense inventories. Modern approaches leverage contextualized word embeddings from large language models (LLMs) like BERT, which are powerful because they capture a word's meaning within its specific sentence. However, these embeddings are rich with various linguistic information, including syntactic roles (e.g., subject, object) and morphological features (e.g., tense, number). The authors hypothesize that this extra information, while useful for general language understanding, can actually act as "noise" for WSI. Specifically, they observe that instances of a word might be clustered together based on shared syntactic or morphological properties rather than their semantic similarity, leading to suboptimal WSI performance. This paper aims to investigate this hypothesis and develop a method to mitigate this issue.
Methods
The core of this paper is an adaptation of the adversarial training framework proposed by Jaiswal et al. (2020). The goal is to train a "forget-gate" (FG) that selectively removes unwanted information from pre-trained LLM embeddings.
Here's how it works:
- Feature Extractor: A pre-trained LLM (BERT in this case) is used solely as a feature extractor. Its parameters are frozen.
- Forget-Gate (FG): A feedforward neural network is added on top of the LLM's output. This FG takes the LLM's contextualized token embeddings as input and outputs a transformed embedding.
- Adversarial Training: The FG is trained within an adversarial setup. This involves:
- Discriminator (D): This module tries to predict the "nuisance" information (e.g., POS tag for morphological features, dependency label for syntactic features) from the FG's output.
- Predictor (P): This module tries to reconstruct the original token identity from the FG's output.
- Objective: The FG is trained to "fool" the discriminator (i.e., make it unable to predict the nuisance features) while still allowing the predictor to recover task-relevant information. This process results in "feature-invariant" representations.
- Targeted Erasure: The authors train two separate adversarial models: one to create morphology-invariant representations and another for syntax-invariant representations.
Datasets
- Training and Evaluation of Forget-Gates: The Brown Corpus (Kučera et al., 1967) was used. Token representations were extracted using BERT. For WSI evaluation, the authors specifically used SemCor. SemCor is a subset of the Brown Corpus where words are annotated with WordNet senses. The authors filtered SemCor to focus on nouns and verbs with multiple senses, and senses occurring at least 10 times.
Key Insights and Ideas
- Morphosyntactic Features as "Noise" for WSI: The central finding is that while LLMs encode rich linguistic information, features like morphology and syntax, which are not directly tied to semantic senses, can negatively impact WSI performance by causing instances to cluster based on non-semantic similarities.
- Adversarial Forgetting is Effective: The proposed adversarial forgetting method successfully creates representations that are invariant to specific linguistic features, and these cleaned representations lead to improved WSI performance.
- Morphological Features are More Disruptive: Morphological features appear to be more detrimental to WSI than syntactic features. Erasing them yields a larger performance gain.
- Contextualized Representations are Not Always Optimal "Out-of-the-Box": This work highlights that simply using the raw output of powerful LLMs might not be the best strategy for every downstream task, especially when those tasks require fine-grained semantic distinctions.
- Task-Specific Representation Engineering: The paper demonstrates the value of actively engineering representations by removing specific types of information to suit a particular task (WSI in this case).
Key Results
- Improved WSI Performance: Using both morphology-invariant and syntax-invariant representations resulted in statistically significant improvements in WSI performance (measured by Adjusted Rand Index - ARI) compared to using original BERT representations.
- Morphology-Invariant Representations Yielded Best Results: Overall, morphology-invariant representations achieved the highest WSI performance.
- Specific Gains: For example, in the evaluation of WSI performance on SemCor, the ARI score for morphology-invariant representations was 0.283, compared to 0.210 for original BERT representations.
- Analysis of Word Cases: The authors categorized words into three cases based on the association of their features with senses and BERT clusters. Case 2 words (where features are associated with BERT clusters but not senses) showed the most significant performance increase after feature erasure, confirming the "noise" hypothesis.
- PCA Visualizations: Visualizations of word instances in a reduced dimensional space (PCA) clearly showed that morphology-invariant representations lead to better separation of word senses compared to original BERT representations, especially for words like "area."
Limitations and Future Work
- Positional Information: The authors experimented with removing positional information but found it led to unexpected behavior in clustering, so they excluded it. They intend to investigate this further.
- Feature Interrelation: While morphological and syntactic features were targeted independently, they are theoretically interrelated. The authors acknowledge this but state it doesn't affect the method's relevance, only linguistic interpretations.
- Subword Tokenization: The method primarily uses the first subword token when a word is split by BERT's tokenizer, which might not capture the full nuance of the word.
- Future Work: The paper suggests further investigation into the underlying reasons for the issues with positional information and exploring the impact of other linguistic features on WSI. The authors also mention that while their method requires feature annotations for training the forget-gate, this annotation is done automatically, making the approach applicable to various datasets.
Summary updated