TL;DR
This paper investigates how word polysemy (words having multiple meanings) impacts semantic categorization tasks. They found that sense-specific word embeddings significantly outperform static embeddings for coarse-grained categorization, suggesting that capturing distinct word senses is crucial for this task. However, for fine-grained categorization based on similarity judgments, static and sense-specific embeddings perform similarly, indicating different cognitive processes are at play.
What Problem Does This Paper Address?
Semantic categorization, the task of grouping words into meaningful categories, is a fundamental aspect of human language understanding. While word embeddings (numerical representations of words) have shown great success in tasks like similarity judgments, their effectiveness in semantic categorization, especially when dealing with polysemous words (words with multiple meanings), is less understood. Previous work suggested that static embeddings (which represent a word with a single vector regardless of context) outperformed contextual embeddings (which generate different representations based on context) in semantic categorization. This paper questions this finding, hypothesizing that the loss of sense-specific information in static embeddings might be the reason for their apparent superiority, and that polysemy is a key factor that needs to be accounted for.
Methods
The researchers evaluated several word embedding models on the SpA-Verb dataset, which includes both coarse-grained (verb sorting) and fine-grained (spatial multi-arrangement similarity judgments) categorization tasks.
- Word2vec: A standard static word embedding model, trained on POS-tagged data to distinguish between senses with different parts of speech.
- BERT (Static): Two methods were used to extract static embeddings from BERT:
- Decontextualized (Decont): Feeding each word in isolation to BERT to get a single, context-free representation.
- Aggregated (Aggr): Averaging BERT embeddings across 100 different contexts for each word.
- BERT (Sense-Specific):
- Multiprototype (MPro) BERT: This novel approach distills token-level BERT embeddings into multiple prototype embeddings, with each prototype representing a distinct sense of a word. This was achieved by clustering token embeddings for each verb and using the number of senses listed in WordNet to determine the number of clusters (k).
The evaluation used k-means clustering to group verbs into predicted classes and measured performance using modified purity (precision) and weighted class accuracy (recall), combined into an F1 score.
Datasets
The primary dataset used is SpA-Verb, a benchmark for semantic categorization. It consists of:
- Phase 1: Contains 825 verbs grouped into 17 broad semantic classes, derived from a word sorting task by non-expert native speakers. This task is used for coarse-grained categorization.
- Phase 2: Provides spatial multi-arrangement similarity data for words within each Phase 1 class. This data is used to derive fine-grained categories based on similarity judgments.
The authors note that 116 verbs in the dataset are polysemous, belonging to more than one class.
Key Insights and Ideas
- Polysemy is Crucial for Coarse-Grained Categorization: The most significant finding is that explicitly modeling word senses (using MPro BERT) dramatically improves performance on coarse-grained categorization tasks. This suggests that when humans perform broad categorization, they are sensitive to the different meanings a word can have.
- Contextual Information Matters, But Needs to be Harnessed: The underperformance of contextual models in previous work (Majewska et al., 2021) was likely due to the way their sense information was discarded or averaged out. MPro BERT demonstrates that BERT's rich contextual information, when properly leveraged to represent distinct senses, is highly beneficial.
- Different Tasks, Different Cognitive Processes: The stark difference in performance between coarse-grained and fine-grained categorization highlights that humans likely employ different cognitive strategies for these tasks. Coarse-grained categorization seems to benefit from sense disambiguation, while fine-grained similarity judgments might rely on a more holistic representation of a word's meanings.
- Static Embeddings Can Be Sufficient for Similarity: For fine-grained categorization, which is based on similarity judgments, static embeddings (especially aggregated BERT) perform comparably to sense-specific embeddings. This suggests that for tasks where a general sense of similarity is needed, averaging over contexts might be adequate.
Key Results
- Coarse-Grained Categorization: MPro BERT achieved an F1 score of 0.743 (optimal) / 0.687 (gold), more than doubling the F1 score of the best previously reported BERT results (Majewska et al., 2021). This significantly outperformed static BERT models (Decont. BERT: 0.309/0.191, Aggr. BERT: 0.398/0.346) and Word2vec (0.355/0.326).
- Fine-Grained Categorization: In contrast, MPro BERT's F1 score dropped to 0.582, performing similarly to aggregated BERT (0.643) and decontextualized BERT (0.586). Word2vec performed best with an F1 of 0.626.
- BERT Layer Analysis: For MPro BERT, embeddings from later layers of BERT were better predictors of coarse-grained categories, while earlier layers performed better for fine-grained categories.
Limitations and Future Work
- Inter-Annotator Agreement: The authors acknowledge that the low inter-annotator agreement in the Phase 1 sorting task (0.400 B-Cubed score) suggests that human categorization can be subjective, meaning that induced categories might be reasonable even if they don't perfectly match the ground truth.
- Replicating Phase 1 for Each Class: To make a more direct comparison between coarse- and fine-grained categorization, the authors plan to replicate the Phase 1 sorting task for each individual semantic class.
- Choosing the Right Model: The paper emphasizes that the choice of embedding model should depend on the specific task and the type of human behavior being modeled. Static representations might be preferred for cross-linguistic or historical comparisons, while sense-specific embeddings are better for category induction.
Summary updated