TL;DR
This paper introduces a novel dataset and evaluation framework for understanding how language models handle polysemy (words with multiple meanings). They demonstrate that while contextualized models like XGLM show promise, capturing the nuances of polysemy remains a significant challenge, even for state-of-the-art models.
What Problem Does This Paper Address?
Polysemy, the phenomenon of words having multiple related meanings (e.g., "book" as an object vs. its content), has long been a challenge for Natural Language Processing (NLP). Traditional NLP approaches, like those in WordNet, often treat each sense as a distinct entity. However, psycholinguistic research suggests a more continuous, "one-representation" view where different senses are facets of a single underlying meaning. This discrepancy means that current NLP evaluation datasets and methods may not fully capture how well models understand polysemy. Existing datasets often rely on the "sense enumeration" view, limiting our ability to assess models' understanding of the subtle, continuous variations in meaning that characterize polysemy. This paper aims to bridge this gap by evaluating models using psycholinguistically motivated variables.
Methods
The authors propose a hybrid approach to polysemy, viewing it as a continuous variation in psycholinguistic properties rather than discrete senses. They collected a dataset of verb-noun phrases where polysemous nouns are used in either concrete or abstract contexts. These phrases were then annotated by human participants for concreteness and sensory strength (involvement of sight, hearing, touch, smell, taste).
To evaluate language models, they employed a two-pronged approach:
- Correlation Analysis: They trained Ridge regression models to predict human ratings of concreteness and sensory strength from word embeddings generated by various language models. This measures how well models can predict these psycholinguistic properties.
- Sense Discrimination Analysis: This method, adapted from cognitive neuroscience, assesses a model's ability to distinguish between the two senses of a polysemous word. It quantifies how well a model can differentiate between the predicted ratings for concrete vs. abstract uses of a word.
They evaluated four types of models:
- Count-based (count-pmi): A baseline model using Pointwise Mutual Information on word co-occurrence counts.
- Static (fastText): A pre-trained word embedding model that generates a single vector for each word, regardless of context.
- Graph-based (ConceptNet Numberbatch): A model that combines distributional information with graph-based semantic knowledge.
- Contextualized (XGLM): A state-of-the-art multilingual transformer model that generates context-dependent word representations.
Datasets
The paper introduces a newly collected English dataset specifically designed for evaluating polysemy. This dataset consists of:
- 25 polysemous nouns (e.g., "book," "newspaper").
- For each noun, two verbs eliciting a concrete sense and two verbs eliciting an abstract sense, forming 100 verb-noun phrases.
- Human ratings for concreteness and five sensory modalities (sight, hearing, touch, smell, taste) for these phrases.
The authors also utilized the UKWaC corpus for extracting verb-noun co-occurrences and for training/evaluating models. For word frequencies, they used WordNet and CoreLex for noun selection and Brysbaert et al.'s (2014) concreteness ratings for verb selection.
Key Insights and Ideas
- Polysemy is a continuous spectrum, not discrete categories: The paper advocates for and tests a framework that treats polysemy as a gradient of psycholinguistic properties, aligning with cognitive science findings.
- Sensory strength is crucial for understanding polysemy: Beyond simple concreteness, the involvement of different senses (sight, touch, etc.) provides a more nuanced way to characterize meaning variations.
- Contextualized models are better, but not perfect: While models like XGLM outperform static models, they still struggle to fully capture the subtle distinctions in polysemous meanings.
- Smaller models can be surprisingly effective: The study found that a smaller version of XGLM (1.7B parameters) often performed as well as or better than larger variants and static models, suggesting that sheer model size isn't the only factor for capturing complex semantic phenomena.
- The dataset is a valuable resource: The newly created dataset, annotated with psycholinguistic variables, can be used for both computational and cognitive experiments, facilitating research at the intersection of NLP and cognitive science.
Key Results
- Correlation: XGLM (best performing layer) achieved the highest Pearson correlation scores for predicting concreteness and most sensory variables, with an overall correlation of 0.771. Static models like fastText and count-pmi performed comparably, with overall correlations around 0.72-0.749, suggesting that simpler approaches can still be effective.
- Sense Discrimination: XGLM significantly outperformed other models in distinguishing between concrete and abstract senses, with an overall accuracy of 0.672 (compared to a chance level of 0.5). However, even XGLM's performance was modest, indicating the difficulty of this task.
- Model Similarity: Representational Similarity Analysis (RSA) showed that static models (count-pmi, fastText, Numberbatch) had higher pairwise similarities, indicating they represent meaning in a more similar fashion. XGLM, as a contextualized model, exhibited a distinct representational space.
Limitations and Future Work
- Dataset Size: The authors acknowledge that their dataset is relatively small and focuses on only one type of regular polysemy (concrete-abstract alternation).
- Broader Polysemy Types: Future work could expand the dataset to include other types of polysemy (e.g., "chicken" for animal vs. food).
- Human-Model Alignment: Investigating the alignment between language model processing of polysemes and human cognition, potentially using brain data, is a promising future direction.
- Low-Resource Languages: The methodology could be adapted for low-resource languages, and the dataset could be expanded to automatically induce word senses in corpora.
Summary updated