TL;DR
This paper introduces the Noun Compound Type and Token Idiomaticity (NCTTI) dataset, a novel resource for evaluating how well computational models understand idiomatic language. They found that current state-of-the-art contextualized models still struggle to capture the nuances of idiomaticity as perceived by humans, especially when considering context-dependent meanings.
What Problem Does This Paper Address?
Understanding idiomatic expressions, like "kick the bucket" or "spill the beans," is a long-standing challenge in Natural Language Processing (NLP). These expressions often have meanings that cannot be deduced from the individual words. A key difficulty is that idiomaticity can exist at two levels:
- Type Level: Does an expression generally have an idiomatic meaning? For example, "olive oil" is always literal, while "shrinking violet" is generally idiomatic.
- Token Level: In a specific instance or sentence, is the expression being used idiomatically or literally? For example, "brass ring" can refer to a literal ring made of brass or a metaphorical prize.
Most existing datasets focus on the type level, making it hard to evaluate how well models handle the contextual variability of idiomaticity. This paper aims to bridge this gap by creating a dataset that captures both type and token level idiomaticity and using it to assess current NLP models.
Methods
The authors introduce the Noun Compound Type and Token Idiomaticity (NCTTI) dataset. This dataset was built upon existing resources (NC Compositionality dataset) and expanded with:
- Human Annotations: Participants rated noun compounds (NCs) on a scale of 0 (fully idiomatic) to 5 (fully compositional) for both their general (type) meaning and their meaning within specific sentences (token).
- Contextualized Evaluation: They adapted existing methods for evaluating compositionality with static word embeddings to contextualized models like BERT. This involved comparing the representation of an NC in a sentence with representations of the NC out of context and with the sum of its components' representations.
- Variability Analysis: They investigated whether models could capture the differences in how humans perceive an NC's idiomaticity across different sentences. This was done by comparing the variability of human scores with the variability of model-generated representations.
- Paraphrase Collection: Annotators also provided synonyms or paraphrases for the NCs in context, which can be useful for tasks like lexical substitution or improving machine translation.
Datasets
- NCTTI Dataset (New): This is the primary contribution. It contains:
- 280 noun compounds in English and 180 in Portuguese.
- Human annotations for idiomaticity at both type and token levels.
- 8,725 token-level annotations for English and 5,091 for Portuguese.
- Paraphrases for NCs at both type and token levels.
- A balanced distribution of compositional, partially compositional, and idiomatic items.
- NC Compositionality dataset (Source): Used as the basis for NCTTI, providing initial type-level compositionality scores.
- ukWaC and brWaC corpora: Used for obtaining frequency data for NCs and their components.
Key Insights and Ideas
- Idiomaticity is nuanced: The strong correlation between type-level and token-level human judgments highlights that while general tendencies exist, the specific context significantly influences how idiomatic an expression is perceived.
- Contextualized models still lag: Despite advances, current state-of-the-art models like BERT and ELMo struggle to consistently capture the fine-grained, context-dependent nuances of idiomaticity that humans perceive.
- Type-level vs. Token-level is crucial: Evaluating models solely on type-level idiomaticity misses a significant aspect of language understanding, particularly the ability to disambiguate literal vs. idiomatic uses in context.
- Paraphrases are valuable: The collected paraphrases offer a rich resource for various NLP tasks, potentially enhancing datasets for lexical substitution and machine translation.
Key Results
- Inter-annotator agreement: The agreement among human annotators was classified as "fair" for English and "moderate" for Portuguese, indicating a reasonable level of consistency in their judgments.
- Model performance: Contextualized models showed only weak to moderate correlations with human judgments for compositionality prediction, often lower than previous results with static word embeddings.
- Variability capture: The study found that models' representations did not significantly correlate with the variability in human idiomaticity judgments across different sentences for the same NC. This suggests models aren't fully capturing the contextual shifts in meaning.
- Specific NC examples: The paper notes that for NCs like "brick wall" and "gold mine," BERT showed higher cosine similarities for idiomatic meanings than for idiomatic vs. compositional senses, hinting at some ability to distinguish senses.
Limitations and Future Work
- Limitations:
- The authors acknowledge that current contextualized models, even the best ones, do not fully model semantic compositionality as humans do.
- The inter-annotator agreement, while acceptable, indicates room for further refinement in annotation guidelines or dataset construction.
- The study focused on noun compounds; other types of Multiword Expressions (MWEs) might behave differently.
- Future Work:
- Enriching the dataset with cross-lingual equivalents of NCs to build cross-lingual representations.
- Expanding the dataset with more sentences containing ambiguous NCs that have both idiomatic and literal interpretations depending on context.
- Developing better representations and measures for idiomaticity identification.
Summary updated