TL;DR
This paper investigates how well different word embedding models capture the meaning of non-compositional multiword expressions (MWEs). Surprisingly, older models like word2vec outperform newer contextualized models like BERT and ELMO, suggesting that current advanced embeddings struggle with the nuances of idiomatic language.
What Problem Does This Paper Address?
Multiword expressions (MWEs) are phrases where the meaning of the whole is not easily predictable from the meaning of its individual words (e.g., "kick the bucket"). Understanding how well computational models can grasp this "non-compositionality" is crucial for natural language processing tasks like machine translation, sentiment analysis, and question answering. While modern embedding models (like BERT and ELMO) have shown great success on many NLP tasks, their ability to handle the subtle, often idiomatic meanings of MWEs remains less understood. Existing methods for evaluating this often require extensive manual annotation or specific preprocessing, limiting their applicability. This paper aims to provide a more comprehensive evaluation of various embedding techniques on MWE compositionality prediction.
Methods
The authors evaluate several embedding models using three main approaches to measure MWE compositionality:
- Direct Composition: This method compares the embedding of the MWE itself with the embeddings of its constituent words. The intuition is that if an MWE is compositional, its meaning (and thus its embedding) should be similar to a combination of its parts. They explore two ways of combining component word embeddings: an element-wise sum and a weighted sum.
- Paraphrase Similarity: This approach leverages existing paraphrases for MWEs. If an MWE is compositional, its embedding should be similar to the embeddings of its paraphrases, as paraphrases often capture the intended meaning.
- Combined Metric: A weighted combination of the direct composition and paraphrase similarity scores, allowing for a more robust evaluation.
They compare a range of embedding models, including:
- Word-level: word2vec, fastText
- Character-level: fastText (which can handle out-of-vocabulary words)
- Contextualized: ELMO, BERT, Flair (a contextualized string embedding model)
- Document-level: infersent, doc2vec
Datasets
The study uses three existing datasets for evaluating MWE compositionality:
- REDDY: Contains 90 binary English noun compounds with human-annotated scores for overall compositionality.
- RAMISCH: Similar to REDDY, with 90 binary English noun compounds and their compositionality scores. It also includes paraphrases for each MWE.
- DISCOADJ: A larger dataset (144 adjective-noun pairs) from a shared task, with compositionality ratings collected from crowdworkers.
The authors note that the DISCOADJ dataset is more skewed towards compositional usages compared to REDDY and RAMISCH, which have a more balanced distribution.
Key Insights and Ideas
- word2vec Reigns Supreme: Despite the advancements in deep learning and contextual embeddings, the older, simpler word2vec model surprisingly performs the best across several metrics, particularly for direct composition.
- Contextual Embeddings Lag: Newer, powerful models like BERT and ELMO, while excellent for many NLP tasks, do not excel at capturing the non-compositionality of MWEs in this study. This suggests that their contextual understanding, while deep, might not be directly translating to understanding idiomatic phrases.
- Paraphrase Power: The paraphrase similarity metric generally yields higher correlation scores across most models compared to direct composition. This highlights the value of having explicit interpretations (paraphrases) for MWEs and suggests that this approach might be more robust.
- Preprocessing Matters (for word2vec): The success of word2vec in this study was partly due to specific preprocessing steps, such as concatenating MWE components into single tokens. This manual intervention is a limitation for models that don't require it.
- Character-level and Document-level Show Promise: Models like fastText (character-level) and infersent (document-level) show competitive results, especially with paraphrase data, and importantly, they don't require extensive MWE-specific preprocessing.
Key Results
- RAMISCH Dataset: word2vec achieved the highest scores for direct composition metrics. infersent outperformed other models on paraphrase metrics.
- REDDY Dataset: word2vec again performed best for direct composition.
- DISCOADJ Dataset: The performance gap between models was smaller here, but word2vec still showed strong results.
- Overall: The paper concludes that simple word2vec, with specific preprocessing, is more adept at capturing MWE non-compositionality than modern contextualized embeddings like BERT and ELMO. fastText and infersent are noted as promising alternatives among newer models.
Limitations and Future Work
- Preprocessing Dependency: The strong performance of word2vec is tied to specific preprocessing steps that are not required by other models. This makes direct comparison slightly nuanced.
- Hyperparameter Tuning: The authors acknowledge that they did not tune hyperparameters (like α and β) on held-out data, which could lead to overfitting.
- Future Work: The authors plan to tune hyperparameters on held-out data, experiment with other languages, and explore language-independent techniques. They also suggest that further research into paraphrase-based metrics could be fruitful, despite the scalability limitations of requiring paraphrase data.
Summary updated