TL;DR
This paper introduces COLF-VID, a new German corpus for disambiguating verbal idioms (VIDs), and presents a BiLSTM-based neural network that achieves a significant 57% error reduction compared to a simple majority baseline. This work is crucial for improving natural language understanding systems, particularly in areas like machine translation, where accurately interpreting figurative language is essential.
What Problem Does This Paper Address?
Figurative language, especially verbal idioms (VIDs), poses a significant challenge for Natural Language Processing (NLP). While many VIDs are overwhelmingly used idiomatically, they can sometimes have literal interpretations. Accurately distinguishing between these literal and idiomatic meanings is vital for tasks like machine translation, sentiment analysis, and question answering. However, creating datasets for training and evaluating such disambiguation systems is difficult due to the need for high-quality, token-level annotations that capture both literal and figurative uses, along with sufficient contextual information. Existing datasets often suffer from class imbalance (predominantly idiomatic uses) or lack the necessary detail for robust model training.
Methods
The authors tackle the VID disambiguation problem using a supervised learning approach with a Bidirectional Long Short-Term Memory (BiLSTM) neural network architecture. The core idea is to leverage pre-trained word embeddings to represent words and then feed these representations through a BiLSTM to capture contextual information. Specifically, the model focuses on the verb and noun components of a potential VID. These contextualized representations are then concatenated and passed through a Multi-Layer Perceptron (MLP) to output scores for each possible reading (literal, idiomatic, undecidable, both). They experiment with three types of pre-trained word embeddings: Word2vec, fastText, and ELMo, hypothesizing that models incorporating subword information (fastText and ELMo) would perform better.
Datasets
The primary contribution in terms of data is the COLF-VID (Corpus of Literal and Figurative Readings of Verbal Idioms) corpus. This is a newly created German annotated corpus containing 6985 sentences with candidate occurrences of 34 preselected VID types. The corpus was specifically designed to include a higher proportion of literal occurrences than typically found in other datasets, aiming to alleviate data sparsity issues for training. The annotations were performed by three annotators, and the corpus includes labels for LITERAL, IDIOMATIC, UNDECIDABLE, and BOTH readings. For training and evaluation, the COLF-VID dataset was split into 70% for training, 15% for validation, and 15% for testing, with care taken to maintain the distribution of VID types across splits.
Key Insights and Ideas
- The importance of balanced datasets: The paper strongly emphasizes that the rarity of literal VID occurrences is a significant hurdle. Their approach of hand-picking VID types with potentially higher literal usage and creating the COLF-VID corpus directly addresses this, showing the necessity of balanced data for effective supervised learning.
- Subword information matters: The experiments highlight that word embeddings incorporating subword information (fastText and ELMo) significantly outperform Word2vec (trained on lemmas). This suggests that capturing morphological variations and word formation is crucial for understanding the nuances of VIDs, especially their fixedness.
- ELMo's contextual advantage: ELMo embeddings, which provide different representations for the same word based on its context, proved to be the most effective. This aligns with the intuition that context is paramount for disambiguation, and ELMo's ability to capture this early in the input pipeline provides a distinct advantage.
- The "BOTH" label is rare but insightful: While the "BOTH" label (indicating both literal and idiomatic readings are intended) was applied to very few instances, its inclusion acknowledges the linguistic complexity of wordplay and metonymy, where multiple interpretations can coexist.
Key Results
The best performing model, using ELMo embeddings with the BiLSTM architecture, achieved a weighted macro-average F1-score of 89.82% on the test set. This represents a 57% error reduction compared to a simple majority baseline (which achieved 76.95% F1-score). The ELMo-based model also showed superior performance on the validation set with an F1-score of 89.14%. While the system performed weaker on the literal class (max F1 of 79.07%), this is attributed to the inherent class imbalance in the dataset.
Limitations and Future Work
The authors acknowledge that the class imbalance remains a challenge, particularly for the UNDECIDABLE and BOTH labels, which the system struggled to classify correctly. They also note that the current setup assumes candidate expressions are pre-identified, and future work could extend the COLF-VID corpus to enable an identification task as well. They plan to explore a wider range of architectures, context sizes, and features for disambiguation and suggest cross-linguistic comparisons with corpora like IDIX to investigate language-specific VID properties.
Summary updated