TL;DR
This paper investigates how different ways of representing words (lemmas vs. surface forms, word embeddings vs. subword embeddings) affect the performance of neural networks in identifying verbal multiword expressions (VMWEs). They found that subword embeddings (FastText) consistently outperform traditional word embeddings (word2vec), and that using both lemmas and surface forms generally yields the best results, especially for morphologically rich languages.
What Problem Does This Paper Address?
Identifying multiword expressions (MWEs) in text, especially verbal MWEs (VMWEs), is a challenging task in Natural Language Processing (NLP). VMWEs often exhibit significant morphological and syntactic variability, making them difficult for traditional NLP systems to handle. While recent advances in neural networks and the availability of annotated corpora have spurred progress, the impact of fundamental choices in word representation on VMWE identification performance remains an open question. Specifically, this paper addresses the trade-offs between using lemmatized forms versus inflected surface forms, and between traditional word embeddings versus subword embeddings (which can handle out-of-vocabulary words and morphological variations).
Methods
The authors employ a sequence tagging neural network architecture called "Veyn" (previously used in the PARSEME shared task) as their base system. This system takes word features (like lemmas and part-of-speech tags) as input and uses a Conditional Random Field (CRF) output layer to predict VMWE labels.
The core of their investigation lies in experimenting with different input representations:
- Feature Sets:
- Lemmas: The base dictionary form of words.
- Surface Forms: The actual inflected words as they appear in text.
- Form-Lemma: A combination of both surface forms and lemmas.
- Word Embeddings:
- word2vec: A standard word embedding model that learns representations based on word co-occurrence.
- FastText: An extension of word2vec that also incorporates character n-grams, allowing it to generate representations for out-of-vocabulary (OOV) words and better capture morphological information.
They trained 256-dimensional word2vec and FastText embeddings on the training corpora for each language.
Datasets
The experiments were conducted on the PARSEME shared task 1.1 corpora for three languages with varying morphological richness:
- Basque (EU)
- French (FR)
- Polish (PL)
These corpora are annotated for verbal multiword expressions and are split into training, development, and test sets. The paper notes that the French test corpus had some missing lemmas, which were handled by lemmatizing them using UD-Pipe to maintain comparability with the "closed track" conditions of the shared task.
Key Insights and Ideas
- Subword Embeddings Shine: FastText embeddings, which leverage character n-grams, consistently outperform word2vec embeddings across most configurations and languages. This highlights their ability to model morphological variations and handle OOV words more effectively.
- Lemmas are Crucial: Using lemmas as input features generally leads to better performance than using only surface forms, especially in morphologically rich languages. This suggests that abstracting away from inflectional variations is beneficial for VMWE identification.
- The Power of Combination: For morphologically rich languages like Basque, combining both lemmas and surface forms as input features yields the best results. This indicates that while lemmas provide a good base, surface forms can still offer valuable information.
- Morphological Richness Matters: The impact of different representations varies with the language's morphological complexity. For instance, the benefit of combining forms and lemmas is more pronounced in Basque than in French.
- OOV Handling is Key: The paper implicitly emphasizes the importance of handling Out-of-Vocabulary (OOV) words, which is a known challenge for NLP systems. FastText's ability to generate representations for OOV words is a key factor in its superior performance.
Key Results
- FastText Dominance: For French and Basque, FastText embeddings provided a significant improvement over word2vec across all feature set combinations. For example, in Basque, the best FastText model achieved an F-MWE score of 69.24, compared to 60.56 with word2vec.
- Form-Lemma Best for Basque: The form-lemma combination with FastText achieved the highest F-MWE score (69.24) for Basque.
- Polish Nuances: In Polish, word2vec performed slightly better on the token-based F-measure (F-TOK), while FastText was better for the MWE-based F-measure (F-MWE). This was attributed to word2vec predicting more (potentially spurious) single-token VMWEs, increasing recall but decreasing precision.
- Top-3 Ranking: The best models developed in this study would have ranked in the top-3 of the official PARSEME shared task for the closed track.
Limitations and Future Work
- OOV Handling: The authors acknowledge the high OOV rate, particularly for verbs, and suggest that future work should focus on improving OOV handling.
- Advanced Embeddings: They propose exploring recent contextual embeddings like BERT, Elmo, and Flair, which are pre-trained on large external corpora, as a potential avenue for better OOV handling and overall performance.
- Post-processing: For languages like Polish where word2vec's tendency to over-predict single-token VMWEs can be an issue, they suggest that simple post-processing techniques (e.g., grouping adjacent single-token predictions) could help compensate for this.
Summary updated