TL;DR
This paper introduces "Veyn," a recurrent neural network (RNN) based system for automatically identifying verbal multiword expressions (VMWEs). Veyn achieved competitive results in the PARSEME Shared Task 2018, demonstrating the effectiveness of RNNs for this challenging NLP task, particularly when using an extended BIO tagging scheme that incorporates VMWE categories.
What Problem Does This Paper Address?
Identifying multiword expressions (MWEs) is crucial for many Natural Language Processing (NLP) tasks, such as machine translation, where a literal translation of an MWE can lead to nonsensical results. Verbal multiword expressions (VMWEs) are a particularly challenging subset, often involving verbs combined with other words that form a single semantic unit (e.g., "give up," "take into account"). The PARSEME Shared Task 2018 focused on the automatic identification of these VMWEs across multiple languages. The challenge lies in accurately detecting the boundaries of these expressions, which can be discontinuous, have variants, and be ambiguous. Existing systems often relied on traditional machine learning models with discrete features, while the authors aimed to leverage the power of modern deep learning approaches, specifically RNNs, for this task.
Methods
The Veyn system is built upon a sequence tagging architecture using recurrent neural networks (RNNs). The core innovation lies in the tagging scheme and the network architecture:
- Tagging Scheme: Veyn employs a variant of the Begin-Inside-Outside (BIO) encoding scheme. Crucially, it extends this by incorporating the VMWE category tag directly into the BIO tags (BIO+cat). This allows the model to not only identify the boundaries of VMWEs but also their specific type (e.g., LVC.full, VID). They also experimented with an Inside-Outside + category (IO+cat) scheme.
- Network Architecture: The system uses a double bidirectional Gated Recurrent Unit (GRU) layer. Input features are derived from the LEMMA and UPOS (Universal Part-of-Speech) tags of each token, which are converted into one-hot vectors and then transformed into dense embeddings. These embeddings are pre-initialized on the training corpora and then fine-tuned. The outputs of the GRU layers are fed into a softmax layer to predict the probability of each tag.
- Pre-initialization: To improve performance, the input embeddings (for LEMMA and UPOS) were pre-trained using word2vec on the training corpus. This was done to leverage linguistic information even without explicit surface form embeddings.
Datasets
The system was trained and evaluated on the PARSEME Shared Task 2018 datasets. These datasets cover 19 languages, with training, development, and test splits. The authors specifically mention using train.cupt for training, dev.cupt for tuning tagging schemes, and test.blind.cupt for final predictions.
Key Insights and Ideas
- BIO+cat is effective: Incorporating VMWE category information directly into the tagging scheme (BIO+cat) generally improved performance, especially for the MWE-based evaluation, suggesting that explicit category information is beneficial for the model.
- Corpus size matters for tagging schemes: The optimal tagging scheme (BIO vs. BIO+cat) was found to be dependent on the size of the training corpus. Larger corpora benefited more from the richer BIO+cat scheme, while smaller ones sometimes saw degradation.
- Pre-initialized embeddings help: Pre-training input embeddings (LEMMA and UPOS) using word2vec significantly improved the model's performance on average, highlighting the value of leveraging external linguistic knowledge.
- Discontinuous VMWEs are hard: The system performed significantly better on continuous VMWEs than on discontinuous ones, suggesting that more sophisticated models might be needed to handle complex structural dependencies.
- RNNs can generalize to unseen VMWEs: The authors observed instances where Veyn could correctly identify VMWEs that were not explicitly present in the training data, suggesting that the RNN's ability to learn patterns from related expressions allows for some generalization.
Key Results
- Veyn was ranked ninth (eighth) on the average MWE-based (Token-based) F1 score across all languages in the official PARSEME Shared Task 2018 ranking.
- The system achieved higher F1 scores for VMWEs that were seen in the training corpus (identical or variants) compared to unseen ones.
- Veyn performed better on multi-token expressions (48.58% MWE-based F1) than on single-token expressions (9.23% MWE-based F1).
- A general trend showed a ~10% higher F1 score on the development corpus compared to the test corpus, which the authors attribute to factors like low performance on languages without development sets and potential issues with random initialization.
Limitations and Future Work
- Discontinuous VMWEs: The current system struggles with discontinuous VMWEs, and the authors suggest that more advanced models (e.g., tree-based or graph-based) might be necessary.
- Heuristic limitations: The post-processing heuristics used to reconstruct VMWE annotations from predicted tags sometimes led to errors, particularly in cases of overlapping expressions or when POS and syntactic dependencies could have provided crucial information.
- Overlapping VMWEs: Veyn cannot handle overlapping VMWEs because it predicts only one tag per token. This is a rare but present phenomenon in many languages.
- Generalization to unseen VMWEs: While some generalization was observed, improving the system's ability to identify VMWEs never seen in training remains a key area for improvement.
- Initialization robustness: The authors noted that the system's training procedure sometimes fails, predicting no tags at all. They plan to investigate this issue and develop automatic ways to prevent it.
- External resources: Integrating external lexical resources and pre-training embeddings on larger, non-annotated corpora are suggested as ways to improve coverage and performance.
Summary updated