Cross-lingual Transfer Learning and Multitask Learning for Capturing Multiword Expressions

Shiva Taslimipoor, Omid Rohanian, Le An HaACL 2019

TL;DR

This paper tackles the challenge of identifying Multiword Expressions (MWEs), which are notoriously tricky due to their idiomatic nature. The authors propose two novel neural network approaches: multitask learning (MTL) that jointly trains MWE identification with dependency parsing, and cross-lingual transfer learning (TRL) to leverage data-rich languages for low-resource ones. Both methods significantly improve MWE identification performance compared to standard approaches.

What Problem Does This Paper Address?

Multiword Expressions (MWEs) are combinations of words that, as a unit, have a meaning different from the sum of their individual parts (think "kick the bucket" or "spill the beans"). This idiomatic nature makes them difficult for natural language processing (NLP) systems to identify and understand. While there's a growing interest in MWE identification, especially with the availability of multilingual datasets like PARSEME, some languages within these datasets are significantly under-resourced. This data scarcity, coupled with the inherent complexity of MWEs (like discontinuous or variable forms), poses a significant challenge for building robust MWE identification models. Existing neural approaches often struggle with these issues, necessitating new methods to enhance performance, particularly for low-resource languages.

Methods

The paper introduces two main neural network architectures:

  • Multitask Learning (MTL): This approach leverages the idea that learning related tasks simultaneously can improve performance on each. Here, the authors jointly train a model to predict MWE tags and dependency parse arcs and labels. The core idea is to use dependency parsing as an auxiliary task, with its supervision feeding into lower layers of the neural network, while MWE tagging is handled in the upper layers. This allows the model to learn richer representations by exploiting the syntactic information inherent in dependency structures.
  • Cross-Lingual Transfer Learning (TRL): Recognizing the data scarcity issue for languages like English in the PARSEME dataset, the authors propose transferring knowledge from a resource-rich language (German) to a resource-poor one. The model is first trained on a larger German MWE dataset, and then its learned weights are transferred to an identical model that is fine-tuned on the English data. This aims to overcome the limitations of insufficient training data by leveraging the generalizable linguistic knowledge acquired from the source language.

The underlying neural architecture combines Convolutional Neural Networks (CNNs) for feature extraction and Bidirectional Long Short-Term Memory (BiLSTM) layers, a common and effective choice for sequence tagging tasks.

Datasets

The primary dataset used is the PARSEME project's multilingual dataset of verbal MWEs. The authors specifically focus on the English portion, which they note is surprisingly small (3,471 training and 3,965 test sequences). For the transfer learning experiments, they utilize the German portion of the PARSEME dataset, which is significantly larger (6,734 training sequences).

Key Insights and Ideas

  • Dependency Parsing as a Powerful Auxiliary Task: Jointly training MWE identification with dependency parsing significantly boosts performance. This highlights that syntactic structure is a crucial signal for understanding MWEs, and explicitly modeling it as an auxiliary task provides valuable supervision.
  • Layered Supervision is Key: The MTL architecture's success in improving MWE prediction when dependency parsing supervision is introduced in lower layers suggests that the network can effectively learn to disentangle and utilize information from different tasks at different levels of abstraction.
  • TRL is a Viable Solution for Low-Resource MWEs: Transferring knowledge from a higher-resource language (German) to a lower-resource one (English) provides a substantial improvement over training solely on the limited English data, demonstrating the effectiveness of TRL in this domain.
  • ELMo Embeddings Provide a Strong Baseline: Even in the standard single-task setting, using pre-trained ELMo embeddings provides a strong foundation, indicating that these contextualized embeddings already capture a significant amount of linguistic knowledge relevant to MWE identification.

Key Results

  • MTL Outperforms Standard Approaches: The MTL model, which jointly predicts MWE tags and dependency parse information, achieved higher F1-scores (both token-based and MWE-based) compared to a single-task baseline (STL) and a model that simply used dependency parse information as additional input features. For example, the MTL model with ELMo+POS achieved an MWE-based F1 of 36.61 ± 1.27, outperforming the STL with ELMO+POS+DEP at 33.68 ± 2.99.
  • TRL Shows Promise: The cross-lingual transfer learning approach (TRL) also showed improvements over the standard single-task learning (STL) on the English data, although it was slightly lower than the best MTL results. This is attributed to the pre-trained ELMo embeddings already providing substantial knowledge.
  • MTL is Efficient: Despite its complexity, the MTL model's training time was comparable to the single-task model, taking approximately 45 minutes for 100 epochs on their hardware setup.

Limitations and Future Work

  • Data Scarcity Remains a Challenge: While TRL helps, the inherent data scarcity for certain languages remains a fundamental challenge. The paper notes that the TRL performance was only slightly better than STL, suggesting that ELMo embeddings already carry significant knowledge.
  • Further Exploration of TRL: The authors suggest that more investigation is needed into TRL, particularly in scenarios where extensive external data like ELMo is not available. They performed a preliminary experiment with a randomly initialized embedding layer (closed STL) which showed the benefits of TRL more clearly, but more research is needed.
  • Hyperparameter Tuning in TRL: The paper mentions the sensitivity of TRL to learning rates and the ongoing debate on how to best tune hyperparameters during transfer. They opted not to alter the learning rate due to observed performance declines, but this is an area for future exploration.
  • Extensive Comparisons: The authors plan to conduct more detailed studies and extensive comparisons between MTL and TRL to better understand their respective strengths and weaknesses under various circumstances for MWE identification.

Summary updated

BibTeX
@inproceedings{taslimipoor2019cross,
  author = {Shiva Taslimipoor and Omid Rohanian and Le An Ha},
  title = {{Cross-lingual Transfer Learning and Multitask Learning for Capturing Multiword Expressions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure