TRAPACC and TRAPACCS at PARSEME Shared Task 2018: Neural Transition Tagging of Verbal Multiword Expressions

Regina Stodden, Behrang QasemiZadeh, Laura KallmeyerCOLING 2018

TL;DR

This paper introduces TRAPACC and TRAPACCS, novel neural transition tagging systems for identifying verbal multiword expressions (VMWEs) across 19 languages. By combining a dimension reduction technique with Convolutional Neural Networks (CNNs) and optionally a Support Vector Machine (SVM), they achieved top performance in 8 languages for the PARSEME shared task, demonstrating the effectiveness of their approach for multilingual VMWE identification.

What Problem Does This Paper Address?

Identifying verbal multiword expressions (VMWEs) – phrases where the meaning isn't simply the sum of their parts (e.g., "kick the bucket" for "to die") – is a challenging task in Natural Language Processing (NLP). Existing methods often struggle with the multilingual nature of language and the complexity of VMWE structures. The PARSEME shared task aims to advance automatic VMWE identification across many languages. This paper tackles the problem of building a robust and accurate system for this task, particularly within the constraints of a "closed track" where participants can only use provided training data and resources. They aim to improve upon existing transition-based parsing methods by incorporating deep learning techniques.

Methods

The core of TRAPACC and TRAPACCS is a modified arc-standard transition system, building on the MAST (Modified Arc-Standard Transition) model. This system parses sentences by making a sequence of decisions (transitions) that move words between a buffer and a stack, ultimately building a representation of the sentence's structure and identifying VMWEs.

What's novel here is how they handle the classification of these transitions:

  • Dimension Reduction: Instead of directly feeding high-dimensional, sparse feature vectors (derived from word forms, lemmas, POS tags, etc.) into a classifier, they first apply a data-independent dimensionality reduction method based on random projections. This is crucial for deep learning models, as it reduces memory requirements and training time without losing essential information.
  • Convolutional Neural Networks (CNNs): TRAPACC uses a CNN to classify the reduced-dimension feature vectors and predict the next transition. This allows the model to learn complex patterns from the input features.
  • TRAPACCS Extension: TRAPACCS further enhances TRAPACC by replacing the final softmax layer of the CNN with a kernel SVM. This leverages the CNN as a feature extractor, feeding its intermediate output to a more powerful classifier, inspired by research suggesting this combination can improve performance.

They also introduced several new features beyond those used in previous work, including the buffer size, token length, and four-grams, to better capture sentence and VMWE characteristics.

Datasets

The primary dataset used is the PARSEME corpus, which provides VMWE annotations for 19 languages. The authors utilized the training and development sets for training their models and the test set for evaluation, adhering to the "closed track" rules of the shared task. No new benchmarks were introduced; they participated in an existing shared task.

Key Insights and Ideas

  • Dimension Reduction is Key for Deep Learning: The paper highlights that for deep learning models like CNNs, reducing the dimensionality of sparse feature vectors is not just an optimization but a necessity for efficient and effective training, especially with large datasets.
  • CNNs as Feature Extractors: TRAPACCS demonstrates that a CNN can be effectively repurposed as a powerful feature extraction mechanism, with its intermediate outputs serving as rich inputs for a subsequent classifier like an SVM.
  • Robustness to Noisy Data: The authors discovered that their CNN-based approach, while generally strong, can be vulnerable to noisy input data (e.g., incorrect POS tags or lemmas). This led to an interesting observation that simpler linear SVMs might generalize better in such noisy scenarios.
  • Multilingual Performance Variation: The systems performed exceptionally well in some languages but poorly in others, suggesting that language-specific characteristics (like tag availability or lemma consistency) significantly impact performance.

Key Results

  • TRAPACC and TRAPACCS ranked third and second, respectively, overall in the PARSEME shared task based on the MWE-based F1 metric.
  • Their systems achieved the best performance for 8 out of 19 languages, including Romanian, Hungarian, and English.
  • For Turkish (TR), where the data had significant issues with lemmatization and POS tagging, their initial performance was very low. However, after removing features sensitive to this noise, the MWE-based F1 score for TRAPACC increased significantly from near zero to 39.34.
  • The comparison between TRAPACC and TRAPACCS showed a very high rank correlation, indicating that the addition of the kernel SVM did not drastically alter the overall performance, suggesting further tuning might be needed.

Limitations and Future Work

  • Closed Track Limitations: The systems were constrained by the "closed track" rules, limiting access to external resources like word embeddings, which could potentially improve performance.
  • Vulnerability to Noisy Data: The authors acknowledge that their CNN-based approach can be sensitive to noise in the training data, as observed with the Turkish corpus.
  • Parameter Tuning: The performance of the kernel SVM in TRAPACCS could likely be improved with more extensive parameter tuning.
  • Future Work:
    • Explore external resources like word vectors and association measures for open-track participation.
  • Investigate different dimension reduction methods or even removing them.
  • Develop a two-step classification process: first detect VMWEs generally, then categorize them.
  • Implement language-specific parameter tuning for CNNs instead of a one-size-fits-all approach.
  • Further investigate the impact of noisy data and explore strategies to mitigate it.

Summary updated

BibTeX
@inproceedings{stodden2018trapacc,
  author = {Regina Stodden and Behrang QasemiZadeh and Laura Kallmeyer},
  title = {{TRAPACC and TRAPACCS at PARSEME Shared Task 2018: Neural Transition Tagging of Verbal Multiword Expressions}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure