VarIDE at PARSEME Shared Task 2018: Are Variants Really as Alike as Two Peas in a Pod?

Caroline Pasquer, Carlos Ramisch, Agata Savary, Jean-Yves AntoineCOLING 2018

TL;DR

This paper introduces VarIDE, a system for identifying variants of verbal multiword expressions (VMWEs). VarIDE tackles the challenge of VMWE variability by using morphosyntactic features and a naive Bayes classifier to distinguish true VMWE variants from literal word combinations, achieving promising results across 19 languages.

What Problem Does This Paper Address?

Multiword expressions (MWEs), particularly verbal MWEs (VMWEs), are notoriously difficult to identify in text. This is because they can appear in various forms due to complex morphological and syntactic transformations. For example, "to make ends meet" can be expressed in slightly different ways. The PARSEME shared task focuses on identifying these variants, which means recognizing that different surface forms can refer to the same underlying idiomatic expression. The challenge lies in distinguishing these true variants from literal, non-idiomatic uses of the same words. VarIDE aims to address this by developing a system that can accurately identify these VMWE variants.

Methods

VarIDE employs a two-stage approach: candidate extraction and classification.

  1. Candidate Extraction: The system first identifies potential VMWE candidates by leveraging patterns observed in the training data. It normalizes POS tags and lemmas to create consistent representations (POSnorm and LemmNorm) and then generates all possible lemma permutations for frequent VMWEs. This process is designed to capture a wide range of potential variants, even those not explicitly seen in the training data. A "Filter20" is applied to limit the number of words between the first and last components of a candidate to 20, helping to reduce noise.
  2. Classification: A naive Bayes classifier is trained to distinguish between true VMWE variants (idiomatic) and literal word combinations. The classifier uses a rich set of features, categorized as:
  3. Absolute (ABS) features: These are based on the local properties of the candidate VMWE itself and its component words (e.g., noun number, verb form).
  4. Relative (REL) features: These are derived by comparing the candidate VMWE to all annotated VMWEs in the training data that share the same LemmNorm. This helps capture the similarity of a candidate to known variants.

The system was evaluated on 19 of the 20 shared task languages.

Datasets

The VarIDE system was trained and evaluated using the corpora provided for the PARSEME shared task 2018. These corpora include:

  • TRAIN: Manually annotated data for VMWEs used for training the classifier.
  • DEV: Development set for tuning the system.
  • TEST: Test set for final evaluation.

The paper notes that the shared task organizers provided these manually annotated corpora, which are crucial for supervised learning approaches like VarIDE.

Key Insights and Ideas

  • The Importance of Morphosyntactic Features: The success of VarIDE hinges on its ability to capture subtle morphosyntactic variations. Features like verb form, noun number, and the presence of inserted words (determiners, adverbs) are crucial for distinguishing variants.
  • Relative Features for Similarity: The use of relative features, which compare a candidate to existing annotated variants with the same LemmNorm, is a key innovation. This allows the system to learn what constitutes a "similar" variant beyond exact string matching.
  • Extrapolation for Variant Discovery: The method of generating all allowed lemma permutations based on observed POS sequences and LemmNorms is a smart way to broaden the search for variants, potentially uncovering less frequent or unseen forms.
  • Language Adaptability is Crucial: The authors highlight that a fixed set of features is not feasible across languages due to linguistic specificities. Their approach of scanning corpora to identify language-specific features is a practical necessity for multilingual MWE identification.

Key Results

VarIDE achieved satisfactory coverage of the top-10 POSnorms across most languages, with recall greater than 0.8 for 17 languages. The F1 score for Variant-of-train identification varied significantly by language, with some languages (FR, PT, EN, BG, FA, HI) showing F1 scores above 0.5, while others (ES, LT, HU) performed poorly. The paper attributes some of the low performance in certain languages to imbalances in the training data (e.g., too few literal examples) or issues with non-UD tagsets and sentence segmentation.

Limitations and Future Work

The authors acknowledge several limitations:

  • Imbalance in Training Data: The performance of the classifier is sensitive to the ratio of idiomatic (ID) to literal (LIT) examples in the training data. Imbalances can negatively impact recall and precision.
  • Language-Specific Issues: Non-UD tagsets, sentence segmentation errors, and missing lemmatization in some languages posed challenges.
  • Feature Set Optimization: The authors suggest that improving the feature set could optimize classification recall and precision.

Future work includes:

  • More Detailed Analysis: Conducting a deeper analysis to explain performance discrepancies across languages, focusing on the most informative features.
  • Exploring Other Classifiers: Evaluating alternative classifiers like linear SVMs or multilayer perceptrons.
  • Developing a Task-Independent Variability Profile: Correlating features with linguistic justifications to create a more generalized understanding of VMWE variability.

Summary updated

BibTeX
@inproceedings{pasquer2018varide,
  author = {Caroline Pasquer and Carlos Ramisch and Agata Savary and Jean-Yves Antoine},
  title = {{VarIDE at PARSEME Shared Task 2018: Are Variants Really as Alike as Two Peas in a Pod?}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure