kpfriends at SemEval-2022 Task 2: NEAMER - Named Entity Augmented Multi-word Expression Recognizer

Minsik OhNAACL 2022

TL;DR

This paper introduces NEAMER, a novel system for recognizing Multi-word Expressions (MWEs) that leverages knowledge transfer from Named Entity Recognition (NER). By exploiting the shared characteristic of non-compositionality between MWEs and Named Entities, NEAMER achieves state-of-the-art results on the SemEval-2022 Task 2, demonstrating improved performance and training stability.

What Problem Does This Paper Address?

Multi-word Expressions (MWEs) are phrases whose meaning isn't simply the sum of their individual words (e.g., "kick the bucket"). While advances in pre-trained language models have improved many NLP tasks, accurately identifying MWEs, especially idiomatic ones, remains a challenge. This is partly because MWEs often exhibit "non-compositionality," meaning their meaning is not compositional. Named Entities (like "Google" or "New York City") also share this non-compositional characteristic, as their meaning as a single unit is not derived from the individual words. This paper tackles the problem of improving MWE classification by exploring how knowledge learned from NER, which also deals with non-compositional units, can be transferred to the MWE task.

Methods

The core of NEAMER lies in transfer learning between NER and MWE classification. The authors hypothesize that the non-compositionality learned by models during NER training can be beneficial for MWE identification.

Here's a breakdown of their approach:

  • Leveraging Pre-trained Models: They start with large pre-trained language models like XLM-RoBERTa.
  • NER Fine-tuning: These models are first fine-tuned on NER tasks. The rationale is that NER requires understanding non-compositional word combinations to correctly identify entities (e.g., recognizing "New York City" as a single location, not three separate words).
  • MWE Classification: The NER-fine-tuned models are then further trained on the MWE classification task. This allows the model to apply the non-compositional understanding gained from NER to the problem of identifying idiomatic MWEs.
  • Locality Features: In addition to transfer learning, they introduce "locality features." These are handcrafted features that capture information about the immediate context of an MWE, such as capitalization, surrounding punctuation, or specific word patterns (e.g., "The *"). These features are fed into a small neural network to augment the text representation.
  • Stability Investigation: The paper also investigates the stability of training, noting that the shift from Masked Language Modeling (MLM) pre-training to specific downstream tasks like MWE classification can sometimes lead to instability. They explore how NER fine-tuning might mitigate this.

Datasets

The primary dataset used for evaluation is from SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding. This task involves classifying sentences based on whether a specific MWE is used idiomatically or not. The authors focus on the OneShot setting, which requires classifying target sentences using idioms not seen during training, leveraging a ZeroShot dataset for this purpose.

Key Insights and Ideas

  • NER as a Proxy for Non-compositionality: The most significant insight is that NER, despite its different objective, effectively teaches models about non-compositionality. This knowledge is transferable and beneficial for MWE classification.
  • Transfer Learning Improves Stability: The paper suggests that fine-tuning on NER can lead to more stable training for MWE classification compared to directly fine-tuning from a general MLM pre-trained model. This is attributed to NER's task complexity being closer to MWE classification than MLM.
  • Locality Features Offer Marginal Gains: While locality features are explored, their impact on performance is found to be relatively small, especially when compared to the gains from NER transfer learning. The authors speculate this might be due to the features being too simple or already implicitly captured by NER fine-tuning.
  • Language-Specific Fine-tuning Matters: The results suggest that fine-tuning on NER in the same language as the MWE task (e.g., English NER for English MWEs) yields better results than cross-lingual NER fine-tuning.

Key Results

  • State-of-the-Art Performance: The NEAMER system achieved a State-of-the-Art (SOTA) F1 score of 0.9395 on the SemEval-2022 Task 2 (Subtask A, OneShot setting) during the post-evaluation phase.
  • Top Ranking: The system secured a top 2 rank during the official evaluation phase with an F1 score of 0.9346.
  • Improved Performance with NER: Models fine-tuned on NER consistently outperformed the baseline XLM-RoBERTa model, showing significant improvements in F1 scores across various languages. For instance, an XLM-Rlarge model augmented with English NER achieved an F1 score of 96.1 on English data, compared to 88.1 for the baseline.
  • Locality Feature Impact: The impact of locality features was marginal, with only slight improvements observed in some cases. For the "The *" feature, locality features did show a notable increase in F1 score (5.8 points) after NER transfer learning.

Limitations and Future Work

  • Locality Feature Effectiveness: The authors acknowledge that the locality features, as implemented, provided only marginal improvements. They suggest that more sophisticated architectures or layer-wise learning rate tuning might be needed to better leverage these features.
  • Simplicity of Locality Features: They hypothesize that locality features might be too simplistic to capture the full complexity of non-compositionality in MWEs, or that NER fine-tuning already implicitly encodes this information.
  • Future Work: The paper suggests future work could involve exploring more advanced architectures for locality features, investigating longer training epochs for cross-lingual transfer, and experimenting with more diverse MWE classification datasets. They also note the potential for further research into the "unlearning" process required when transferring knowledge between tasks with different objectives.

Summary updated

BibTeX
@inproceedings{oh2022kpfriends,
  author = {Minsik Oh},
  title = {{kpfriends at SemEval-2022 Task 2: NEAMER - Named Entity Augmented Multi-word Expression Recognizer}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure