YNU-HPCC at SemEval-2022 Task 2: Representing Multilingual Idiomaticity based on Contrastive Learning

Kuanghong Liu, Jin Wang, Xuejie ZhangNAACL 2022

TL;DR

This paper tackles the challenge of understanding and representing idiomatic expressions across multiple languages. The YNU-HPCC team developed contrastive learning methods for sentence representation and explored transformer-based approaches for idiomaticity detection, achieving competitive results in the SemEval-2022 Task 2.

What Problem Does This Paper Address?

Idiomatic expressions, like "kick the bucket" or "break a leg," pose a significant challenge for natural language processing models because their meaning isn't compositional – you can't simply add up the meanings of individual words. This non-compositionality makes it difficult for models to accurately represent and understand sentences containing idioms, especially across different languages. SemEval-2022 Task 2 specifically aimed to address this by focusing on multilingual idiomaticity detection (identifying if a sentence contains an idiom) and sentence embedding (representing sentences in a way that captures their meaning, even with idioms). The task covered English, Portuguese, and Galician, highlighting the need for cross-lingual understanding of idiomaticity.

Methods

The YNU-HPCC team approached the task with distinct strategies for the two subtasks:

  • Subtask A: Idiomaticity Detection: For this binary classification task (idiomatic vs. non-idiomatic), they experimented with the XLM-RoBERTa model. In a zero-shot setting, they found that concatenating sentences with Multi-Word Expressions (MWEs) as input yielded decent results. However, incorporating context proved less effective in zero-shot scenarios. For the one-shot setting, they achieved better performance.
  • Subtask B: Sentence Representation: This subtask involved predicting Semantic Text Similarity (STS) scores.
  • Pre-train Setting: They utilized a Siamese network structure based on SBERT, optimizing with a contrastive learning approach called CoSENT. This method aims to ensure that embeddings of similar sentences are closer in the vector space, while dissimilar ones are further apart.
  • Fine-tune Setting: Here, they employed contrastive learning with two specific loss functions: triplet loss and multiple negatives ranking loss. These functions are designed to explicitly train the model to distinguish between correct and incorrect representations of idiomatic meaning. They also explored knowledge distillation to create multilingual sentence embeddings.

Datasets

The experiments utilized datasets for English (EN), Portuguese (PT), and Galician (GL). Notably, Galician was primarily present in the test set, serving as a crucial benchmark for evaluating cross-lingual transfer learning capabilities. For Subtask B's pre-train setting, they used the STSBenchmark (English) and ASSIN2 (Portuguese) datasets.

Key Insights and Ideas

  • Contrastive Learning is Key: The paper strongly suggests that contrastive learning is a powerful paradigm for handling idiomaticity. By explicitly training models to differentiate between correct and incorrect semantic representations of idioms, performance significantly improves.
  • Zero-Shot Idiomaticity Detection is Tough: The zero-shot setting for idiomaticity detection proved challenging, especially when relying on contextual information. This indicates that models struggle to generalize idiomaticity detection to unseen idioms without explicit examples.
  • MWEs are Crucial: Incorporating Multi-Word Expressions (MWEs) directly into the input or training data was shown to be beneficial for both subtasks, helping models better recognize and represent idiomatic phrases.
  • Cross-Lingual Transfer is a Hurdle: While models like XLM-RoBERTa show promise, achieving robust cross-lingual understanding of idioms, particularly in zero-shot scenarios, remains an open challenge. The performance drop on Galician in the test set highlights this.

Key Results

  • Subtask A: The YNU-HPCC team achieved competitive results, securing 19th place in the zero-shot setting and 6th place in the one-shot setting for idiomaticity detection. XLM-RoBERTa with CLS tokens from the last layer showed strong performance in the one-shot setting.
  • Subtask B: They achieved 5th place in the pre-train setting and 1st place in the fine-tune setting for sentence representation. The fine-tune setting, leveraging contrastive learning with triplet and multiple negatives ranking loss, demonstrated significant improvements.

Limitations and Future Work

  • Zero-Shot Generalization: The authors acknowledge the limitations in zero-shot idiomaticity detection, noting a lack of generalization ability. They express an intention to explore and improve zero-shot and few-shot learning for this task in future work.
  • Model Training on More Languages: While distiluse-base-multilingual-cased v2 supports more languages, it performed slightly worse than v1, suggesting that training on a larger number of languages might dilute the performance on specific tasks like idiomaticity. This points to a potential trade-off between breadth and depth in multilingual model training.
  • Data Distribution Differences: The difference in results between development and test sets was attributed to the inclusion of Galician data in the test set, highlighting the challenges of evaluating cross-lingual performance.

Summary updated

BibTeX
@inproceedings{liu2022ynu,
  author = {Kuanghong Liu and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2022 Task 2: Representing Multilingual Idiomaticity based on Contrastive Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure