ZhichunRoad at SemEval-2022 Task 2: Adversarial Training and Contrastive Learning for Multiword Representations

Xuange Cui, Wei Xiong, Songlin WangNAACL 2022

TL;DR

This paper tackles the challenge of accurately representing multiword expressions (MWEs) in multiple languages, particularly distinguishing between literal and idiomatic meanings. They achieve this by enhancing pre-trained multilingual language models with adversarial training for robustness and contrastive learning for better sentence embeddings, leading to competitive results on the SemEval-2022 Task 2.

What Problem Does This Paper Address?

Natural language processing models, even powerful pre-trained ones like BERT, struggle to accurately capture the nuances of idiomatic expressions. This is because many phrases can be used both literally and figuratively, making it difficult for models to distinguish between these meanings. Furthermore, representing these multiword expressions (MWEs) effectively, especially across different languages, is a significant challenge. The SemEval-2022 Task 2 specifically aims to address this by providing a multilingual dataset for detecting and representing MWEs, focusing on idiomaticity and semantic similarity.

Methods

The authors employ a multi-pronged approach to improve MWE representation:

  • Pre-trained Multilingual Models: They experiment with several state-of-the-art multilingual language models from the XTREME Leaderboard, including mBERTbase, XLM-Rbase, and InfoXLMbase, to serve as their base text encoders.
  • Adversarial Training: To enhance model generalization and robustness, they incorporate adversarial training. This involves adding small perturbations to the input data to make the model more resilient to variations and potential attacks. They explore methods like SMART and FREELB for this purpose.
  • Exponential Moving Average (EMA): EMA is used to smooth the model's trained parameters, which can lead to improved performance and more stable evaluations.
  • Cross-Attention Module: For SubTaskB, which focuses on semantic similarity, they introduce a cross-attention module. This module helps in modeling the relationships between two sentences more effectively.
  • Contrastive Learning: To improve the quality of sentence embeddings, they jointly train the model with a contrastive learning objective. This method encourages similar sentences to have close embeddings while pushing dissimilar ones further apart. They also leverage a momentum contrast mechanism to increase the number of negative pairs, further refining the embeddings.
  • Alignment and Uniformity Metrics: The quality of sentence embeddings is evaluated using alignment (how well positive pairs are represented) and uniformity (how uniformly embeddings are distributed in space).

Datasets

The paper utilizes datasets from the SemEval-2022 Task 2, which comprises two subtasks:

  • SubTaskA (Idiomaticity Detection): This subtask involves binary classification to determine if a sentence contains an idiomatic expression. The dataset includes context sentences, the target sentence with the MWE, and a label indicating "Idiomatic" or "non-idiomatic."
  • SubTaskB (Sentence Embedding for Semantic Similarity): This subtask requires models to generate sentence embeddings that accurately reflect semantic similarity. The evaluation is based on the Spearman correlation between the cosine similarity of embeddings and gold labels. The paper mentions using the STS-Benchmark and ASSIN2 datasets for tuning and evaluation in this subtask.

Key Insights and Ideas

  • Adversarial training significantly boosts robustness: The authors found that adversarial training strategies like SMART and FREELB provided better performance than simpler methods like FGM and PGD, highlighting the importance of robust training for MWE tasks.
  • InfoXLMbase excels as a multilingual encoder: Among the tested pre-trained models, InfoXLMbase consistently provided the best results, suggesting its suitability for multilingual MWE tasks.
  • Contrastive learning is crucial for sentence embeddings: Jointly training with contrastive learning yielded substantial improvements in sentence representation quality, as evidenced by better performance on the STS-Test dataset.
  • Alignment and uniformity are key to good embeddings: The study reinforces the idea that a good balance between embedding alignment and uniformity is essential for high-quality sentence representations. However, achieving perfect alignment and uniformity simultaneously remains a challenge.
  • Cross-attention enhances sentence relationship modeling: The addition of a cross-attention module proved beneficial for modeling sentence relationships in the semantic similarity task.

Key Results

  • SubTaskA (Idiomaticity Detection): The proposed approach achieved competitive results, ranking 12th in zero-shot settings and 4th in one-shot settings. Specifically, they reported performance gains of 2.91% and 3.87% over baseline solutions in zero-shot and one-shot settings, respectively.
  • SubTaskB (Sentence Embedding): With fine-tuning on supervised target datasets, their approach achieved an 8.22% and 4.5% improvement compared to the baseline, ranking in the top-4 for pre-train and fine-tune settings.
  • Ablation Studies: Table 4 shows that combining mBERTbase with EMA and SMART led to a dev-F1 score of 76.21, a significant improvement. Table 5 indicates that InfoXLMbase with EMA and contrastive learning (MoCo) achieved a Spearman correlation of 56.32 on the STS-Test dataset.

Limitations and Future Work

  • Limitations: The paper notes that achieving perfect alignment and uniformity in sentence embeddings simultaneously is difficult in practice. The performance was also found to be sensitive to batch size.
  • Future Work: The authors suggest exploring larger pre-trained language models (e.g., mBertlarge, InfoXLMlarge) and investigating other data augmentation techniques like Token-Shuffle, Token-Cutoff, and Mix-Up to further enhance model performance.

Summary updated

BibTeX
@inproceedings{cui2022zhichunroad,
  author = {Xuange Cui and Wei Xiong and Songlin Wang},
  title = {{ZhichunRoad at SemEval-2022 Task 2: Adversarial Training and Contrastive Learning for Multiword Representations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure