Cambridge at SemEval-2021 Task 2: Neural WiC-Model with Data Augmentation and Exploration of Representation

Zheng Yuan, David StrohmaierACL 2021

TL;DR

This paper presents a novel neural network approach for the challenging task of Word-in-Context (WiC) disambiguation, particularly in multilingual and cross-lingual settings. The system leverages a pre-trained Transformer model, augmented with novel distance-based features and a two-step training strategy involving out-of-domain pre-training and in-domain fine-tuning, achieving state-of-the-art results.

What Problem Does This Paper Address?

Polysemy, the phenomenon where a word can have multiple meanings, remains a significant hurdle for natural language processing (NLP) applications. Word-in-Context (WiC) disambiguation specifically tackles this by determining if a target word in two different sentences shares the same meaning. The SemEval-2021 Task 2 (MCL-WiC) extended this challenge by introducing multilingual (same language) and cross-lingual (different languages) tracks, including a "zero-shot" setting for cross-lingual evaluation, meaning no direct cross-lingual training data is provided. This task requires models to accurately distinguish word senses across diverse linguistic contexts, a problem that existing methods struggle to solve effectively, especially when dealing with multiple languages and limited training data.

Methods

The Cambridge team's system builds upon the pre-trained multilingual masked language model XLM-R. Their approach introduces several key innovations:

  • Two-Step Training Strategy: The model is first pre-trained on a large corpus of "out-of-domain" data (data not from the specific MCL-WiC task) and then fine-tuned on the in-domain MCL-WiC training data. This strategy aims to leverage broader linguistic knowledge before specializing in the target task.
  • Data Augmentation: To enrich the training data, the authors employ automatic data augmentation techniques. They cleverly create new training instances by pairing existing sentences that share a target word but have different contexts, inferring the correct label based on the original instances. They also incorporate data from external resources like WiC, XL-WiC, CALD, and SeCoDa.
  • Novel Representations: Beyond standard token and sequence representations from XLM-R, the system incorporates three distance-based features: element-wise absolute difference, cosine similarity, and Euclidean distance between the vector representations of the target word in the two sentences. This aims to capture nuanced semantic relationships.
  • Model Architecture: A Transformer-based model (XLM-R) is used, with two additional linear layers on top for classification. For words split into sub-tokens, averaged representations are used instead of just the first sub-token's representation.

Datasets

The experiments utilize several datasets:

  • MCL-WiC: The primary dataset for the shared task, containing multilingual and cross-lingual sentence pairs for disambiguation.
  • (*)WiC Datasets: This includes the original WiC dataset and the XL-WiC dataset, which covers 12 languages. These are used for out-of-domain pre-training.
  • CALD (Cambridge Advanced Learner's Dictionary): Used to generate out-of-domain data by pairing sentences with different complexity and sense information.
  • SeCoDa (Sense Complexity Dataset): Another source for out-of-domain data, providing annotated English sentences with complexity and word senses.

Key Insights and Ideas

  • The Power of Two-Step Training: The authors demonstrate that a two-step training approach (pre-training on diverse data followed by fine-tuning on task-specific data) is crucial for achieving high performance. Removing either step significantly degrades accuracy.
  • In-Domain Data is King (Even with Less of It): Despite using a much larger volume of out-of-domain data for pre-training, the fine-tuning stage on the in-domain MCL-WiC data proved more impactful for performance gains, highlighting the importance of high-quality, task-relevant data.
  • Distance Features Matter: The inclusion of distance-based features (absolute difference, cosine similarity, Euclidean distance) between word representations significantly improves performance, suggesting that capturing the geometric relationships between word embeddings in context is beneficial for disambiguation.
  • Cross-Lingual Generalization is Possible: The system achieves strong results in the cross-lingual track even without direct cross-lingual training, indicating that the multilingual pre-training of XLM-R and the general WiC disambiguation strategy can generalize well across languages.

Key Results

  • State-of-the-Art Multilingual Performance: The ensemble system achieved state-of-the-art results in the multilingual track, ranking first in Arabic (AR-AR) and Russian (RU-RU), and second in French (FR-FR) out of 171 submitted systems.
  • Competitive Cross-Lingual Performance: The "zero-shot" cross-lingual system consistently ranked within the top ten across all cross-lingual sub-tracks.
  • Data Augmentation Benefits: Augmenting the WiC and XL-WiC datasets improved performance across most languages in the multilingual track.
  • Representation Effectiveness: The proposed representation combining sequence, token, and distance-based features yielded the best overall performance in the multilingual track.

Limitations and Future Work

  • CALD Positive Instance Limitation: Due to time constraints, the authors did not utilize positive instances from the CALD dataset for training, leaving this as a potential area for future work.
  • Zero-Shot Cross-Lingual Potential: While the cross-lingual results are competitive, further exploration into explicit cross-lingual pre-training or fine-tuning strategies could potentially yield even better performance.
  • Ensemble Complementarity: The success of the ensemble model suggests that combining diverse systems could further enhance performance, a common direction in NLP research.

Summary updated

BibTeX
@inproceedings{yuan2021cambridge,
  author = {Zheng Yuan and David Strohmaier},
  title = {{Cambridge at SemEval-2021 Task 2: Neural WiC-Model with Data Augmentation and Exploration of Representation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure