NRK at SemEval-2024 Task 1: Semantic Textual Relatedness through Domain Adaptation and Ensemble Learning on BERT-based models

Nguyen Tuan Kiet, Dang Van ThinNAACL 2024

TL;DR

This paper presents an ensemble approach for Semantic Textual Relatedness (STR) across 14 languages, leveraging domain-adapted BERT-based models. Their system achieved competitive results, ranking in the Top 3 and Top 4 for Algerian Arabic and Amharic, respectively, demonstrating the effectiveness of combining multiple pre-trained models with a weighted voting strategy.

What Problem Does This Paper Address?

Semantic Textual Relatedness (STR) is a crucial task in Natural Language Processing (NLP) that aims to quantify how semantically similar two sentences are. This capability is vital for applications like chatbots, question answering, and text summarization. However, accurately measuring STR is challenging due to the nuances of language, such as synonyms and ambiguity, and the complexities of cross-lingual and cross-cultural understanding. The SemEval-2024 Task 1 specifically targets this problem by providing a benchmark across 14 diverse languages, pushing the boundaries of existing NLP models.

Methods

The NRK team's approach for SemEval-2024 Task 1 (Track A: Supervised) is built upon two core components: domain adaptation and ensemble learning.

  1. Domain Adaptation: They begin by adapting pre-trained transformer models (like BERT, DeBERTa-V3, XLM-ROBERTa, and E5) to the specific domain of sentence-pair relatedness. This is achieved through Masked Language Modeling (MLM) on an "in-domain corpus" derived from the training dataset. This process helps the models better understand the semantic relationships within sentence pairs.
  2. Cross-Encoder Architecture: Following domain adaptation, they employ a Sentence-BERT (SBERT) based cross-encoder. This architecture processes sentence pairs jointly, encoding them into dense vectors that capture their contextual information and semantic relationships. A sigmoid function is then applied to output a relatedness score between 0 and 1.
  3. Ensemble Learning with Weighted Voting: To further enhance performance and robustness, the predictions from multiple fine-tuned cross-encoder models are combined using a weighted voting technique. The weights for each model are determined based on its performance on a development set, allowing the system to prioritize more accurate models.

Datasets

The primary dataset used in this work is the SemEval-2024 Task 1 dataset. This dataset contains sentence pairs across 14 languages, each annotated with a relatedness score from 0 (completely unrelated) to 1 (maximally related). The paper also mentions using the XNLI dataset and multilingual-NLI-26lang-2mil7 dataset for fine-tuning the mDeBERTa-V3 model.

Key Insights and Ideas

  • Domain Adaptation is Key: Fine-tuning pre-trained language models on domain-specific data (even through MLM) significantly improves their ability to capture nuanced semantic relationships in sentence pairs.
  • Ensembling Boosts Robustness: Combining predictions from multiple diverse models, even if some individual models perform moderately, can lead to a more robust and often superior overall performance. The weighted voting scheme effectively leverages the strengths of different models.
  • Model Diversity Matters: The choice of base models (e.g., multilingual vs. monolingual, different architectures like BERT, DeBERTa, E5) contributes to the ensemble's effectiveness by capturing different aspects of semantic relatedness.
  • Challenges in Ensemble Weighting: The paper notes that in some cases, a single superior base model can degrade the ensemble's performance, highlighting the complexity of optimal ensemble construction and the need for careful weight tuning.

Key Results

The NRK system achieved competitive results on the SemEval-2024 Task 1, particularly in Track A (Supervised). Their submission ranked:

  • Top 3 for Algerian Arabic.
  • Top 4 for Amharic.
  • They also showed strong performance in English and Moroccan Arabic.

Table 1 in the paper details their performance against top-performing teams on 9 languages, and Table 2 shows the performance of individual base models and their ensemble on the development dataset, generally indicating that the ensemble outperforms most individual models.

Limitations and Future Work

Limitations: * The paper acknowledges that in some tracks, the ensemble's performance declined due to the presence of a single, highly dominant base model. This suggests that the ensemble strategy might need further refinement to handle such scenarios. * While the system performed well, it did not consistently outperform the top-ranked teams across all languages.

Future Work: * The authors propose enhancing their system by integrating African transformer-based models, which could improve performance on languages from that region. * Exploring data augmentation techniques is suggested as a way to further boost the overall performance of their STR system.

Summary updated

BibTeX
@inproceedings{kiet2024nrk,
  author = {Nguyen Tuan Kiet and Dang Van Thin},
  title = {{NRK at SemEval-2024 Task 1: Semantic Textual Relatedness through Domain Adaptation and Ensemble Learning on BERT-based models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure