MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness

Dhiman Goswami, Sadiya Sayara Chowdhury Puspo, Nishat Raihan, Al Nahian Bin Emran, Amrita Ganguly, Marcos ZampieriNAACL 2024

TL;DR

This paper presents MasonTigers' approach to SemEval-2024 Task 1, focusing on Semantic Textual Relatedness (STR) across 14 languages and three tracks (supervised, unsupervised, and cross-lingual). Their ensemble method, combining statistical machine learning with BERT-based models and sentence transformers, achieved competitive results, highlighting the effectiveness of ensemble strategies in diverse linguistic settings.

What Problem Does This Paper Address?

Understanding how words, phrases, and sentences relate to each other in meaning (semantic textual relatedness) is crucial for many Natural Language Processing (NLP) tasks, from question answering to machine translation. While significant research has focused on English, there's a growing need to develop and evaluate models for a wider range of languages. SemEval-2024 Task 1 specifically aimed to push the boundaries of STR modeling across 14 diverse languages, presenting challenges related to data availability, linguistic variation, and the need for robust, generalizable approaches. The task was divided into three tracks: supervised (using labeled data), unsupervised (without labeled data), and cross-lingual (transferring knowledge between languages).

Methods

The MasonTigers team employed an ensemble approach across all three tracks. For each track, they generated sentence embeddings using various methods:

  • Statistical Embeddings: TF-IDF (Term Frequency-Inverse Document Frequency) and PPMI (Positive Pointwise Mutual Information) were used as baseline statistical methods.
  • Transformer-Based Embeddings:
    • Language-Specific BERTs: For supervised and unsupervised tracks, they leveraged pre-trained BERT models fine-tuned for specific languages (e.g., DziriBERT for Algerian Arabic, AmROBERTa for Amharic, etc.).
  • Sentence Transformers: LaBSE (Language-Agnostic BERT Sentence Embedding) was used as a general-purpose sentence embedding model.
  • Prediction Models: ElasticNet and Linear Regression were applied to the generated embeddings to predict the semantic relatedness scores.
  • Ensembling: The core of their strategy was ensembling the predictions from these different models. For Track A, they used a weighted ensemble based on Spearman correlation coefficients from the development phase. For Tracks B and C, they used an average ensemble.

Datasets

The experiments were conducted using the datasets provided for SemEval-2024 Task 1. These datasets are structured with pairs of sentences and a corresponding human-annotated "score" indicating their semantic textual relatedness.

  • Track A (Supervised): Datasets for 9 languages (Algerian Arabic, Amharic, English, Hausa, Kinyarwanda, Marathi, Moroccan Arabic, Spanish, Telugu) were used for training, development, and testing.
  • Track B (Unsupervised): Datasets for 12 languages (Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Modern Standard Arabic, Moroccan Arabic, Punjabi, Spanish) were used for development and testing. No training data was provided for this track.
  • Track C (Cross-Lingual): Datasets for the same 12 languages as Track B were used for development and testing. For training, the team selected 5 languages from the supervised Track A dataset (excluding the target language) and merged their training data.

Key Insights and Ideas

  • Ensembling is Key: The paper consistently demonstrates that combining predictions from multiple models (statistical, language-specific BERTs, sentence transformers) significantly improves performance across all tracks and languages. This highlights the robustness gained by leveraging diverse modeling strategies.
  • Language-Specific Models Shine (When Available): For languages with dedicated BERT models (e.g., Amharic, Hausa, Kinyarwanda), these often provided the best individual results, underscoring the importance of linguistic specificity.
  • Challenges in Low-Resource/Unsupervised Settings: Tracks B and C, particularly for languages with limited data or where cross-lingual transfer is difficult (like Punjabi), presented significant challenges, leading to lower scores and larger performance drops between development and evaluation phases.
  • Dev vs. Test Set Size Discrepancies: The error analysis points to issues where smaller test sets relative to training data (Track A) or very small development sets (Track B for Afrikaans) could lead to misleading performance metrics and drops in evaluation.

Key Results

The MasonTigers team achieved competitive rankings across the board:

  • Track A (Supervised): Ranks ranged from 11th (English) to 21st (Moroccan Arabic). Spearman correlation on test sets reached 0.84 for English.
  • Track B (Unsupervised): Ranks ranged from 1st (Kinyarwanda) to 8th (English). Test Spearman correlation reached 0.77 for English.
  • Track C (Cross-Lingual): Ranks ranged from 5th (Punjabi) to 12th (Hausa, Kinyarwanda). Test Spearman correlation reached 0.56 for Spanish.

The ensemble approach consistently improved results over individual models, typically by 0-3% in development and evaluation phases.

Limitations and Future Work

  • Data Scarcity: The paper acknowledges that limited dataset sizes, especially for certain languages and in unsupervised/cross-lingual settings, posed a significant challenge and hindered model learning.
  • Model Assumptions: The use of ElasticNet and Linear Regression, while effective, assumes linearity which might not fully capture the complex, non-linear relationships in textual data.
  • Unknown Training Data (Track B/C): The unsupervised and cross-lingual tracks presented difficulties due to the lack of direct training data, forcing reliance on pre-trained models and transfer learning.
  • Linguistic Diversity: The inherent subjectivity, context dependency, and ambiguity in semantic relatedness tasks, coupled with cross-lingual challenges, remain areas for ongoing research.
  • Future Work: The authors implicitly suggest that continued research into more robust cross-lingual transfer methods, better handling of low-resource languages, and potentially exploring more advanced non-linear prediction models would be beneficial. Addressing the impact of data size discrepancies between development and test sets is also crucial.

Summary updated

BibTeX
@inproceedings{goswami2024masontigers,
  author = {Dhiman Goswami and Sadiya Sayara Chowdhury Puspo and Nishat Raihan and Al Nahian Bin Emran and Amrita Ganguly and Marcos Zampieri},
  title = {{MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure