YSP at SemEval-2024 Task 1: Enhancing Sentence Relatedness Assessment using Siamese Networks

Yasamin Aali, Sardar Hamidian, Parsa FarinneyaNAACL 2024

TL;DR

This paper tackles the challenge of accurately measuring semantic relatedness between sentences, especially in under-resourced languages. They propose a Siamese Network architecture, leveraging pre-trained language models like BERT, RoBERTa, and the Universal Sentence Encoder (USE), and find that USE generally outperforms others, particularly in English and Marathi.

What Problem Does This Paper Address?

Measuring how semantically related two pieces of text are is crucial for many Natural Language Processing (NLP) tasks, from information retrieval to question answering and summarization. While this has been extensively studied for English, applying these methods to other languages is significantly harder due to a lack of comprehensive linguistic resources and annotated datasets. This disparity creates a challenge in accurately capturing subtle meanings and establishing cross-language associations, making it difficult to develop language-independent models for semantic relatedness.

Methods

The core of the proposed system is a Siamese Network architecture. This architecture uses two identical subnetworks (networks) that process two input sentences separately. The outputs of these subnetworks (sentence embeddings) are then compared to determine their similarity. The key innovation here is the use of pre-trained language models as the backbone for these subnetworks. Specifically, the authors experimented with:

  • BERT (Bidirectional Encoder Representations from Transformers): A powerful model known for its deep understanding of language context.
  • RoBERTa (A Robustly Optimized BERT Pretraining Approach): An optimized version of BERT.
  • Universal Sentence Encoder (USE): A model specifically designed to generate high-quality sentence embeddings that capture semantic meaning.

These embeddings are then fed into a Euclidean distance calculation, followed by a Dense layer with sigmoid activation to predict a similarity score between 0 and 1. The model is trained using Mean Squared Error (MSE) loss and the Adam optimizer.

Datasets

The experiments were conducted on data from the SemEval-2024 Task 1, Subtask 1 (supervised). This dataset consists of 5,500 samples with 8 language pairs. The official training set was used, and an 80/20 train/dev split was employed for experiments, with the official development set serving as the test set. The task involves predicting a semantic textual relatedness score between sentence pairs, ranging from 0 (least related) to 1 (most related), based on human annotations.

Key Insights and Ideas

  • USE shines in semantic capture: The Universal Sentence Encoder consistently demonstrated superior performance in capturing semantic similarities compared to BERT and RoBERTa across most languages. This suggests that models specifically trained for sentence embeddings can be more effective for this task than general-purpose language models.
  • Language-specific performance variations: While USE was strong overall, its exceptional performance in English and Marathi highlights the importance of considering linguistic characteristics and task requirements when selecting pre-trained models.
  • Siamese Networks are efficient: The paper emphasizes that Siamese architectures are effective because they use the same model for similar inputs, reducing training parameters and making them less prone to overfitting, especially with limited data.

Key Results

The paper presents a table summarizing the performance across different languages.

  • English: The USE model achieved the highest scores, with 0.75 on the training set and 0.82 on the test set. The top overall score reached 0.86.
  • Marathi and Telugu: These languages also showed strong performance with the USE model, with Marathi achieving a score of 0.69.
  • Underperforming Languages: The model struggled with Kinyarwanda, Hausa, and Algerian Arabic, indicating areas where further research and resource development are needed.
  • Comparison: The USE model generally outperformed BERT and RoBERTa across most languages in the training set. For instance, in English, USE achieved 0.75 compared to BERT's 0.42 and RoBERTa's 0.38.

Limitations and Future Work

The authors acknowledge that the model struggled with certain languages like Kinyarwanda, Hausa, and Algerian Arabic. They also noted that the model had difficulty calculating the lowest semantic relatedness scores between sentences.

Future work could focus on:

  • Further optimization and refinement: This could enhance performance in under-performing languages.
  • Broadening applicability: Expanding the system's capabilities to more real-world scenarios.
  • Developing language-specific models: Addressing the challenges posed by limited linguistic resources in non-English languages.

Summary updated

BibTeX
@inproceedings{aali2024ysp,
  author = {Yasamin Aali and Sardar Hamidian and Parsa Farinneya},
  title = {{YSP at SemEval-2024 Task 1: Enhancing Sentence Relatedness Assessment using Siamese Networks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure