SATLab at SemEval-2024 Task 1: A Fully Instance-Specific Approach for Semantic Textual Relatedness Prediction

Yves BestgenNAACL 2024

TL;DR

This paper presents SATLab's approach to SemEval-2024 Task 1, focusing on predicting semantic textual relatedness (STR) using a novel, fully instance-specific method based on character n-gram frequencies. While achieving top performance in several languages, the study highlights the limitations of purely instance-specific methods and the significant impact of incorporating broader contextual features like TF-IDF.

What Problem Does This Paper Address?

The paper tackles the challenge of Semantic Textual Relatedness (STR), a broader concept than semantic similarity, where sentences can be related through shared themes or viewpoints even if they don't strictly paraphrase or entail each other. Existing research in STR is less advanced due to a scarcity of annotated datasets. SemEval-2024 Task 1 aimed to address this by providing a multilingual dataset and focusing on relatedness, with subtasks for supervised, unsupervised, and cross-lingual settings. The authors specifically focused on the unsupervised subtask, seeking a language-independent approach that doesn't rely on external resources or information from other instances in the dataset.

Methods

SATLab's proposed system is a fully instance-specific approach, meaning each sentence pair is processed independently without any knowledge of other pairs in the dataset. The core of the method involves calculating the Euclidean distance between character n-grams (1 to 5 characters long) of the two sentences. All characters, including punctuation and symbols, are considered. The frequency of each n-gram is weighted using a logarithmic function, and the resulting feature vectors are L2 normalized. This approach is designed to be language-agnostic and requires no external linguistic resources or tokenization, making it particularly suitable for low-resource languages.

Datasets

The paper primarily discusses the SemEval-2024 Task 1 dataset. This dataset is multilingual, covering twelve languages: Algerian Arabic, Amharic, Hausa, Modern Standard Arabic, Moroccan Arabic, Afrikaans, English, Hindi, Punjabi, Spanish, Indonesian, and Kinyarwanda. The data was collected from various sources like semantic similarity datasets, news articles, and Wikipedia, and then annotated by native speakers using a Best-Worst Scaling procedure to assess semantic relatedness.

Key Insights and Ideas

  • Instance-specific vs. Contextual Information: The study strongly suggests that a purely instance-specific approach, while appealing for its simplicity and language independence, is significantly less effective than methods that incorporate broader contextual information.
  • The Power of TF-IDF: Incorporating TF-IDF weighting for n-gram features provided a substantial benefit across most languages, underscoring the importance of considering feature importance within the entire dataset.
  • Language-Specific Challenges: Performance varied significantly across languages, indicating that the inherent complexity or nature of the material for certain languages poses a greater challenge for STR prediction.
  • Character N-grams' Robustness: Character n-grams, despite their simplicity, proved to be a robust feature representation, especially for tasks like language identification and, in this case, STR, as they avoid tokenization issues and can capture morphological information.

Key Results

The SATLab system achieved strong results, coming in first place in five out of the twelve languages. However, the paper notes that the difference between the top-performing systems was often minimal. Notably, the organizers' baseline system outperformed all participant systems in Hindi and Moroccan Arabic, suggesting a general underperformance by participants on these specific languages.

In the system component analysis, TF-IDF weighting was found to be beneficial in ten out of eleven languages, significantly improving performance. Conversely, not using L2 normalization negatively impacted performance in most languages. The study also found that Dice distance was more efficient than Euclidean distance, but the gain was less significant than that from TF-IDF weighting.

Limitations and Future Work

The authors acknowledge that their fully instance-specific approach, while achieving good results, is more akin to a baseline than a state-of-the-art system. The significant gains observed when incorporating TF-IDF suggest that this unsupervised approach, without considering other instances, is not optimal. The paper also points out the considerable performance variations across languages, suggesting that further research is needed to understand the origins of these fluctuations. This type of unsupervised approach is not recommended due to observed negative correlations in one language. Future work could explore more sophisticated methods for incorporating contextual information while maintaining language independence.

Summary updated

BibTeX
@inproceedings{bestgen2024satlab,
  author = {Yves Bestgen},
  title = {{SATLab at SemEval-2024 Task 1: A Fully Instance-Specific Approach for Semantic Textual Relatedness Prediction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure