PEAR at SemEval-2024 Task 1: Pair Encoding with Augmented Re-sampling for Semantic Textual Relatedness

Tollef JørgensenNAACL 2024

TL;DR

This paper introduces PEAR, a system for Semantic Textual Relatedness (STR) that leverages smaller pre-trained language models with a pair-encoding approach. By employing augmented re-sampling with a bi-encoder and optimizing hyperparameters, PEAR demonstrates an effective and lightweight solution for evaluating sentence relatedness, particularly for low-resource languages.

What Problem Does This Paper Address?

Semantic Textual Relatedness (STR) is a task that aims to quantify how closely two sentences are related in meaning, going beyond simple similarity to encompass entailment and cause-effect relationships. While Semantic Textual Similarity (STS) has been extensively studied with numerous datasets, STR is a newer field with fewer established benchmarks, especially for low-resource languages. The challenge lies in building effective models for STR that can adapt to datasets of varying sizes, including very small ones (as low as 800 samples), without requiring extensive language-specific engineering. Existing multilingual models are powerful but can be computationally expensive, and adapting them to these smaller, diverse datasets is a key hurdle.

Methods

The PEAR system employs a pair-encoding strategy, where sentences are processed together by a masked language model (MLM) with full cross-attention. This approach is more computationally efficient than separate sentence encoders for this specific task. To address the challenge of limited data, the authors implemented augmented re-sampling. This involves using a pre-trained multilingual bi-encoder to generate sentence embeddings and then finding nearest neighbors to augment the training data with synthetically generated related pairs. This process is iterated for a few epochs before training the main cross-encoder.

The system also focuses on hyperparameter optimization using Optuna to tune learning rates, the number of nearest neighbors (k) for augmentation, and other training parameters. A key design principle was modularity, allowing easy swapping of models and configurations. The authors deliberately avoided language-specific rules, back-translation, or text manipulation to ensure the system's generalizability across languages.

Datasets

The primary dataset used is the SemEval-2024 Task 1 (SemRel) dataset, which comprises 14 languages. The paper specifically focuses on Track A (supervised), utilizing the provided training, development, and test splits for each language. The dataset is noted for its varying sizes, with some languages having very few samples. The authors also experimented with incorporating the STS-Benchmark dataset, but found it did not yield benefits, likely due to differing definitions of relatedness versus similarity.

Key Insights and Ideas

  • Smaller MLMs can be effective for STR: The paper demonstrates that even smaller, pre-trained MLMs, when used in a pair-encoding setup, can achieve competitive results, especially when combined with data augmentation.
  • Augmented re-sampling is crucial for low-resource settings: For datasets with limited samples, generating additional training data via a bi-encoder and k-NN search proved to be a valuable technique for improving model performance.
  • Modularity is key for rapid experimentation: The system's modular design allowed for quick iteration and testing of different models and augmentation strategies, which is essential when working with diverse and low-resource languages.
  • Out-of-distribution data is a challenge: The adversarial validation revealed that while models might perform well on in-domain data, they can struggle when test sets exhibit distributions different from the training data, as seen with the English dataset.

Key Results

The PEAR system achieved competitive results, ranking in the top 2-4 for most languages on the development sets. It outperformed the provided LaBSE baseline for 6 out of 9 languages in the official submission and 8 out of 9 in a rerun without optimization changes. While the augmentation strategy showed marginal improvements on average, it was crucial for some languages. The paper highlights that combining data across languages generally yielded better results than training language-specific models, especially when considering data size.

Limitations and Future Work

The authors acknowledge that only a limited number of models were tested per language, suggesting that exploring alternative multi- and monolingual models could lead to better performance, particularly for languages like Algerian Arabic. Automating relatedness predictions raises ethical considerations regarding potential biases and fairness, especially when applied to real-world applications. Future work could explore more advanced augmentation strategies and investigate how to better handle out-of-distribution data. The authors hope their system will serve as a valuable tool for future STR research.

Summary updated

BibTeX
@inproceedings{jrgensen2024pear,
  author = {Tollef Jørgensen},
  title = {{PEAR at SemEval-2024 Task 1: Pair Encoding with Augmented Re-sampling for Semantic Textual Relatedness}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure