Team dina at SemEval-2022 Task 8: Pre-trained Language Models as Baselines for Semantic Similarity

Dina Pisarevskaya, Arkaitz ZubiagaNAACL 2022

TL;DR

This paper explores the effectiveness of "pure" pre-trained multilingual language models for the SemEval-2022 Task 8: Multilingual News Article Similarity. The team "dina" found that while these models offer a strong baseline, incorporating named entity and date overlap features slightly improved performance, suggesting that more sophisticated fine-tuning and domain adaptation are needed for better results.

What Problem Does This Paper Address?

The core challenge this paper tackles is determining the semantic similarity between news articles, especially when they are written in different languages. The SemEval-2022 Task 8 specifically aims to assess if pairs of news articles, which can be in any combination of languages, describe the same underlying story. This is a crucial problem for tasks like news aggregation, duplicate detection, and information retrieval, where understanding the semantic content across language barriers is essential. Traditional approaches often rely on machine translation or complex feature engineering, but the authors investigate whether off-the-shelf, pre-trained multilingual models can provide a competitive baseline without extensive task-specific fine-tuning.

Methods

The team "dina" primarily experimented with "pure" pre-trained multilingual language models. These models, such as LaBSE, paraphrase-xlm-r-multilingual-v1, and distiluse-base-multilingual-cased-v1, were not fine-tuned on the specific task data. Instead, they were used to generate sentence embeddings, and similarity was calculated using cosine similarity. To enhance these models, they incorporated two additional features:

  • Named Entity Recognition (NER): Using the spacy library, they identified named entities (people, organizations, locations, etc.) in both original and translated texts. The overlap of these entities between article pairs was used as a feature.
  • Date Extraction: A rule-based approach was used to extract months and days from English texts (original or translated), assuming that related articles are likely published around similar dates. The overlap of these dates was also used as a feature.

For cross-lingual comparisons, texts were translated into English using m2m100 models. The "inverted pyramid" principle was also applied, meaning only the first 10 sentences of each article were considered to capture the most important information.

Datasets

  • SemEval-2022 Task 8 Training Data: Consisted of 4,964 monolingual and cross-lingual news article pairs across several languages (English, German, Spanish, Turkish, Polish, Arabic, French).
  • SemEval-2022 Task 8 Test Data: Included the training languages plus Italian, Russian, and Chinese.
  • XLSum Dataset: Used for domain-specific fine-tuning experiments. This dataset contains 1.35 million article-summary pairs across 44 languages. A subset of 98,697 news texts was selected for their experiments.

Key Insights and Ideas

  • "Pure" models as strong baselines: The study highlights that pre-trained multilingual models, without task-specific fine-tuning, can serve as surprisingly strong baselines for multilingual semantic similarity tasks.
  • Feature augmentation offers marginal gains: While adding named entity and date overlap features provided a slight improvement over the pure models, the gains were not substantial, suggesting that these features might not be as discriminative as hoped or that their integration could be more sophisticated.
  • LaBSE's limitations: The findings suggest that LaBSE might not perform as well on sentence pairs that are not direct translations, reinforcing the idea that model suitability depends on the specific nature of the cross-lingual task.
  • Fine-tuning challenges: Initial attempts at unsupervised domain-specific fine-tuning on the XLSum dataset did not yield significant improvements, indicating that more advanced fine-tuning strategies or larger, more relevant datasets might be necessary.

Key Results

The team's best-performing submission achieved a Pearson's correlation score of 0.511 on the test set. This was achieved using the "pure" paraphrase-xlm-r-multilingual-v1 model combined with named entity and date intersection features. This slightly outperformed the pure paraphrase-xlm-r-multilingual-v1 model alone (0.502) and the model with only named entity intersections (0.508). However, this score is noted as moderate, with room for improvement compared to other participants who achieved higher scores.

Limitations and Future Work

The authors acknowledge several limitations:

  • Text Parsing Errors: Some test articles contained non-relevant text (e.g., error messages, metadata) that their parsing rules did not fully exclude, leading to noisy data.
  • Translation Errors: For articles originally in Chinese and Arabic, translation errors were observed, particularly in the generated named entities, impacting the effectiveness of the NER feature.
  • Moderate Performance: The achieved score of 0.511 is considered moderate, falling short of the best system's score of 0.818.

Future work suggested includes:

  • More Sophisticated Fine-tuning: Exploring advanced fine-tuning techniques for domain adaptation.
  • Improved Pre-processing: Developing more robust methods to clean and pre-process text data, especially for handling noisy or non-relevant content.
  • Alternative Translation Approaches: Investigating other translation APIs (e.g., Google Translate API) to potentially improve the quality of translations for cross-lingual tasks.
  • Monolingual Models: Considering the use of separate monolingual models for languages that performed poorly.
  • Further XLSum Experiments: Conducting more experiments with the XLSum dataset, potentially varying sampling techniques and sample sizes.

Summary updated

BibTeX
@inproceedings{pisarevskaya2022team,
  author = {Dina Pisarevskaya and Arkaitz Zubiaga},
  title = {{Team dina at SemEval-2022 Task 8: Pre-trained Language Models as Baselines for Semantic Similarity}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure