LSX_team5 at SemEval-2022 Task 8: Multilingual News Article Similarity Assessment based on Word- and Sentence Mover’s Distance

Stefan Heil, Karina Kopp, Albin Zehe, Konstantin Kobs, Andreas HothoNAACL 2022

TL;DR

This paper tackles the challenge of assessing similarity between news articles across different languages. The authors explored two parameter-free methods, Word Mover's Distance (WMD) and Sentence Mover's Distance (SMD), leveraging pre-trained multilingual embeddings. Their findings suggest that translating articles to a common language (English) before embedding significantly improves similarity assessment, with SMD ultimately performing best.

What Problem Does This Paper Address?

Accurately measuring the similarity between documents is a fundamental problem in information retrieval with broad applications, such as tracking regional news coverage or sentiment analysis. However, extending this to multilingual contexts presents a significant hurdle: how do you compare articles written in different languages? SemEval-2022 Task 8 specifically addressed this by providing a dataset of news article pairs in seven languages, along with bilingual pairs, and asking participants to predict similarity scores. The challenge lies in bridging the linguistic divide without relying on language-specific models or extensive labeled training data.

Methods

The LSX_team5 submission focused on two established distance metrics that are inherently parameter-free, making them attractive for tasks with limited labeled data:

  • Word Mover's Distance (WMD): This method quantifies the distance between two texts by finding the minimum "cost" to transform the word embeddings of one text into the word embeddings of the other. It uses a linear programming approach to find an optimal mapping between words. To handle multilingualism, the authors utilized ConceptNet Numberbatch, a multilingual word embedding model. They also explored the impact of translating articles to English before generating embeddings. Word importance was weighted using TF-IDF derived from the WorldLex database.
  • Sentence Mover's Distance (SMD): An extension of WMD, SMD applies the same principle but operates on sentence embeddings instead of word embeddings. This allows for capturing more compositional information within sentences. For sentence representations, the authors leveraged Sentence-BERT, a powerful pre-trained transformer model. Crucially, since the Sentence-BERT model used was English-only, they employed a Neural Machine Translation (NMT) system to translate all multilingual articles into English before generating sentence embeddings.

Datasets

  • SemEval-2022 Task 8 Dataset: This dataset provided news article pairs across seven languages (en, pl, es, de, zh, ru, tr, fr, it, ar) and bilingual pairs for training and testing. The task required predicting an overall similarity score.
  • ConceptNet Numberbatch: A pre-trained multilingual word embedding model used for WMD.
  • WorldLex: A database of word frequencies from Twitter, blogs, and newspapers used for TF-IDF weighting.
  • Sentence-BERT (all-MiniLM-L12-v2): A pre-trained English sentence embedding model.
  • WMT's 2021 News Translation Task Model: An NMT system used to translate articles into English.

Key Insights and Ideas

  • Translation as a Unifier: The most significant insight is that translating all articles into a common language (English) before applying embedding-based similarity measures dramatically improves performance. This bypasses the need for complex multilingual embedding models for every language pair.
  • Sentence-Level is Generally Better: SMD, which operates on sentence embeddings, generally outperformed WMD. This highlights the importance of capturing sentence-level semantics and compositionality over just word-level similarity.
  • Parameter-Free Advantage: The reliance on parameter-free methods like WMD and SMD, combined with pre-trained embeddings, allowed for direct evaluation without the need for task-specific training data, a significant advantage for multilingual tasks.
  • NMT as a Crucial Component: The success of SMD was heavily dependent on the quality of the NMT system used to translate articles into English, underscoring the growing importance of robust machine translation in cross-lingual NLP tasks.

Key Results

The paper reports Pearson Correlation Coefficients (PCC) as the evaluation metric.

  • WMD vs. Translated WMD: WMD applied to translated articles consistently showed better results than WMD on original multilingual embeddings across most language pairs.
  • SMD Performance: SMD generally outperformed WMD, achieving higher PCC scores on the majority of language pairs. For instance, on the training data, SMD achieved a PCC of 0.78 for en-en, compared to WMD's 0.77 and translated WMD's 0.77.
  • Overall Submission: The team's final submission, using SMD with translated input texts and the Sentence-BERT model, achieved an overall Pearson Correlation Coefficient of 0.57 on the test set, placing them 23rd in the competition.
  • Test vs. Training Performance: For most language pairs, test set results were similar to or slightly better than training set results, which the authors attribute to the lack of task-specific training and thus no overfitting. However, some German language pairs showed notable performance fluctuations.

Limitations and Future Work

  • Data Extraction Issues: The authors noted that some articles, particularly German ones, could not be fully extracted, leading to incomplete content and negatively impacting similarity scores. They suggest overhauling the data scraping system.
  • German Language Performance: Performance on German language pairs was observed to be worse than average, potentially due to the extraction issues or inherent complexities of the language.
  • NMT Quality: While the NMT system performed well, its quality directly impacts the SMD results. Further improvements in NMT could lead to better similarity assessments.
  • Future Work: The authors suggest that NMT can improve multilingual word embeddings. They also highlight the potential of their approach for fast, low-cost implementation without GPUs, making it accessible for various applications.

Summary updated

BibTeX
@inproceedings{heil2022lsx,
  author = {Stefan Heil and Karina Kopp and Albin Zehe and Konstantin Kobs and Andreas Hotho},
  title = {{LSX\_team5 at SemEval-2022 Task 8: Multilingual News Article Similarity Assessment based on Word- and Sentence Mover’s Distance}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure