EMBEDDIA at SemEval-2022 Task 8: Investigating Sentence, Image, and Knowledge Graph Representations for Multilingual News Article Similarity

Elaine Zosa, Emanuela Boros, Boshko Koloski, Lidia PivovarovaNAACL 2022

TL;DR

This paper tackles the challenge of determining how similar news articles are, even when they're in different languages. The EMBEDDIA team explored using text, images, and knowledge graphs, finding that while multimodal approaches show promise, sentence embeddings alone remain the most effective for this task.

What Problem Does This Paper Address?

In today's information-saturated world, news aggregators and researchers alike need to efficiently group and understand news stories. A significant challenge arises when these stories are published in multiple languages, making direct comparison difficult. This paper addresses the problem of multilingual news article similarity, aiming to develop systems that can accurately assess how much two news articles, potentially in different languages, cover the same topic or event. This is crucial for tasks like building comprehensive news archives, tracking global events, and monitoring information spread, especially during critical periods like disease outbreaks.

Methods

The EMBEDDIA team employed a multi-pronged approach, investigating various representations of news articles:

  • Semantic Textual Similarity (STS): This was the core of their textual analysis. They utilized Sentence-BERT (SBERT), a powerful model for generating semantically meaningful sentence embeddings.
  • Similarity-based: They directly computed cosine similarity between SBERT embeddings of article titles and content.
  • Machine Translation: To handle multilingualism, they translated non-English articles to English using Google Translate and then applied SBERT.
  • Regression-based: They also fine-tuned SBERT models on the similarity scores, treating the task as a regression problem.
  • Image Similarity & Regression: They leveraged pre-trained models like CLIP and ViT to extract image embeddings.
  • They computed cosine similarity between paired images.
  • They also explored a "cross-image" strategy that combined image-to-image, text-to-text, and image-to-text similarities.
  • Regression-based approaches were also tested using image embeddings.
  • Knowledge Graph (KG) Similarity & Regression: They utilized the Wikidata5m knowledge graph to extract entity and relation embeddings.
  • They combined various KG embedding techniques (e.g., TransE, RotatE) with LSA and SBERT representations.
  • Both similarity-based and regression-based methods were applied to KG features.
  • Text & Image Regression Models: This approach combined textual and image features. They used pre-trained multilingual language models (like XLM-RoBERTa) and image encoders (like CLIP and ViT), concatenating their representations for a final regression task.

Datasets

The primary dataset used was from SemEval-2022 Task 8 (Multilingual News Article Similarity).

  • Training Data: 4,964 article pairs across seven languages (English, German, Spanish, Arabic, Polish, Turkish, French) with gold standard similarity scores for six dimensions and an overall score.
  • Evaluation Data: 4,902 article pairs, including three "surprise" languages (Chinese, Italian, Russian) not present in the training data.
  • The dataset also included article titles, keywords, topics, and links to representative images.

Key Insights and Ideas

  • The Unbeatable Power of Text: Despite exploring multimodal approaches, the paper strongly concludes that semantic textual similarity, particularly using sentence embeddings, remains the most effective method for news article similarity.
  • Machine Translation as a Bridge: Translating articles to a high-resource language (English) significantly improved performance, demonstrating its utility in cross-lingual tasks.
  • Images Offer Limited Gains (in this context): While images can be informative, their impact on similarity prediction was less pronounced than text. This was partly attributed to the fact that only about 60% of articles had usable images, and often only one article in a pair had a representative image.
  • Knowledge Graphs Show Promise but Face Challenges: KG-based methods showed competitive results, but performance dropped for non-English articles due to the KG's English-centric nature and translation losses.
  • Specific Image Types Matter: Images with clear text or faces were more beneficial for image-based similarity prediction, suggesting that the content of the image is key.

Key Results

  • The EMBEDDIA team ranked fifth overall in the multilingual and cross-lingual categories of SemEval-2022 Task 8.
  • They achieved a strong second place in the English-only results.
  • Their best-performing model for both categories was SBERT fine-tuned on the STS task.
  • Specifically, the SBERT model fine-tuned on machine-translated articles (1d) achieved a Pearson correlation of 0.7835.
  • Text-based models generally outperformed image-based or KG-based models in terms of correlation scores. For instance, the best text-based model (1d) achieved 0.7835, while the best image-based model (2a) achieved 0.2991.

Limitations and Future Work

  • Image Data Sparsity: The limited availability and representativeness of images across article pairs was a significant limitation.
  • KG Language Bias: The reliance on English-centric knowledge graphs hindered performance on non-English articles.
  • False Positives from Phrasing: Even strong textual models can be misled by similar phrasing in articles that cover different events, as demonstrated in their error analysis.

Future work could involve:

  • Developing more adaptable methods for incorporating images, especially for handling missing images.
  • Exploring multilingual knowledge graph representations.
  • Investigating ways to mitigate the impact of superficial textual similarities when events differ.

Summary updated

BibTeX
@inproceedings{zosa2022embeddia,
  author = {Elaine Zosa and Emanuela Boros and Boshko Koloski and Lidia Pivovarova},
  title = {{EMBEDDIA at SemEval-2022 Task 8: Investigating Sentence, Image, and Knowledge Graph Representations for Multilingual News Article Similarity}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure