Wolfies at SemEval-2022 Task 8: Feature extraction pipeline with transformers for Multi-lingual news article similarity

Nikhil Goel, Ranjith Reddy BommidiNAACL 2022

TL;DR

This paper tackles the challenge of determining news article similarity across multiple languages. The authors propose a feature extraction pipeline combined with transformer-based models, demonstrating significant improvements over a baseline cosine similarity approach, especially when incorporating domain-specific features.

What Problem Does This Paper Address?

The core problem this paper addresses is accurately measuring the similarity between news articles, particularly when those articles are in different languages. While general document similarity approaches exist, they often struggle with the nuances of specific similarity metrics (like geolocation or entity mentions) and the complexities introduced by multilingualism. Existing methods might rely on translating all articles to a common language, which can introduce errors, or use generic embeddings that don't capture domain-specific information effectively. This work aims to build a robust, multilingual system that leverages feature engineering and specialized models to improve similarity assessment.

Methods

The authors employ a multi-stage approach. First, they utilize a pre-trained sentence similarity model, specifically S-BERT (Sentence-BERT) and its variants like miniLM. For a baseline, they calculate cosine similarity on the embeddings of article pairs.

The key innovation lies in their domain-specific feature extraction pipeline. For objective metrics like geolocation, entities, and time, they use the Stanza NLP library to extract relevant named entities. This extracted information is then fed into the model. For more subjective metrics like narrative, style, and tone, they rely more heavily on the pre-trained embeddings themselves.

To further enhance performance, they train a three-layer feed-forward neural network (FFN) on top of the extracted features and embeddings. This FFN learns domain-specific knowledge, aiming to improve the similarity scores beyond what a simple cosine similarity can achieve. They also explore a larger BERT model (Doc S-BERT) and a multilingual version of miniLM to handle cross-lingual similarity.

Datasets

The primary dataset used is from the SemEval-2022 Task 8, which contains news articles in multiple languages. The dataset includes pairs of articles rated on seven different similarity metrics. The authors split this dataset into training and testing sets with a 67:33 ratio. They note that the dataset is "tiny" and did not use a separate validation set.

Key Insights and Ideas

  • Domain-specific feature extraction is crucial: Simply using generic embeddings and cosine similarity is insufficient for many similarity metrics, especially objective ones like geolocation and entities. Extracting specific information (e.g., locations, dates) significantly improves performance.
  • Feed-forward networks add domain knowledge: Training a simple FFN on top of pre-trained embeddings allows the model to learn task-specific nuances, leading to substantial improvements over the baseline.
  • Multilingualism is challenging but achievable: While a multilingual miniLM model showed promise, it sometimes underperformed compared to a single-language model, especially for style and tone metrics. However, the multilingual approach offered more consistent results across various languages, suggesting its potential for broader applicability.
  • Limitations of generic models: The paper highlights that generic sentence embedding models might struggle with certain tasks. For instance, the miniLM model's input truncation (256 words) can lead to information loss in longer articles.

Key Results

The paper reports significant improvements in Mean Squared Error (MSE) loss across various similarity metrics when using their proposed feature extraction and FFN approach compared to the baseline.

  • For Geography, the MSE improved from 0.15 (baseline) to 0.135 (miniLM FFN).
  • For Time, the MSE dropped from 0.132 to 0.074.
  • Entity similarity saw the most dramatic improvement, with MSE decreasing from 0.288 to 0.106.
  • Style and Tone metrics also showed substantial gains, with MSE reducing from around 0.19 to approximately 0.075.
  • The multilingual miniLM model achieved a Pearson correlation coefficient of 0.288 on the test data, which is presented as the official task metric.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset size: The dataset was considered "tiny," potentially limiting the generalization capabilities of the models.
  • Stanza's language support: The Stanza library, used for feature extraction, did not fully support all languages in the dataset (specifically 'pl' and 'tr'), impacting the performance of the multilingual model.
  • Input truncation: The miniLM model's 256-word input limit can lead to information loss in longer articles.
  • Translation errors: Their initial approach of translating non-English articles to English for testing was noted as error-prone.

Future work suggested includes:

  • Exploring dependency parse of sentences as input for style and tone metrics.
  • Investigating alternative parsers for Named Entity Recognition.
  • Developing or utilizing different multilingual models that offer broader language support.
  • Potentially using larger models that can handle longer document contexts more effectively.

Summary updated

BibTeX
@inproceedings{goel2022wolfies,
  author = {Nikhil Goel and Ranjith Reddy Bommidi},
  title = {{Wolfies at SemEval-2022 Task 8: Feature extraction pipeline with transformers for Multi-lingual news article similarity}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure