TL;DR
This paper presents a Transformer-based ensemble model for SemEval-2022 Task 8, which aims to evaluate the similarity between multilingual news articles based on objective facts rather than subjective style. The team found that the multilingual BERT (M-BERT) model, pre-trained on a vast corpus of 104 languages, significantly outperformed other Transformer models, achieving a Pearson Correlation Coefficient of 0.738.
What Problem Does This Paper Address?
Measuring text similarity is a long-standing challenge in Natural Language Processing (NLP), with applications ranging from information retrieval to machine translation. However, traditional methods often struggle with the nuances of real-world news, where different articles might cover the same event but with varying political stances or writing styles. SemEval-2022 Task 8 specifically targets this by requiring systems to assess the similarity of news articles based on objective facts (like time, geography, and entities) while ignoring subjective elements. A key challenge is the multilingual nature of the task, with datasets spanning multiple languages and even introducing new languages in the test set, demanding robust cross-lingual understanding.
Methods
The YNU-HPCC team employed a Transformer-based approach, leveraging pre-trained language models. Their system architecture consists of three main components:
- Tokenizer: This component preprocesses the input news articles by splitting them into tokens and adding special tokens like
[SEP]to delineate articles and[CLS]at the beginning of the first article for subsequent processing. - Model Layer: This is the core of the system, where various pre-trained Transformer models are fine-tuned. The paper experimented with BERT, ALBERT, ELECTRA, RoBERTa, and M-BERT. These models generate 768-dimensional word embeddings, and the
[CLS]token's embedding is used to capture the semantic information of the entire article pair for similarity calculation. - Fully Connected Layer & Loss Function: A fully connected layer maps the embeddings to a single similarity score. The Mean Squared Error (MSE) was chosen as the loss function after comparing it with cosine similarity.
Datasets
The primary datasets used were provided by the SemEval-2022 Task 8 organizers:
- Training Data: Consisted of 4,964 pairs of news articles with various language combinations (e.g., en-en, de-de, es-es, tr-tr, pl-pl, ar-ar, fr-fr). Scores for "Overall Similarity" and sub-categories like "Geography," "Entities," "Time," "Narrative," "Style," and "Tone" were provided.
- Test Data: Comprised 4,954 pairs of news articles. Notably, the test set included multilingual pairs and languages not present in the training data, posing a significant challenge for generalization.
Key Insights and Ideas
- Multilingual Pre-training is Key: The most striking insight is the superior performance of M-BERT. This strongly suggests that pre-training on a diverse, large-scale multilingual corpus (104 languages) provides a significant advantage for cross-lingual text similarity tasks compared to models pre-trained primarily on English.
- Objective Fact Extraction is Challenging: The task's focus on objective facts over subjective style highlights the difficulty in disentangling these aspects, even for advanced models.
- Transformer Models are Viable: The study confirms that Transformer-based models are effective for multilingual text similarity, with even the lower-performing models achieving reasonable scores.
- Hyperparameter Tuning Matters: Fine-tuning parameters like the loss function (MSE over cosine similarity) and batch size (8) was crucial for optimizing performance.
Key Results
The paper reports the following Pearson Correlation Coefficients on the test data:
- BERT: 0.464
- ALBERT: 0.543
- ELECTRA: 0.474
- RoBERTa: 0.475
- M-BERT: 0.738
M-BERT achieved the highest score, demonstrating its strong capability in handling multilingual news article similarity. The authors also noted that the highest Pearson's correlation score for M-BERT was achieved at epoch 3.
Limitations and Future Work
The authors acknowledge a potential limitation of M-BERT: while it can map learned structures to new vocabularies, it might not systematically learn transformations for different word orders in target languages, potentially affecting cross-script transfer accuracy.
Future work could explore:
- Improving the model's ability to handle systematic transformations across languages.
- Investigating more advanced techniques to better disentangle objective facts from subjective elements in news articles.
- Employing other interesting methods to further enhance performance on this challenging task.
Summary updated