TL;DR
This paper presents a system for determining the informational similarity between news articles across different languages. By leveraging multilingual sentence embeddings and a novel convolutional neural network, the WueDevils team achieved a strong 9th place in the SemEval-2022 Task 8 competition, demonstrating the effectiveness of their ensemble approach.
What Problem Does This Paper Address?
The core challenge this paper tackles is measuring the informational content similarity between news articles written in various languages. This is crucial for applications like understanding global news coverage, but it's tricky because direct translation can lose nuance, and different languages have distinct structures and writing styles. The SemEval-2022 Task 8 specifically aimed to evaluate systems on their ability to identify what an article is about, rather than how it's written, across a diverse set of languages including English, German, Spanish, Turkish, Polish, French, Arabic, Italian, Russian, and Chinese. A key difficulty lies in the multilingual nature of the task, requiring models that can handle diverse linguistic inputs and overcome limitations in existing monolingual tools for tasks like sentence splitting and named entity recognition.
Methods
The WueDevils system employs an ensemble approach built upon several key components:
- Multilingual Sentence Embeddings: They utilize two primary methods for generating sentence embeddings:
- Universal Sentence Encoder (USE): This model provides fixed embeddings, and the similarity is calculated using an arccosine-based text similarity measure, converting cosine values into angular distances.
- Sentence-BERT (SBERT): For SBERT, the choice of model depends on the language pair. If articles are in the same language, a specialized SBERT model is used if available; otherwise, a default multilingual SBERT model is employed. Similarity is computed using cosine similarity.
- Pair-wise Similarity Matrices: For each pair of articles, they construct similarity matrices by comparing every sentence in one article to every sentence in the other, using both USE and SBERT embeddings.
- SimCNN (Similarity Convolutional Neural Network): Inspired by the TextCNN architecture, their SimCNN takes the concatenated SBERT and USE similarity matrices as input. This network is designed to learn complex patterns within these matrices to predict a similarity score.
- Ensemble Scoring:
- Matrix Operations: Simple maximum and average operations are applied to the similarity matrices, similar to prior work (Ginzburg et al., 2021), to derive four initial scores.
- Random Forest Regressor: These initial scores, along with the SimCNN's output, are fed into a random forest regressor (trained on 100 decision trees) to produce a consolidated score.
- Publishing Date Score (Optional): A score based on the difference in publishing dates is also computed. If the difference is significant, it can positively influence the final score. This date score is then combined with the random forest score via a weighted average.
Datasets
The primary dataset used is the SemEval-2022 Task 8 training data, consisting of 4,964 labeled article pairs. A static 10% subset (470 pairs) was reserved for model validation. The evaluation data was used for the final assessment, but it did not provide gold standard scores for direct comparison during development.
Key Insights and Ideas
- Ensemble Power: Combining multiple scoring mechanisms (matrix operations, SimCNN, and a random forest) proved more robust than relying on a single method.
- Multilingual Embeddings are Key: The success hinges on using pre-trained multilingual models like USE and SBERT, which can bridge language barriers.
- SimCNN for Matrix Patterns: The SimCNN architecture effectively captures nuanced relationships within the sentence similarity matrices, improving upon simpler aggregation methods.
- Publishing Date as a Signal: Incorporating the publishing date difference, especially for articles that are far apart in time, can be a valuable signal for detecting differing topics.
- Language Distribution Shift: The evaluation set featured a significant increase in non-Latin languages and previously unseen language combinations (especially Chinese-Chinese), which posed challenges for the system and highlighted the need for more robust multilingual sentence tokenization.
Key Results
The WueDevils system achieved 9th place in the SemEval-2022 Task 8 competition with a Pearson correlation coefficient of 0.759 on the evaluation set.
- The random forest regressor alone showed good performance, but combining it with the publishing date score led to an improvement.
- Performance varied significantly across language pairs, with Chinese-Chinese pairs showing a lower Pearson score (0.64) compared to English-English pairs.
- The system achieved a Pearson correlation score of approximately 0.6 even for the most challenging language combinations.
Limitations and Future Work
The authors acknowledge several limitations:
- Sentence Tokenization: The current sentence tokenizer primarily works reliably for English and similar languages. For languages with different structures and alphabets (like Chinese or Japanese), this is a significant bottleneck, leading to worse performance.
- Named Entity Recognition: The system could be further improved by incorporating named entities (locations, persons) into the similarity calculation. However, this requires more sophisticated multilingual named entity recognition algorithms, which are not yet widely available or reliable.
Future work could focus on developing more robust multilingual sentence splitting and exploring advanced multilingual NER techniques to enhance the system's accuracy across a wider range of languages and linguistic complexities.
Summary updated