TL;DR
This paper tackles the challenge of accurately comparing news articles across different languages, even when they have varying writing styles and potential biases. The team developed a multi-task learning approach that leverages machine translation and text ranking to improve cross-lingual article similarity assessment, achieving a respectable 16th place in the SemEval-2022 competition.
What Problem Does This Paper Address?
The internet is increasingly multilingual, but most information verification and analysis tools are English-centric. This creates a significant hurdle for understanding and comparing news content globally. The SemEval-2022 Task 8 specifically aimed to address this by evaluating systems on their ability to determine the similarity between news articles written in different languages. The core challenge lies in overcoming differences in writing style, tone, political leanings (hyperpartisanship), and even the inherent biases of news outlets, all while handling multiple languages. Simply translating articles isn't enough, as nuances and cultural contexts can be lost.
Methods
The proposed system employs a four-module pipeline:
- Extraction: Gathers titles, descriptions, and the main text from news articles. They found that merging meta-descriptions with descriptions ("extra text") provided valuable context and improved scores.
- Machine Translation: Utilizes the OPUS-MT transformer-based neural machine translation model to translate all articles into English. This standardizes the language, allowing for easier comparison.
- Text Ranking: Addresses the issue of long articles (exceeding typical model input limits) and irrelevant sentences. They concatenate articles, split them into sentences, and use fastText embeddings and a TextRank-like algorithm to identify and rank the most important sentences, effectively filtering out noise and focusing on core content.
- Multi-task Training: This is the core innovation. They train a single DeBERTa-based encoder with multiple task-specific "heads." The main task is the multilingual news article similarity. They also incorporate auxiliary tasks: hyperpartisan news detection and semantic textual similarity (STS). The intuition is that learning these related tasks simultaneously helps the model develop a more robust understanding of text, leading to better performance on the primary similarity task. This approach allows for knowledge transfer between tasks.
Datasets
The primary dataset used is from SemEval-2022 Task 8: Multilingual News Article Similarity. This dataset includes pairs of news articles across ten languages (English, German, French, Italian, Polish, Russian, Chinese, Turkish, Arabic, and Spanish) with overall similarity scores and other dimensional scores (Geo-location, Time, Shared Entities, Shared Narratives). They also mention using a hyperpartisan dataset and the STS benchmark dataset for their subtasks.
Key Insights and Ideas
- Translation as a Bridge: Standardizing all articles to English via machine translation is a crucial first step for cross-lingual comparison, enabling the use of powerful monolingual models.
- Text Ranking for Focus: Filtering out less relevant sentences using a TextRank-inspired approach is vital for handling long articles and improving the signal-to-noise ratio.
- Multi-task Learning Synergy: Training on related tasks (hyperpartisanship, STS) alongside the main similarity task significantly boosts performance. This suggests that understanding why an article might be biased or what its core semantic meaning is, helps in determining its overall similarity to another.
- DeBERTa's Strength: The DeBERTa model proved to be a strong encoder for this multi-task learning setup.
Key Results
The team's best model achieved a Pearson score of 0.733 on the test set, ranking them 16th out of all participating teams in SemEval-2022. They observed that their model performed particularly well on English-English pairs and pairs involving French and Spanish, likely due to the quality of the machine translation for these languages. Performance dipped on German articles and other languages not present in the training set, highlighting the challenges of cross-lingual generalization.
Limitations and Future Work
The authors acknowledge that the performance on certain languages, particularly German, was not as strong. They attribute this to the machine translation module's limitations for those specific language pairs, as indicated by lower BLEU scores. The test dataset also contained languages not present in the training set, posing a generalization challenge.
Future work suggestions include:
- Improved Machine Translation: Exploring more advanced translation systems like AWS Translate for better English sentence generation.
- Document-level Pre-trained Models: Investigating models specifically designed for document-level understanding, such as DocMT5, which was unavailable at the time of their research.
Summary updated