TL;DR
This paper introduces an "entity-enriched Siamese Transformer" that significantly improves multilingual news article similarity detection by incorporating explicit entity information (like locations, organizations, and dates) alongside the narrative. This novel approach achieved a second-place ranking in the SemEval-2022 Task 8 competition, demonstrating its effectiveness in understanding nuanced similarities between news events across different languages.
What Problem Does This Paper Address?
Accurately determining the similarity between news articles is crucial for various applications, including news clustering, duplicate detection, fact-checking, and tracking information spread. While existing methods often rely on the overall text content, they can struggle to capture the specific real-world events that news articles describe. The SemEval-2022 Task 8 specifically focuses on this challenge, assessing similarity based on aspects like geolocation, time, entities, and narratives, rather than just writing style or political slant. A key difficulty is that news articles, especially across different languages, may discuss the same event but use varying vocabulary and sentence structures, making direct textual comparison insufficient. Furthermore, the provided dataset exhibits an imbalance, with a disproportionate number of low-similarity pairs, posing a challenge for model training.
Methods
The core of the proposed system is an entity-enriched Siamese Transformer. This architecture uses two identical Transformer encoders (sharing weights) to process two news articles independently, generating document-level representations. The novelty lies in how these representations are enriched:
- Siamese Network for Narrative: The Siamese network architecture is employed to learn document-level representations that capture the overall narrative of the article. This is achieved by concatenating the title and article text and feeding it into a Transformer encoder. For longer articles, the initial part is used as a proxy for the narrative.
- Entity Extraction and Enrichment: To explicitly capture key event details, the authors use SpaCy's Named Entity Recognition (NER) to extract four types of entities: Geolocation (GEO), Organization (ORG), Date (DATE), and Quantity (QTY). For non-English articles, machine translation is used to enable entity extraction.
- Feature Combination: The cosine similarity scores of these extracted entity types between the two articles are computed. These entity-based similarity scores are then combined with the narrative similarity score from the Siamese Transformer. This combined representation is fed into a simple Multi-Layer Perceptron (MLP) with a sigmoid activation to predict the final similarity score.
- Semi-Supervised Learning for Data Augmentation: To address the imbalanced dataset, the authors employ a semi-supervised learning approach. They first augment the training data using BM25 sampling and machine translation to generate more high-similarity pairs. Then, they train the model on the original data and use it to predict similarity scores for the augmented data, effectively creating a larger, more balanced training set.
Datasets
The primary dataset used is from SemEval-2022 Task 8: Multilingual News Article Similarity.
- Training Data: Consists of 4,964 article pairs across seven languages (English, German, Spanish, Turkish, Polish, Arabic, and French), with a majority of monolingual pairs and one cross-lingual pair (German-English).
- Test Data: Includes 4,902 pairs, covering three additional languages (Chinese, Russian, and Italian) and more cross-lingual pairs.
- Augmented Data: The authors generated additional training data through three methods:
- Augment Data 1: Using titles from the SemEval training set as queries to retrieve articles from the same training set indexed in Elasticsearch (25,125 pairs).
- Augment Data 2: Machine translating English document pairs with high similarity scores ([0.5-1]) into other languages (2,845 pairs).
- Augment Data 3: Using titles from the SemEval training set as queries to retrieve articles from the "All the news" Kaggle dataset (143,000 articles) (34,818 pairs). This resulted in a total of 62,788 additional document pairs used for training.
Key Insights and Ideas
- Explicit Entities Matter: The most significant insight is that explicitly incorporating entity-level information (locations, organizations, dates) provides a crucial signal for news article similarity that is often missed by purely narrative-based models. This suggests that understanding what is being discussed (entities) is as important as how it's being discussed (narrative).
- Granularity of Similarity: The entity-enrichment approach allows the model to capture similarity at different granularities, from the overarching narrative to specific factual details, leading to a more robust assessment.
- Data Augmentation is Key: The semi-supervised learning strategy combined with diverse data augmentation techniques was vital for improving performance, especially in overcoming the dataset's imbalance and achieving better generalization.
- LaBSE's Strength: The Language-Agnostic BERT Sentence Representations (LaBSE) model, when combined with entity enrichment and augmented data, proved to be the strongest backbone, likely due to its extensive multilingual pre-training.
Key Results
- The proposed entity-enriched Siamese LaBSE model achieved the highest Pearson correlation coefficient of 0.80164 on the official test data, ranking 2nd in the SemEval-2022 Task 8 competition.
- Entity enrichment consistently led to statistically significant improvements (p-value < 0.01) across various Siamese Transformer backbones (mBERT, XLM-RoBERTa, USE, LaBSE) when trained on augmented data.
- The model performed better on monolingual pairs (average score 0.79833) than on cross-lingual pairs (average score 0.78205).
- An error analysis revealed that scraping issues with Arabic news content significantly impacted performance, and addressing these issues led to a substantial improvement in Arabic data correlation.
Limitations and Future Work
- Scraping Issues: The paper highlights that issues with web scraping tools, particularly for Arabic news, led to inaccurate content and negatively impacted model performance. This suggests a need for more robust data preprocessing pipelines.
- Overestimation of Similarity: The model occasionally overestimated similarity, especially when scraping issues led to irrelevant content being presented as news. The authors suggest that entity features might help mitigate this overestimation.
- Future Work: While not explicitly detailed, the success of entity enrichment suggests future work could explore more sophisticated ways to integrate entity information, potentially through graph-based methods or by fine-tuning models to better recognize and utilize specific entity types for similarity assessment. Further investigation into cross-lingual entity alignment could also be beneficial.
Summary updated