TL;DR
This paper presents a lightweight and language-agnostic news similarity classifier for the SemEval-2022 Task 8. The system uses a combination of lexicographic and multilingual embedding features fed into a random forest classifier, achieving competitive results by framing the task as classification rather than regression.
What Problem Does This Paper Address?
The paper tackles the challenge of identifying similar news articles across multiple languages, a crucial task for media monitoring and information aggregation. Traditional approaches often struggle with the sheer volume of news and the complexity of multilingualism. The authors aim to develop a system that is both efficient (lightweight) and capable of handling diverse languages without requiring language-specific models. They also highlight the practical need to determine not just if articles are similar, but to what extent, and how to do this cost-effectively for millions of articles daily.
Methods
The core of the system is a Random Forest (RF) classifier. Instead of treating the task as a regression problem (predicting a continuous similarity score), the authors frame it as a classification problem with four discrete labels: "similar" (sim), "close similarity" (close), "different" (diff), and "unrelated" (unrel). This simplification, they argue, leads to more coherent results.
The features used to train the RF classifier are a blend of:
- Lexicographic Features: These capture similarities based on word overlap and multi-word expressions (MWEs). MWEs are identified using a RAKE-like keyword extraction method, with a tolerance for slight variations in MWEs.
- Embedding-based Features: These leverage LASER (Language-Agnostic Sentence Embeddings) which are multilingual sentence embeddings designed for cross-lingual transfer. These embeddings are used to represent titles, descriptions, first sentences, and snippets of the articles.
- Non-linguistic Features: The only non-linguistic feature considered is the publication date difference between articles.
The system processes four key parts of a news item: title, description, first sentence, and snippet (first four sentences within 512 characters). Features are computed for each part and also for cross-field comparisons (e.g., title vs. description).
Datasets
The primary dataset used is from the SemEval-2022 Task 8: Multilingual news article similarity.
- Training Data: Consists of 4964 article pairs covering 9431 articles in 16 languages. The majority (99%) are from 7 main languages. Pairs are associated with a similarity score from 1.0 (maximum) to 4.0 (minimum), which the authors use to derive their four classification labels.
- Test Data: Consists of 4890 pairs over 9715 articles in 10 languages. Notably, the test set includes a higher proportion of cross-language pairs (15%) and introduces Chinese, which was absent in the training data.
The authors also detail the challenges in acquiring data from the provided URLs, employing multiple scrapping strategies and noting that about 7% of the test data was impossible to download, with Chinese articles being particularly problematic.
Key Insights and Ideas
- Classification vs. Regression: Framing the news similarity task as classification (discrete labels) rather than regression (continuous score) surprisingly yielded more coherent and better-performing results, especially when compared to a rounded regression output. This suggests that the nuances of similarity might be better captured by distinct categories for this task.
- Lightweight Features Perform Well: The combination of simple lexicographic features (word overlap, MWEs) and multilingual embeddings (LASER) proved effective. This is significant because it allows for efficient processing, crucial for large-scale applications.
- Language Agnosticism is Achievable: The use of LASER embeddings and careful feature engineering allowed the system to perform reasonably well across different language pairs, demonstrating the viability of language-agnostic approaches.
- Cross-language Challenges: The system performed better on same-language pairs than on cross-language pairs. This suggests that while language-agnostic embeddings help, lexical cues are still highly valuable, and their effectiveness can be diminished across languages.
- Date Feature's Limited Generalizability: The publication date difference feature, while included, showed potential bias due to the dataset's construction (news fetched over years with a significant portion of related articles). The authors suggest it might not generalize well to more uniform datasets.
Key Results
- The Random Forest classifier (RF-C) achieved a micro F1 score of 93.6% and macro F1 of 93.7% on the training data (5-fold cross-validation).
- The system's label distribution was significantly closer to the ground truth than that of a regression-based approach (measured by Jensen-Shannon divergence).
- On the test dataset, the performance dropped, with a micro F1 of 46.5% and macro F1 of 41.5% on all downloaded articles. This drop is attributed to data acquisition issues and the inherent difficulty of cross-language comparison.
- Binary classification (e.g., "similar" vs. "rest") performed significantly better than multilabel classification, achieving higher F1 scores.
- Lexicographic features alone performed surprisingly well, only slightly underperforming embedding-based features on the training set.
Limitations and Future Work
- Data Acquisition Issues: A significant limitation was the inability to download and extract text for a portion of the test data (especially Chinese articles), leading to incomplete data and potentially biased evaluation.
- Classification Simplification: While beneficial for performance, the discrete classification approach loses the fine-grained similarity information present in the original regression scores.
- Cross-language Performance Gap: The system's underperformance on cross-language pairs indicates room for improvement.
- Future Work: The authors suggest exploring:
- Using language pair information as features.
- Incorporating more advanced features, such as named entity extraction.
- Evaluating exclusively on datasets with complete and correctly extracted data.
- Investigating the impact of using only snippets versus full articles.
Summary updated