TL;DR
This paper presents a system for evaluating semantic textual similarity between news articles at the document level. They leverage a diverse set of semantic features, including key terms, named entities, and document summaries, to train a supervised model. While achieving a respectable Pearson correlation of 0.706, they highlight the challenges of multilingualism and data quality in this complex task.
What Problem Does This Paper Address?
The core problem this paper tackles is measuring document-level semantic textual similarity (STS), specifically for news articles. Unlike previous STS tasks that often focused on sentences or short text snippets, SemEval-2022 Task 8 required comparing entire news documents. This presents a significant challenge due to the increased complexity, potential for multiple languages within a single pair, and the need to capture nuanced similarities beyond simple keyword overlap. Accurately modeling this similarity is crucial for applications like document clustering, classification, and understanding how different news outlets cover the same events.
Methods
The authors employ a feature-based supervised learning approach. Their system extracts a variety of semantic features from document pairs, which are then used to train a classification or regression model. Key features include:
- Title Similarity: Using sentence transformers with a pretrained encoder model.
- Text Summarization Similarity: Measuring similarity between summaries generated by pretrained transformer models (e.g., BART, Sequence-to-Sequence models).
- Keyword/Key Term Extraction: Identifying common terms using the PKE toolkit and Word2Vec for semantic proximity.
- Named Entity Recognition (NER): Extracting and comparing common named entities (persons, organizations, locations, dates) using libraries like Spacy and Stanza.
- Geographical Entity Proximity: Calculating similarity based on the proximity of detected geographical entities.
- Zero-Shot Topic Classification: Assigning documents to predefined topics (politics, sport, health, economy, technology) and measuring topic overlap.
- Sentiment Analysis: Comparing the sentiment polarity (positive, negative, neutral) of the texts.
For multilingual aspects, they either build dedicated models for main languages (English, French, Spanish, German, Arabic, Turkish) or translate texts into English using a pivot language. They also experimented with different classification (Random Forest, Logistic Regression) and regression algorithms, ultimately using PyCaret for model selection and optimization.
Datasets
The paper utilizes the SemEval-2022 Task 8 dataset for both training and evaluation.
- Training Corpus: Contained 4,964 pairs of news articles in seven languages (English, French, Spanish, German, Polish, Arabic, Turkish), with mostly monolingual pairs and one cross-lingual pair (DE_EN).
- Test Corpus: Consisted of 4,953 pairs of news articles across ten languages, including three new languages (Russian, Chinese, Italian) and eight new cross-lingual pair types.
The document pairs were annotated by 1-8 annotators on a 4-point scale for categories like "Geography," "Entities," "Time," "Narrative," "Overall," "Style," and "Tone." The "Overall score" was used for evaluation.
Key Insights and Ideas
- Multilingualism is a Major Hurdle: The performance significantly drops when dealing with languages not well-represented in the training data or when relying on translation. The authors note that their German model, which was not optimized, performed poorly.
- Data Quality Matters: Scraping issues (incomplete or incorrect text) led to significant deviations in evaluation, often resulting in artificially low scores for otherwise similar documents.
- Feature Diversity is Key: Combining various semantic features (entities, keywords, summaries, topics) proved more effective than relying on a single type of information.
- Translation Drift: While translating to English for cross-lingual pairs often yielded good results (e.g., ZH_EN), direct monolingual evaluation for some languages (e.g., ZH_ZH) showed a performance decrease.
Key Results
The system achieved a final Pearson correlation score of 0.706 on the test corpus.
- Strong Performance in English and French: Achieved Pearson correlations of 0.82.
- Good Performance in Spanish and Turkish: Achieved 0.75 and 0.74 respectively.
- Disappointing Performance in German and Arabic: Scores were 0.60 and 0.64, attributed to poor optimization and technical difficulties.
- Cross-lingual performance: ZH_EN pairs translated into English scored 0.79, while direct ZH_ZH scored 0.69. IT_IT and SP_IT also showed a performance reduction compared to English-based evaluations.
Limitations and Future Work
- German Model Optimization: The authors acknowledge that the German model was not optimized and could have performed much better.
- Arabic Model Difficulties: Technical challenges prevented them from achieving a correct Arabic model.
- Data Imbalance: The training dataset, particularly for English, was unbalanced, requiring rebalancing.
- Reliance on Translation: For languages not directly supported, translation into English was used, which can introduce errors and performance degradation.
Future work could involve further optimization of models for underperforming languages, exploring more robust methods for handling data scraping errors, and investigating more advanced multilingual modeling techniques that avoid reliance on pivot languages.
Summary updated