TL;DR
This paper tackles the challenge of determining how similar two news articles are, even if they're in different languages. They propose a Siamese neural network architecture that leverages pre-trained multilingual language models and find that data augmentation is the most effective strategy for improving performance.
What Problem Does This Paper Address?
In today's interconnected world, news spreads rapidly across various languages and sources. However, articles covering the same event can differ significantly in their regional biases, writing style, conciseness, and intended audience. This can lead to confusion and misinformation. The core problem this paper addresses is the need for an automated system that can accurately measure the similarity of news articles in a multilingual setting, focusing on the factual coverage of events rather than subjective stylistic elements. This is crucial for tasks like news aggregation, trend analysis, and identifying propaganda.
Methods
The authors employ a Siamese neural network architecture. This architecture uses two identical encoders (with shared weights) to process each news article independently. The outputs from these encoders are then combined and fed through fully connected layers to predict a similarity score.
Key methodological choices include:
- Encoder Models: They experimented with two powerful pre-trained multilingual transformer models: XLM-ROBERTa (XLM-R) and Multilingual DistilBERT. DistilBERT was chosen for some experiments due to its smaller size, which can be beneficial in low-data scenarios.
- Feature Engineering: Beyond just the raw news text, they explored incorporating metadata such as title, description, keywords, and tags. They also experimented with Named Entity Recognition (NER) features.
- Representation Aggregation: Various strategies were tested to combine the representations of the two articles, including simple concatenation and element-wise operations.
- Data Augmentation: A significant focus was placed on augmenting the training data by randomly permuting sentence order, creating datasets 3, 4, and 5 times the original size.
- Output Activation: Different activation functions (Sigmoid, ReLU, linear) were evaluated for the final output layer.
- Ensembling: They explored combining predictions from multiple models using simple and weighted averaging.
Datasets
The primary dataset used is the SemEval-2022 Task 8: Multilingual News Article Similarity dataset. This dataset consists of pairs of news articles annotated with similarity scores ranging from 1 to 4. The training and test sets were carefully constructed to ensure no language overlap between them, highlighting the need for robust multilingual representations. The paper also mentions that 10% of the training data was used as a validation set.
Key Insights and Ideas
- Plain Text is King (Mostly): Surprisingly, using only the plain text of the news articles (XLM-TXT, DB-TXT) as input to the encoder often yielded the best results. Incorporating metadata alone or even concatenated with text (XLM-MTD-TXT) resulted in suboptimal performance.
- Data Augmentation is a Game-Changer: Augmenting the training data by permuting sentences proved to be the most effective strategy, leading to significant performance improvements on both validation and test sets. Augmenting the data four times the original size (DB-DA4) showed the best results among the tested augmentation factors.
- DistilBERT's Advantage: The smaller, distilled version of BERT (DistilBERT) often performed comparably or even better than the larger XLM-ROBERTa across similar experiments. This suggests that for this specific task and dataset size, the reduced parameter count of DistilBERT might help mitigate overfitting.
- Aggregation Matters: The strategy of concatenating the absolute difference and average of the two article representations (DB-CAT3) performed better than other concatenation strategies.
- Sigmoid for the Win: Sigmoid activation at the output layer consistently led to better training trajectories and final results compared to ReLU or linear activations.
Key Results
The paper reports performance using Pearson's Correlation Coefficient (PCC), Mean Absolute Percentage Error (MAPE), and Mean Squared Error (MSE).
- On the validation set, the best performing experiment was DB-DA4, achieving a PCC of 0.58, MAPE of 0.41, and MSE of 0.94.
- On the test set, DB-DA4 achieved a PCC of 0.441.
- The simple text-only approach DB-TXT achieved a validation PCC of 0.55.
- XLM-TXT achieved a validation PCC of 0.53.
Limitations and Future Work
The authors acknowledge that globally representing a news article by a single vector after capping its length might not be the optimal approach. They suggest that a solution exploiting information at a more granular level, possibly through explicit feature extraction or implicit detection of features across languages, could yield better results.
A key conjecture for the poorer performance on the test set (compared to validation) is that finetuning the multilingual transformer models might have negatively impacted their effectiveness for languages present in the test set but not in the training/validation sets. They propose that future work should focus on solutions that train model parameters for the task without disrupting the inherent multilingual capabilities of the base models, which should lead to a more scalable solution.
Summary updated