HFL at SemEval-2022 Task 8: A Linguistics-inspired Regression Model with Data Augmentation for Multilingual News Similarity

Zihang Xu, Ziqing Yang, Yiming Cui, Zhigang ChenNAACL 2022

TL;DR

This paper presents a linguistics-inspired regression model that achieved first place in the SemEval-2022 Task 8 for multilingual news similarity. By combining data augmentation techniques, a multi-label loss function, and an adapted R-Drop regularization, the model significantly improved its ability to assess news similarity across various languages.

What Problem Does This Paper Address?

Accurately assessing the similarity between news articles is crucial for tasks like news clustering and information retrieval. However, doing this effectively across multiple languages presents a significant challenge. Traditional methods struggle with the nuances of different languages and the sheer volume of news data. SemEval-2022 Task 8 specifically aimed to tackle this by evaluating systems on their ability to measure the similarity of news articles in ten different languages, focusing on the events described rather than just writing style. The core challenge is to build a robust model that can generalize well across these diverse linguistic landscapes, especially when training data might be limited or imbalanced.

Methods

The authors built their system upon the powerful XLM-ROBERTa (XLM-R) multilingual pre-trained language model. Their approach incorporated several key techniques:

  • Data Augmentation: To address the limited training data, they employed two main strategies:
  • Back Translation: Non-English articles were translated to English and then back to their original language to create synthetic training examples.
  • Translate Train: To fill gaps in specific language pairs (especially those present in the test set but not the training set), they generated new training data by translating existing articles.
  • Multi-label Loss: While the evaluation focused solely on the "Overall" similarity score, the authors hypothesized that incorporating other sub-dimensions (Geography, Entities, Time, Narrative, Style, Tone) into the loss function could indirectly improve performance. They experimented with assigning weights to these dimensions.
  • Adapted R-Drop: This is a regularization technique that aims to minimize the difference between predictions from different forward passes of the model. The authors adapted the standard R-Drop by replacing its KL-divergence loss with Mean Squared Error (MSE) loss, making it suitable for this regression task.
  • Head-tail Combination: Recognizing that news titles and the initial/final paragraphs are often highly informative, they experimented with different ways of combining these "head" and "tail" sections of articles for representation.
  • Extra Linear Layers: To enhance the model's capacity to map sentence representations to similarity scores, they added a few extra linear layers on top of XLM-R's output, exploring both with and without activation functions.

Datasets

The primary dataset used was the SemEval-2022 Task 8 dataset. This dataset consists of:

  • Training Set: Approximately 4,964 samples with 8 language pairs.
  • Test Set: Approximately 4,593 samples with 18 different language pairs.

The similarity scores were rated on a 4-point scale across seven sub-dimensions, with only the "Overall" score used for final evaluation. The authors also generated additional data through their data augmentation techniques.

Key Insights and Ideas

  • Data Augmentation is Crucial: The paper strongly emphasizes that increasing the quantity and diversity of training data, even through synthetic means like back-translation and translate-train, significantly boosts performance.
  • Head Matters (A Lot): The "head" (title) of a news article appears to be the most critical component for similarity assessment, contributing more than the "tail" or the entire article content when used in isolation. However, combining head and tail still offers benefits.
  • Multi-label Loss Helps, But "Overall" Dominates: While incorporating other similarity dimensions into the loss function provides a performance boost, the "Overall" score remains the most important factor for this task.
  • Adapted R-Drop is a Powerful Regularizer: The adapted R-Drop method, even with a marginal increase in forwarding passes, consistently improves results, suggesting its effectiveness in preventing overfitting and improving generalization.
  • More Parameters Can Help: Adding extra linear layers on top of XLM-R, while not showing a perfectly linear improvement, generally leads to better performance, indicating the need for more capacity in the final regression stage.

Key Results

The system achieved a Pearson's Correlation Coefficient (Pearson's CC) of 0.818 on the official evaluation set, securing 1st place in the SemEval-2022 Task 8 competition.

  • Data augmentation alone improved the baseline performance from 83.49 to 85.86 (Pearson's CC).
  • The combination of strategies, including Head-tail Combination, Multi-label Loss, Adapted R-Drop, and Extra Linear Layers, further pushed the performance to achieve the top ranking.
  • Individual language pair results showed that the model performed worse on language pairs that were rare or absent in the training set, such as German to Polish (64.31).

Limitations and Future Work

  • Limitations:
    • The model's performance is significantly impacted by the availability of training data for specific language pairs. Rare language pairs or those with complex linguistic features (like Polish) pose a challenge.
  • The paper notes that a two-tower architecture (where each article in a pair is processed by a separate, shared-parameter model) led to a decrease in performance, highlighting the importance of direct semantic interaction within the model.
  • Future Work:
    • The authors suggest further enriching the training data.
  • They propose analyzing languages individually and leveraging their unique rules and features, rather than relying solely on general pre-trained models, to improve performance, especially for minority languages.

Summary updated

BibTeX
@inproceedings{xu2022hfl,
  author = {Zihang Xu and Ziqing Yang and Yiming Cui and Zhigang Chen},
  title = {{HFL at SemEval-2022 Task 8: A Linguistics-inspired Regression Model with Data Augmentation for Multilingual News Similarity}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure