MAD-TSC: A Multilingual Aligned News Dataset for Target-dependent Sentiment Classification

Evan Dufraisse, Adrian Popescu, Julien Tourille, Armelle Brun, Jérôme DeshayesACL 2023 · DOI

Introduction

Target-dependent Sentiment Classification (TSC) aims to identify sentiment expressed toward specific entities within a text. While well-established for social media and product reviews, TSC in the news domain remains challenging due to implicit sentiment and complex sentence structures. Researchers have introduced MAD-TSC, the first large-scale, multilingual, and sentence-aligned dataset designed to bridge this gap. By providing professionally translated versions of 4,714 sentences across eight languages, the dataset enables rigorous cross-lingual comparisons.

The Challenge of News Sentiment

Unlike tweets, news articles often contain multiple entities and nuanced language. Sentiment is frequently buried in long sentences where a single sentence might express different sentiments toward different people. Furthermore, most TSC research has focused on English. Existing multilingual datasets typically lack alignment—meaning they don't use the same sentences across languages—making it difficult to isolate linguistic effects from content differences.

Dataset Construction and Characteristics

The authors curated MAD-TSC using articles from Voxeurop, a platform featuring professional translations. The pipeline involved aligning sentences across eight European languages using multilingual-BERT and linking person entities using specialized libraries.

Word cloud of the MAD-TSC dataset, highlighting its focus on European political figures and institutions.

MAD-TSC contains 5,110 annotated entity mentions. Compared to previous datasets like NewsMTSC (which is US-centric), MAD-TSC offers a more diverse geographic spread and focuses heavily on European politics and international relations.

Key Insights: Complexity and Multilinguality

The study reveals a "Complexity Gap": performance declines as sentence length and the number of entities increase. Models struggle significantly when a sentence mentions three or more individuals with potentially conflicting sentiments.

A major finding is the utility of Machine Translation (MT). Models trained on MT data performed nearly as well as those trained on professional translations. Furthermore, the researchers identified an "English Pivot" strategy: translating non-English test examples into English and using a high-quality English-specific model (like RoBERTa) often yields better results than using a native-language model. This strategy effectively equalized performance across all eight languages at approximately 72-73% F1.

Experimental Results

The researchers evaluated four TSC methods: Sentence Pair Classification (SPC), Target-Dependent (TD) pooling, Prompt Modeling (PM), and a BASE control.

F1 scores for various TSC models (SPC, TD, PM, BASE) across the NewsMTSC (NM) and MAD-TSC (MAD) datasets.

As shown in the results, English and French models generally outperformed others. The "pivot" through English via DeepL provided a significant boost for lower-resource languages; for instance, Dutch performance jumped from 62.1% to 72.1% F1, highlighting the maturity of modern MT for sentiment tasks.

Limitations and Future Outlook

While MAD-TSC is a significant step forward, it is currently focused on European politics. Future work should expand to include business, sports, and culture to improve generalization. Additionally, developing "entity-aware" models that can better handle sentences with multiple targets and expanding the dataset to include non-European languages remain key priorities.

Summary updated

BibTeX
@inproceedings{dufraisse2023mad,
  author = {Evan Dufraisse and Adrian Popescu and Julien Tourille and Armelle Brun and Jérôme Deshayes},
  title = {{MAD-TSC: A Multilingual Aligned News Dataset for Target-dependent Sentiment Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.461},
}
Figure