The ParlaSent Multilingual Training Dataset for Sentiment Identification in Parliamentary Proceedings

Michal Mochtak, Peter Rupnik, Nikola LjubešićCOLING 2024

TL;DR

This paper introduces ParlaSent, a novel multilingual dataset for sentiment analysis in parliamentary proceedings across seven languages. They also present XLM-R-parla, a transformer model specifically pre-trained on parliamentary data, demonstrating significant improvements in sentiment identification accuracy, especially for low-resource languages.

What Problem Does This Paper Address?

Understanding political sentiment is crucial for analyzing democratic processes, but existing sentiment analysis tools often struggle with political discourse, particularly in languages other than English. This is due to a lack of domain-specific datasets and models, leading researchers to rely on less accurate, dictionary-based methods. The paper aims to bridge this gap by creating a comprehensive dataset and a specialized language model to enable robust, multilingual sentiment analysis of parliamentary proceedings.

Methods

The authors developed a sentence-level sentiment analysis approach, moving away from longer text units to capture finer-grained sentiment. They employed a stratified random sampling strategy using sentiment lexicons to create manually annotated datasets. For their model, they leveraged the XLM-R-large architecture and further pre-trained it on a massive corpus of parliamentary proceedings from 27 European parliaments, creating the XLM-R-parla model. This pre-training is a key innovation, adapting a general-purpose multilingual model to the specific linguistic nuances of political discourse.

Datasets

The core contribution is the ParlaSent dataset, which includes manually annotated sentences from parliamentary proceedings in seven languages: Bosnian, Croatian, Czech, English, Serbian, Slovak, and Slovenian. This dataset is split into five training sets (one for each language group: BCS, CZ, EN, SK, SL) and two test sets (BCS-test and EN-test). The authors also utilized a large, unlabeled corpus of parliamentary proceedings from 27 European parliaments for the additional pre-training of their XLM-R-parla model.

Key Insights and Ideas

  • Domain-Specific Pre-training Matters: Adapting general-purpose multilingual models (like XLM-R) with domain-specific data (parliamentary proceedings) significantly boosts performance on downstream tasks like sentiment analysis.
  • Multilingual Models are Robust: The XLM-R-parla model performs well even on languages it wasn't explicitly fine-tuned on, highlighting the power of cross-lingual transfer learning.
  • More Data is Better (Even if Different): Training a multilingual model on data from multiple parliaments, even if they are linguistically diverse, yields better results than training a monolingual model on a smaller, single-parliament dataset. This suggests that the annotation guidelines were effective in creating comparable sentiment labels across different political contexts.
  • Sentence-Level Analysis is Effective: Focusing on individual sentences allows for a more granular understanding of sentiment, which can be aggregated to higher levels, offering a more nuanced view than analyzing longer text segments.

Key Results

  • The XLM-R-parla model, after additional pre-training on parliamentary data, achieved improved sentiment identification performance compared to vanilla XLM-R-base and XLM-R-large models. For example, on the BCS test set, XLM-R-parla achieved an R² score of 0.672, compared to 0.653 for XLM-R-large.
  • The model showed strong performance on unseen languages, with only a minor drop in accuracy when a language was excluded from the training data.
  • Multilingual training consistently outperformed monolingual training across most tested languages, demonstrating the benefit of leveraging data from multiple parliaments. For instance, in the BCS dataset, multilingual training yielded an R² of 0.737, while monolingual training achieved 0.699.

Limitations and Future Work

  • Linguistic Relatedness: The authors note that the languages and parliaments tested for robustness were linguistically and traditionally related. Caution is advised when applying the model to more distant languages or parliaments.
  • Inter-Annotator Agreement: While considered reasonably high for this endeavor, the inter-annotator agreement (Krippendorff's alpha) was not exceptionally high, suggesting the subjective nature of sentiment perception and potential for "hard calls" during annotation.
  • Future Work: The paper suggests testing the XLM-R-parla model's general potency for processing political texts beyond sentiment analysis. Further research could explore its application to a wider range of political texts and languages, especially those that are more distant from the current training data.

Summary updated

BibTeX
@inproceedings{mochtak2024parlasent,
  author = {Michal Mochtak and Peter Rupnik and Nikola Ljubešić},
  title = {{The ParlaSent Multilingual Training Dataset for Sentiment Identification in Parliamentary Proceedings}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure