Data Set for Stance and Sentiment Analysis from User Comments on Croatian News

Mihaela Bošnjak, Vanja Mladen KaranACL 2019

TL;DR

This paper introduces a novel dataset for analyzing user comments on Croatian news, focusing on claim verifiability, stance, and sentiment. They also present preliminary experiments using machine learning models to demonstrate the dataset's utility, highlighting the challenges and potential of automated analysis in this domain.

What Problem Does This Paper Address?

The internet is flooded with user-generated content, making it increasingly difficult to manually sift through opinions and reactions to news articles. Automated analysis of this unstructured data is challenging but crucial for understanding public sentiment and combating misinformation. Existing methods often struggle with the nuances of user comments, which can be informal, opinionated, and sometimes unverifiable. This paper aims to address this by creating a specialized dataset and annotation scheme for Croatian news comments, enabling more robust stance, sentiment, and claim verifiability analysis.

Methods

The authors developed a detailed annotation scheme with three main categories: Claim, Stance, and Sentiment.

  • Claim: This category focuses on whether a comment contains a verifiable claim. Labels include "Spam" (unrelated content), "Non-Claim" (questions, imperatives, etc.), "Verifiable" (objectively checkable claims), and "Non-Verifiable" (claims about the future, subjective descriptions, private facts).
  • Stance: This label, determined with respect to the article's title, captures whether a comment supports or disapproves of the topic. It's a multi-label task to account for conflicting stances within a single comment.
  • Sentiment: This refers to the commentator's manner of speaking, categorized as "Positive" or "Negative," with annotators instructed to disregard their own sentiment towards the topic.

The annotation process involved multiple annotators and several batches to calibrate and estimate inter-annotator agreement (IAA) using Cohen's kappa. They opted for comment-level annotations due to complexity and time constraints, though they acknowledge EDU-level annotations could offer more granularity.

For analysis, they experimented with several models:

  • A baseline linear SVM using TF-IDF weighted vectors of the title, body, and comment.
  • An enhanced SVM with additional features like word count and punctuation.
  • Deep learning models including Convolutional Neural Networks (CNNs), Gated Recurrent Units (GRUs), and Long Short-Term Memory (LSTM) networks, using word2vec embeddings trained on a Croatian corpus.

Datasets

The primary dataset introduced in this paper is a collection of user comments from the Croatian news outlet "24sata.hr." Comments were scraped from articles across three categories: newest, trending, and news. Articles with fewer than five comments were excluded, and a random subset of "root" comments (those not replies to other comments) was selected for annotation. The dataset comprises 54 articles and 904 comments, with an average of 16.74 comments per article. The authors make this dataset publicly available.

Key Insights and Ideas

  • The Nuance of User Comments: User comments are highly informal and spontaneous, presenting a significant challenge for structured analysis. The authors' annotation scheme attempts to capture this complexity by allowing for multi-label annotations in certain categories (like Stance).
  • Claim Verifiability is Crucial: The ability to automatically identify verifiable claims is vital for combating fake news. The dataset's focus on this aspect is a key contribution.
  • Title as Stance Target: Using the article title as the target for stance detection simplifies the annotation process and reduces subjectivity compared to using the entire article.
  • IAA Calibration is Essential: The iterative process of annotator calibration and IAA measurement was critical for ensuring data quality, especially given the challenging nature of the labels.
  • Data Imbalance is a Major Hurdle: The "Spam" class, in particular, was heavily underrepresented, posing a challenge for model training. The authors addressed this by artificially balancing the training set.

Key Results

The paper reports preliminary results on the test set using F1 scores. Performance across most labels was "rather low," indicating the complexity of the task.

  • Adding features to the baseline SVM model improved performance, with statistically significant gains for "Verifiable" and "Non-Verifiable" labels.
  • The SVM baseline performed best on the "Negative" sentiment label.
  • Deep learning models showed respectable results, particularly for the "Claim" group, though the authors note the small dataset size might limit their full potential.
  • The "Spam" class had the fewest positive examples (32).

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Small Dataset Size: The dataset is relatively small, limiting the scope of analysis and the potential for deep learning models. They envision annotating more data using the same methodology.
  • Comment-Level vs. EDU-Level Annotation: While comment-level annotation was chosen for practicality, sentence-level (EDU) annotations could enable deeper analysis.
  • Transfer Learning: Applying transfer learning could leverage relationships between labels and improve model performance.
  • Incorporating Thread Information: Including comments from the same thread and other metadata could enhance analysis.
  • Jointly Training Models: Exploring models that jointly train on multiple labels could reveal inter-label connections.

Summary updated

BibTeX
@inproceedings{bosnjak2019data,
  author = {Mihaela Bošnjak and Vanja Mladen Karan},
  title = {{Data Set for Stance and Sentiment Analysis from User Comments on Croatian News}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure