Analyzing Subjectivity Using a Transformer-Based Regressor Trained on Naïve Speakers’ Judgements

Elena Savinova, Fermin Moscoso Del PradoACL 2023

TL;DR

This paper tackles the challenge of subjectivity detection by framing it as a regression task, moving beyond traditional binary classification. They demonstrate that a transformer-based model (RoBERTa) trained on a small set of human-annotated data can effectively capture nuanced subjectivity, outperforming existing rule-based methods and even state-of-the-art classifiers on a widely-used benchmark dataset. This work highlights the limitations of current benchmark datasets and emphasizes the importance of human judgment in defining and detecting subjectivity.

What Problem Does This Paper Address?

Subjectivity detection, the task of identifying opinions, attitudes, and private states in text, is often treated as a binary classification problem, primarily as a precursor to sentiment analysis. However, subjectivity is inherently a matter of degree, not a simple on/off switch. Existing methods, while advanced, often rely on large, automatically-labeled datasets that may not accurately reflect human perception of subjectivity. This paper addresses this gap by proposing a regression-based approach to subjectivity detection, aiming to capture the gradual nature of subjectivity and to train a model that aligns better with human intuition, using a smaller, human-annotated dataset.

Methods

The core of this paper's methodology lies in leveraging a pre-trained transformer model, specifically RoBERTa-base, and fine-tuning it for subjectivity detection. Instead of a binary classification, they treat subjectivity as a continuous variable and train the model to predict a score on a 7-point scale, later normalized to a [0-1] scale. This fine-tuning is performed using a semi-supervised approach on a custom dataset of news articles and social media posts. They also compare their model's performance against a rule-based subjectivity regressor (TextBlob) and state-of-the-art binary classifiers.

Datasets

  • Custom Dataset: The authors created a dataset consisting of 7,751 sentences from Facebook news posts and online news articles from four major UK news sources, covering "crime" and "Covid-19" topics. A subset of 400 sentences was human-annotated by 20 native English speakers on a 7-point scale for subjectivity.
  • Benchmark Dataset (SUBJ): The paper also evaluates their model on the widely-used SUBJ dataset, which contains 10,000 movie review snippets (labeled subjective) and plot summaries (labeled objective).

Key Insights and Ideas

  • Subjectivity as a Spectrum: The paper strongly advocates for viewing subjectivity as a continuous spectrum rather than a binary choice, arguing that this better reflects human perception and leads to more nuanced analysis.
  • Human Annotation is Key: The research highlights the limitations of automatically generated "gold standard" datasets. Their findings suggest that such datasets might be learning proxies for subjectivity (like text source) rather than actual subjective expression as perceived by humans.
  • Transformer Models Excel with Nuance: RoBERTa, when fine-tuned on human-annotated data, demonstrates a remarkable ability to capture the subtle degrees of subjectivity, outperforming established methods.
  • The "Gold Standard" is Flawed: The paper provides compelling evidence that the SUBJ dataset, despite its widespread use, contains significant mislabeling, leading state-of-the-art classifiers trained on it to perform poorly on data that genuinely reflects human subjectivity.

Key Results

  • Regressor Performance: The RoBERTa-based regressor achieved a high correlation (r=.79) with average human ratings on their test set, explaining over 62% of the variance. This performance surpassed the correlation of individual human raters with the average.
  • Outperforming Baselines: The model significantly outperformed the rule-based TextBlob regressor (r=.28 vs. r=.79).
  • Binary Classifier Performance: When used as a binary classifier, the model achieved an accuracy of 92% and an F1 score of .80 on their test set using an optimized threshold.
  • Benchmark Dataset Performance: On the SUBJ dataset, the model achieved an accuracy of 78.2% and an F1 score of .79, demonstrating good generalizability.
  • State-of-the-Art Classifier Failure: Crucially, state-of-the-art classifiers trained on the SUBJ dataset performed poorly on the authors' human-labeled data, indicating they learned dataset-specific biases rather than true subjectivity.

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that their model's performance on the SUBJ dataset, while good, is lower than some reported state-of-the-art results. They attribute this to the likely mislabeling within the SUBJ dataset itself.
  • The study notes that there might be different ways of understanding subjectivity among English speakers, suggesting potential for further research into these variations.
  • Future Work:
    • Further analysis of state-of-the-art classifiers trained on benchmark datasets to understand what they are truly learning.
  • Comparing systems trained on theoretically instructed datasets versus those trained on naïve human judgments.
  • Investigating the origins of differences in subjectivity perception across native speakers.

Summary updated

BibTeX
@inproceedings{savinova2023analyzing,
  author = {Elena Savinova and Fermin Moscoso Del Prado},
  title = {{Analyzing Subjectivity Using a Transformer-Based Regressor Trained on Naïve Speakers’ Judgements}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure