Subjectivity Theory vs. Speaker Intuitions: Explaining the Results of a Subjectivity Regressor Trained on Native Speaker Judgements

Elena Savinova, Jet HoekACL 2024

TL;DR

This paper investigates why a machine learning model trained on native English speakers' judgments of subjectivity makes the predictions it does. By combining theoretical linguistic features with data-driven explanations, the authors found that the model largely aligns with established subjectivity theories, but interestingly, it doesn't strongly distinguish between the author's own subjectivity and that of quoted sources.

What Problem Does This Paper Address?

Subjectivity detection, the task of identifying opinions, attitudes, and beliefs in text, is crucial for applications like fact-checking and sentiment analysis. While current state-of-the-art models often treat subjectivity as a binary classification, linguistic theory suggests it's a more nuanced, gradual concept. Furthermore, the "black box" nature of modern machine learning models, particularly transformers, makes it difficult to understand why they classify a sentence as subjective or objective. This paper tackles this explainability challenge by investigating a subjectivity regressor trained on human judgments, aiming to bridge the gap between linguistic theory and machine learning model behavior.

Methods

The researchers employed a two-pronged approach to explain their subjectivity regressor:

  • Top-Down Approach: They manually identified and annotated sentences for theoretically defined subjectivity features (e.g., emojis, evaluative adjectives, intensifiers) based on linguistic literature. A linear regression model was then used to assess the predictive power of these features on the regressor's subjectivity scores.
  • Bottom-Up Approach: They utilized the Local Interpretable Model-agnostic Explanations (LIME) method. LIME generates local explanations for individual predictions by creating perturbations of the input text and training a simple, interpretable model on these perturbations. This allowed them to identify words and phrases that were most influential for the regressor's subjectivity score on a sentence-by-sentence basis.

Datasets

The study used a dataset of sentences collected from British news outlets (BBC, Sky News, Daily Mail, Metro) covering the topics of "crime" and "Covid-19." This dataset comprised sentences from both Facebook news posts and articles from news websites. A subset of 398 sentences was manually annotated by 19 native English speakers on a 7-point subjectivity scale. The authors then trained a RoBERTa-base model on this annotated data.

Key Insights and Ideas

  • Theory Meets Data: The study demonstrates that a machine learning model trained on human intuition largely aligns with established linguistic theories of subjectivity. This suggests that our intuitive understanding of subjectivity is well-captured by these theoretical frameworks.
  • Emoji Power: Emojis emerged as particularly strong predictors of subjectivity, significantly influencing the model's scores. This highlights the increasing importance of non-textual elements in conveying subjective meaning in modern communication.
  • Author vs. Source Ambiguity: A surprising finding was that the model, and by extension, native speakers, did not strongly differentiate between the subjectivity expressed by the author of a text and the subjectivity present in quoted sources. This challenges a common distinction made in linguistic theory.
  • Context Matters: The importance of certain features, like modal adverbials, seemed to depend on their context. For instance, adverbials like "allegedly" or "reportedly" were associated with lower subjectivity scores when used to indicate common evidence, thus supporting objectivity.

Key Results

  • The top-down analysis revealed that features like emojis, first and second-person pronouns, exclamations, questions, and intensifiers were significant predictors of subjectivity scores.
  • LIME analysis, after filtering for feature frequency, also identified a strong presence of theoretically defined subjectivity features among the top subjective features. Notably, evaluative adjectives, emojis, modal verbs, and intensifiers were prominent.
  • The linear regression model explained 40% of the variance in subjectivity scores, indicating that while the identified features are important, they don't fully capture all aspects of subjectivity.
  • A generalized additive model (GAM) showed that increasing the number of subjective elements in a sentence positively impacts its subjectivity score, but this effect plateaus after a certain point.

Limitations and Future Work

The authors acknowledge several limitations:

  • Feature Exhaustiveness: The list of subjectivity features used in the top-down approach was not exhaustive, excluding potentially important features like affective nouns and verbs.
  • Density Measurement: The analysis of subjectivity feature density was a coarse measure, and the interplay between different types of features might be more complex.
  • Explicit Markers: Both manual feature selection and LIME are limited to explicit subjective markers, potentially missing implicit subjectivity.
  • Author vs. Source Distinction: The study's annotation process did not explicitly instruct raters on distinguishing author vs. source subjectivity, which may have influenced the results.

Future work could explore more comprehensive feature sets, refine the density analysis, and more deeply investigate how native speakers perceive the distinction between author and source subjectivity, potentially within broader contextual analyses.

Summary updated

BibTeX
@inproceedings{savinova2024subjectivity,
  author = {Elena Savinova and Jet Hoek},
  title = {{Subjectivity Theory vs. Speaker Intuitions: Explaining the Results of a Subjectivity Regressor Trained on Native Speaker Judgements}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure