Detecting Hyperpartisanship and Rhetorical Bias in Climate Journalism: A Sentence-Level Italian Dataset

M. Maggini, Davide Bassi, Pablo Gamallo2025 · DOI

Abstract: We present the first Italian dataset for joint hyperpartisan and rhetorical bias detection in climate change discourse, enhancing the complexity in modeling hyperpartisan detection. Our annotation scheme achieves a Cohen's kappa agreement of 0.63 on the gold test set (173 sentences). The dataset comprises 48 articles (1,010 sentences) from far-right media, annotated at sentence level for both binary hyperpartisan classification and the multi-label classification of 17 rhetorical biases. We conduct extensive text analysis revealing significant correlations between hyperpartisan content and specific rhetorical techniques. Our experiments with state-of-the-art language models (GPT-4o-mini) and Italian BERTbase models establish strong baselines for both classification tasks. To ensure reproducibility while addressing copyright concerns, we release article URLs, article id and paragraph's number alongside comprehensive annotation guidelines. This resource advances research in crosslingual hyperpartisan detection and provides insights into the rhetorical strategies employed in Italian climate change discourse. To the best of our knowledge, we are the first to tackle hyperpartisan detection related to logical fallacies, focusing on the sentence level.

Introduction

Most automated systems for detecting media bias treat "hyperpartisanship"—extreme, one-sided messaging—as a simple binary classification task. However, this "black box" approach fails to uncover the actual rhetorical mechanisms used to manipulate an audience. While English-language resources for detecting propaganda are relatively abundant, other languages like Italian are significantly underserved. This research addresses the need for granular, explainable AI tools that can identify how a text persuades or misleads, specifically within the polarizing context of climate change discourse.

Methodology and Dataset

The study introduces a novel Italian dataset consisting of 48 articles (1,010 sentences) sourced from NicolaPorro.it, an independent libertarian-right media outlet. The focus remains on climate change, green policies, and Euroscepticism.

Researchers adapted a comprehensive taxonomy of 23 rhetorical biases down to 17 categories (e.g., Loaded Language, Reductio ad Hitlerum, Appeal to Fear). Two native Italian Ph.D. experts annotated the text, achieving high agreement (Krippendorf’s $\alpha$ of 0.92) for hyperpartisan detection and moderate agreement (0.63) for the more subjective task of identifying specific rhetorical fallacies.

Topological Analysis of Bias

A key finding of this research is that hyperpartisan content is not spread uniformly across a text. Instead, it follows a strategic "topological" pattern. Bias tends to peak in the first 10% of an article (titles and introductions) and the final 10% (conclusions), with a noticeable "dip" in the middle where more factual reporting typically resides.

Average Hyperpartisan Score by Article Position

Rhetorical Strategy and Positioning

Different rhetorical techniques serve distinct functional roles based on their placement within the article. For instance, Name Calling and Reductio ad Hitlerum (inappropriate comparisons to Nazism/fascism) are frequently used in the first quartile (Q1) to set a biased tone immediately. Conversely, Slogans peak in the final section (Q4) to leave a lasting, emotional impression on the reader.

Distribution of Rhetorical Techniques across Article Quartiles

Certain techniques act as "fingerprints" for hyperpartisanship. Name Calling, Loaded Language, and Reductio ad Hitlerum showed the strongest correlations with hyperpartisan labels, suggesting these are the primary tools for ideological signaling in this dataset.

Experimental Results

The researchers established baselines using Italian-specific BERT models and Large Language Models (LLMs) like GPT-4o.

  • Hyperpartisan Detection (Binary): GPT-4o achieved an impressive F1 score of 0.942 and an accuracy of 96.9%, outperforming specialized Italian BERT models (F1 ~0.85).
  • Rhetorical Bias Identification (Multi-label): This task proved significantly more challenging. Even fine-tuned GPT-4o models only reached an F1 score of 0.410. This difficulty is largely attributed to "class imbalance," where common techniques like Loaded Language are overrepresented compared to rare strategies like Appeal to Popularity.

Conclusion and Future Work

While the current dataset provides a vital foundation for Italian bias detection, its size (48 articles) and focus on far-right media are acknowledged limitations. Future work aims to expand the corpus to include centrist and left-leaning outlets to capture the full spectrum of climate discourse. Additionally, the team plans to explore "span detection" to identify the exact words or phrases that constitute a bias, moving beyond sentence-level labeling for even greater granularity.

Summary updated

Abstract

We present the first Italian dataset for joint hyperpartisan and rhetorical bias detection in climate change discourse, enhancing the complexity in modeling hyperpartisan detection.Our annotation scheme achieves a Cohen's kappa agreement of 0.63 on the gold test set (173 sentences).The dataset comprises 48 articles (1,010 sentences) from far-right media, annotated at sentence level for both binary hyperpartisan classification and the multi-label classification of 17 rhetorical biases.We conduct extensive text analysis revealing significant correlations between hyperpartisan content and specific rhetorical techniques.Our experiments with state-of-the-art language models (GPT-4o-mini) and Italian BERTbase models establish strong baselines for both classification tasks.To ensure reproducibility while addressing copyright concerns, we release article URLs, article id and paragraph's number alongside comprehensive annotation guidelines.This resource advances research in crosslingual hyperpartisan detection and provides insights into the rhetorical strategies employed in Italian climate change discourse.To the best of our knowledge, we are the first to tackle hyperpartisan detection related to logical fallacies, focusing on on the sentence level.
BibTeX
@misc{maggini2025detecting,
  author = {M. Maggini and Davide Bassi and Pablo Gamallo},
  title = {{Detecting Hyperpartisanship and Rhetorical Bias in Climate Journalism: A Sentence-Level Italian Dataset}},
  year = {2025},
  doi = {10.18653/v1/2025.climatenlp-1.11},
}
Figure