Towards Detecting Political Bias in Hindi News Articles

Samyak Agrawal, Kshitij Gupta, Devansh Gautam, Radhika MamidiACL 2022

TL;DR

This paper tackles the challenge of automatically detecting political bias in Hindi news articles, a crucial task given the amplification of misinformation through biased reporting. The authors introduce a novel annotated dataset of Hindi news articles and demonstrate that multilingual transformer models, particularly XLM-RoBERTa, achieve the best performance in identifying bias.

What Problem Does This Paper Address?

In today's digital age, news media plays a significant role in shaping public opinion. However, biased reporting, where news outlets intentionally or unintentionally present information in a way that favors a particular political party or ideology, is a widespread phenomenon. This biased reporting can create untruthful narratives and misinform the public, especially concerning political events and policies. While manual bias detection is possible, it's a tedious and time-consuming process. Therefore, automating the detection of political bias in news articles is essential for verifying information and promoting a more objective media landscape. This paper focuses specifically on the Hindi language, which has a vast number of speakers, yet lacks readily available datasets for political bias detection.

Methods

The core of this research involves leveraging transfer learning with transformer-based models. The authors fine-tune pre-trained language models on their newly created dataset. They experiment with several baseline approaches, including traditional machine learning models like Support Vector Machines (SVM) and Random Forests, as well as deep learning models. The key deep learning models explored are mBERT (a multilingual BERT model), XLM-RoBERTa (a multilingual RoBERTa model), and a Hindi-specific version of XLM-RoBERTa, and IndicBERT. For classification, they use a standard approach of attaching a feed-forward network to the [CLS] token embedding of the transformer output. The training utilizes the Adam optimizer with a learning rate of 1e-5 and a batch size of 16, employing the standard cross-entropy loss.

Datasets

The paper introduces a novel dataset specifically for detecting political bias in Hindi news articles. This dataset comprises 1388 Hindi news articles and their corresponding headlines. The articles were collected from four different Indian news websites: The Wire and The Quint (known for being critical of the current government and more liberal) and OPIndia and The Frustrated Indian (known for their pro-BJP articles and right-wing leanings). This diverse selection was made to ensure a balance of articles biased for and against the Bhartiya Janta Party (BJP), the current ruling party in India. The articles were annotated into three categories: biased towards the BJP, biased against the BJP, or neutral. The authors also mention using existing datasets like the Ideological Books Corpus (IBC) and ontheissues (OTI) in related work, but their primary contribution is the new Hindi dataset.

Key Insights and Ideas

  • The Neutrality Challenge: The research highlights that detecting neutrality in news articles is significantly more challenging than identifying bias. This is attributed to the subjective nature of neutrality, where an article might appear biased to one person but unbiased to another.
  • Multilingual Models Shine: Multilingual transformer models, particularly XLM-RoBERTa, demonstrate superior performance compared to traditional machine learning methods and even monolingual models. This suggests that the vast pre-training on diverse languages equips these models with a robust understanding of linguistic nuances applicable to bias detection.
  • Headline Importance: The paper implicitly acknowledges the importance of headlines, as they are provided alongside the articles for annotation and analysis. Headlines often contain strong biases and are the first point of contact for readers.
  • Dataset Diversity is Key: The deliberate selection of news sources with opposing political leanings was crucial for creating a balanced dataset, enabling models to learn to distinguish between genuine bias and objective reporting.

Key Results

The paper reports that the XLM-RoBERTa model achieved the highest performance on their dataset, obtaining an accuracy of 83%, an F1-macro score of 76.4%, and a Matthews Correlation Coefficient (MCC) of 72.1%. While traditional machine learning models like SVM and Random Forest also showed decent performance (around 78.7% accuracy), the deep learning approaches, especially XLM-RoBERTa, significantly outperformed them. The confusion matrix analysis revealed that the models struggled most with correctly classifying neutral articles, a finding consistent with the annotator observations.

Limitations and Future Work

The authors acknowledge that their system is trained on articles from a limited number of sources. This means its performance might not generalize well to news articles from other outlets. They also caution that their system should be used as a tool to filter potentially biased articles rather than a definitive judgment of bias, as incorrect predictions could lead to accusations against media houses. For future work, the authors aim to expand their approach to detect bias towards any political party, moving beyond a single-party focus, and to develop a more general approach for political bias detection.

Summary updated

BibTeX
@inproceedings{agrawal2022towards,
  author = {Samyak Agrawal and Kshitij Gupta and Devansh Gautam and Radhika Mamidi},
  title = {{Towards Detecting Political Bias in Hindi News Articles}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure