The Sally Smedley Hyperpartisan News Detector at SemEval-2019 Task 4

Kazuaki Hanawa, Shota Sasaki, Hiroki Ouchi, Jun Suzuki, Kentaro Inui

Abstract

Identifying hyperpartisan news—content characterized by extreme bias—is a significant challenge in modern natural language processing. This paper presents the "Sally Smedley" system, which achieved 3rd place in the SemEval-2019 Task 4 competition. The system utilizes a hybrid linear classifier that integrates contextual BERT embeddings with statistical features, including article length and informative phrase extraction. By prioritizing high-quality training data and employing a massive 400-model ensemble, the system achieved an official accuracy of 80.9% on the balanced test set.

Introduction

In the current media landscape, the distinction between objective reporting and hyperpartisan advocacy is increasingly blurred. Unlike "fake news" detection, which focuses on factual verification, hyperpartisan news detection aims to identify the style and bias of argumentation. The SemEval-2019 Task 4 challenge tasked participants with classifying articles as hyperpartisan or neutral. A primary difficulty of this task lies in the data distribution: a small, human-labeled by-article dataset (645 articles) and a massive, noisy by-publisher dataset (750,000 articles) labeled by source reputation.

Methodology

The Sally Smedley system employs a linear classifier based on a concatenated feature vector $\mathbf{f}$, consisting of three primary components:

  1. BERT Features ($f_1$): Rather than fine-tuning the entire BERT model, the authors adopted a feature-based approach. They extracted and averaged representations from the top four hidden layers of a pre-trained BERT-Base model. This strategy allowed for efficient reuse of features across multiple models.
  2. Article Length Features ($f_2$): To capture structural differences between biased and neutral content, the system uses a one-hot vector representing article length in 100-word bins.
  3. Informative Phrase Features ($f_3$): Using $\chi^2$ statistics on the noisy by-publisher dataset, the authors identified N-grams strongly associated with hyperpartisanship. After filtering out publisher-specific terms, these phrases were represented using GloVe embeddings.

To enhance robustness, the system utilized word dropout for regularization, oversampling for class balance, and a massive ensemble of 400 models trained with varying seeds and epochs.

Datasets

The study utilized two primary datasets:

  • by-article: A high-quality set of 645 articles used for training the final classifiers.
  • by-publisher: A large-scale collection of 750,000 articles. While too noisy for direct training, it served as a valuable resource for mining informative phrases ($f_3$).

Results and Discussion

The Sally Smedley system achieved an official accuracy of 80.9%, ranking 3rd among 42 competing teams.

Ablation Study

An ablation study conducted via cross-validation demonstrated that each component of the system contributed to its overall performance. As shown in Figure 1, the baseline BERT features ($f_1$) achieved 76.0% accuracy. The addition of length ($f_2$) and phrase ($f_3$) features increased accuracy to 77.7%, while the final ensemble approach pushed performance to 78.8%.

Ablation study showing the impact of feature components and ensembling on accuracy.

BERT Integration Strategy

The authors also investigated different architectures for utilizing BERT embeddings. Interestingly, a simple averaging of the top layers outperformed more complex sequential models like BiLSTMs or CNNs placed on top of the embeddings.

Comparison of different architectures for processing BERT embeddings.

Conclusion

The success of the Sally Smedley system underscores several key insights. First, training on a small, clean dataset is often more effective than using a massive but noisy one. Second, noisy datasets can still provide value through statistical feature mining. Third, simple features like article length can serve as effective proxies for intent. Finally, large-scale ensembling is critical for achieving stable performance when training on limited data. Future work will explore advanced denoising techniques to better leverage large-scale noisy datasets.

Summary updated

Abstract

This paper describes our system submitted to the formal run of SemEval-2019 Task 4: Hyperpartisan news detection. Our system is based on a linear classifier using several features, i.e., 1) embedding features based on the pre-trained BERT embeddings, 2) article length features, and 3) embedding features of informative phrases extracted from by-publisher dataset. Our system achieved 80.9% accuracy on the test set for the formal run and got the 3rd place out of 42 teams.
BibTeX
@inproceedings{hanawa2019sally,
  author = {Kazuaki Hanawa and Shota Sasaki and Hiroki Ouchi and Jun Suzuki and Kentaro Inui},
  title = {{The Sally Smedley Hyperpartisan News Detector at SemEval-2019 Task 4}},
  booktitle = {SemEval @ NAACL},
  year = {2019},
  doi = {10.18653/v1/s19-2185},
}
Figure