Never guess what I heard... Rumor Detection in Finnish News: a Dataset and a Baseline

Mika Hämäläinen, Khalid Alnajjar, Niko Partanen, Jack RueterNAACL 2021

TL;DR

This paper introduces the first dataset and baseline for rumor detection in Finnish news headlines, a crucial task for combating misinformation. They evaluate several models, finding that while a fine-tuned FinBERT achieves high overall accuracy, a multilingual BERT model excels at factual detection, highlighting the impact of training data bias.

What Problem Does This Paper Address?

In today's digital age, distinguishing between credible news and misinformation is increasingly challenging. Fake news and rumors can have significant societal impacts, influencing public opinion and decision-making. While research on fake news detection has flourished in English, other languages, including Finnish, have been largely overlooked. This paper addresses this gap by creating a new dataset and establishing baseline models for rumor detection specifically in Finnish news headlines.

Methods

The authors experimented with four different models for rumor detection:

  • LSTM-based models: They trained two Long Short-Term Memory (LSTM) models. One used standard word embeddings, while the other leveraged pre-trained Finnish word2vec embeddings.
  • BERT-based models: They also fine-tuned two BERT models: a Finnish-specific BERT (FinBERT) and a multilingual BERT.

These models were trained to classify news headlines into either "rumorous" or "factual" categories. The BERT models were fine-tuned using the transformers package with standard hyperparameters.

Datasets

The core contribution is the introduction of a new dataset for Finnish rumor detection. This dataset was created by crawling headlines from a Finnish news aggregation website, specifically collecting articles from the "rumor" category and the "news from Finland" category to represent factual news.

  • Total Samples: 3896 headlines
  • Factual Headlines: 2385
  • Rumorous Headlines: 1511

The data was split into training (70%), validation (15%), and testing (15%) sets. The dataset has been made publicly available on Zenodo.

Key Insights and Ideas

  • Training Data Matters Immensely: The performance differences between FinBERT and Multilingual BERT were strongly linked to their training data. FinBERT, trained on data from a Finnish forum notorious for misinformation, excelled at detecting rumors. Multilingual BERT, trained on more general data like Wikipedia, performed better on factual headlines.
  • Pre-trained Embeddings Aren't Always Better: Surprisingly, the LSTM model trained without pre-trained word2vec embeddings outperformed the one that used them. This suggests that the Finnish word2vec embeddings might not have sufficient overlap in vocabulary or syntactic usage with the target dataset, or that the English-centric nature of many pre-trained models (even those claiming multilingualism) can be a hindrance for languages like Finnish.
  • Finnish Language Nuances: The paper highlights that for languages like Finnish, simply having a large vocabulary in pre-trained models isn't enough. Overlap in syntactic positions and lemmatization are crucial, and current pre-trained models may not adequately capture these for effective transfer learning.
  • A Starting Point, Not an Endpoint: The authors position their work as a baseline, acknowledging that robust rumor detection likely requires incorporating external real-world knowledge beyond just linguistic features of headlines.

Key Results

  • FinBERT: Achieved the best overall accuracy (94.3%) and rumor label accuracy (96.0%).
  • Multilingual BERT: Achieved the best factual label accuracy (97.2%).
  • LSTM (without word2vec): Outperformed the LSTM with word2vec embeddings.
  • Performance Variance: Significant accuracy differences (tens of percentage points) were observed between the different models.

Limitations and Future Work

  • Dataset Size and Annotation: The authors acknowledge that larger, manually annotated datasets with more variation are needed for more robust solutions. They suggest that automatic generation of such datasets using Finnish semantic databases and generation methods could be a future direction.
  • Qualitative Analysis: Further qualitative analysis is needed to understand why models make certain classifications and to identify specific error types.
  • Real-World Knowledge Integration: The current models rely solely on linguistic features of headlines. Future work should explore integrating real-world knowledge and context to build more effective rumor detection systems.
  • Broader Genre Coverage: The current dataset focuses on news headlines. Future work should incorporate a wider range of text types and genres to improve model robustness.
  • Pre-trained Model Quality: The paper strongly suggests that the quality and suitability of existing pre-trained models for Finnish need improvement, emphasizing the need for more academic interest in producing high-quality, domain-specific models.

Summary updated

BibTeX
@inproceedings{hamalainen2021never,
  author = {Mika Hämäläinen and Khalid Alnajjar and Niko Partanen and Jack Rueter},
  title = {{Never guess what I heard... Rumor Detection in Finnish News: a Dataset and a Baseline}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure