One_by_zero@DravidianLangTech 2025: Fake News Detection in Malayalam Language Leveraging Transformer-based Approach

Dola Chakraborty, Shamima Afroz, Jawad Hossain, Mohammed Moshiul HoqueNAACL 2025

TL;DR

This paper tackles the challenge of fake news detection in Malayalam, a low-resource language with limited datasets and pre-trained models. The authors propose and evaluate a transformer-based approach, demonstrating that specialized models like Malayalam BERT significantly outperform traditional machine learning and general deep learning methods, achieving a strong F1-score of 0.892.

What Problem Does This Paper Address?

The proliferation of misinformation online poses a significant threat, and detecting fake news is crucial. However, most research focuses on high-resource languages like English, leaving low-resource languages such as Malayalam underserved. Malayalam presents unique challenges, including dialectal variations, complex morphology, and a scarcity of annotated datasets and pre-trained language models. This gap makes it difficult to build effective fake news detection systems for its speakers, despite the potential for significant societal and political repercussions. This paper aims to bridge this gap by developing and evaluating robust fake news detection models specifically for Malayalam social media content.

Methods

The researchers explored a range of machine learning (ML), deep learning (DL), and transformer-based approaches.

  • ML Models: Logistic Regression, Decision Trees, and Support Vector Machines (SVM) were tested, using TF-IDF for feature extraction.
  • DL Models: BiLSTM, LSTM + BiLSTM, and CNN + BiLSTM architectures were employed, utilizing embedding layers for text processing.
  • Transformer Models: The core of their approach involved fine-tuning pre-trained transformer models. These included:
  • MURIL: A multilingual model trained on 17 Indian languages and English.
  • IndicBERT: Pre-trained on 12 Indic languages and English.
  • XLM-ROBERTa: A robust multilingual model.
  • Malayalam-BERT: Specifically pre-trained on Malayalam text, making it a strong candidate for this task.
  • Malayalam Sentence-BERT: Fine-tuned for sentence-pair tasks.

The methodology involved a comprehensive data preprocessing pipeline to ensure data consistency and clarity, including punctuation removal, stopword filtering, and transliteration of code-mixed text into Malayalam. Feature extraction used TF-IDF, GloVe, and Keras embeddings.

Datasets

The study utilized the dataset provided for the DravidianLangTech@NAACL 2025 shared task on fake news detection in Malayalam. This dataset is described as fairly balanced, with:

  • Training Set: 1,658 original and 1,599 fake samples.
  • Validation Set: 409 original and 406 fake samples.
  • Test Set: 512 original and 507 fake samples.

The dataset consists of social media texts and YouTube comments.

Key Insights and Ideas

  • Specialization Matters: The most significant insight is that transformer models specifically pre-trained on Malayalam (like Malayalam BERT) significantly outperform general-purpose multilingual models and traditional ML/DL approaches for this task.
  • Transformer Dominance: Transformer-based models, in general, showed superior performance compared to both ML and DL models, highlighting their effectiveness in capturing complex linguistic nuances.
  • Low-Resource Challenges: The paper underscores the inherent difficulties in low-resource language NLP, emphasizing the need for tailored models and datasets.
  • Code-Mixing is Common: The preprocessing step highlights the presence of code-mixed text (Malayalam and English) in social media, necessitating robust handling mechanisms like transliteration.

Key Results

The paper reports the performance of various models on the test dataset using the macro F1-score.

  • Malayalam BERT achieved the best performance with a macro F1-score of 0.892, securing 3rd place in the competition.
  • Among DL models, BiLSTM achieved the highest score of 0.782.
  • The best-performing ML model was Logistic Regression (LR) with a score of 0.5267.
  • Quantitative analysis showed that the best model (Malayalam BERT) correctly predicted 889 out of 1,019 samples, misclassifying 93 original samples as fake and 37 fake samples as original.
  • Qualitative analysis revealed that while the model performed well, misclassifications often occurred due to the linguistic complexity of Malayalam and the subtle imitation of genuine news styles by fake news.

Limitations and Future Work

The authors acknowledge several limitations:

  • Token Limitation: Malayalam BERT has a token limitation, which can lead to truncation of longer news articles, potentially losing crucial information.
  • Text-Only Analysis: The current approach only analyzes text, making it ineffective against misinformation spread through images, memes, or misleading visuals.
  • Dataset Size: The relatively small dataset size might limit the model's ability to generalize well to diverse fake news patterns.

Future work could involve:

  • Utilizing larger datasets.
  • Leveraging ensemble transformer models.
  • Exploring other advanced large language models.
  • Investigating multimodal approaches to incorporate images and other media.

Summary updated

BibTeX
@inproceedings{chakraborty2025one,
  author = {Dola Chakraborty and Shamima Afroz and Jawad Hossain and Mohammed Moshiul Hoque},
  title = {{One\_by\_zero@DravidianLangTech 2025: Fake News Detection in Malayalam Language Leveraging Transformer-based Approach}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure