NLP_goats@DravidianLangTech 2025: Detecting Fake News in Dravidian Languages: A Text Classification Approach

Srihari V K, Vijay Karthick Vaidyanathan, Thenmozhi DurairajNAACL 2025

TL;DR

This paper tackles the critical challenge of detecting fake news in Dravidian languages, specifically Malayalam and English, by adapting transformer-based models like mBERT. The authors achieved strong results on two tasks: binary classification of social media posts and multi-class categorization of news articles, highlighting the effectiveness of advanced language models for low-resource languages.

What Problem Does This Paper Address?

The rapid growth of social media has unfortunately facilitated the widespread dissemination of fake news, which can significantly influence public opinion and decision-making. While fake news detection has seen progress in well-resourced languages, underrepresented languages like those in the Dravidian family (Malayalam, Tamil, Telugu) face a significant hurdle due to a lack of computational resources and annotated datasets. This paper addresses this gap by participating in the DravidianLangTech@NAACL 2025 shared task, aiming to develop effective methods for detecting fake news in these languages.

Methods

The core of this study involves leveraging pre-trained transformer models, specifically mBERT (multilingual BERT), for text classification. mBERT, trained on a corpus encompassing 104 languages, is well-suited for cross-lingual tasks and capturing contextual nuances. The authors also employed traditional classifiers like Naive Bayes as a baseline for comparison.

The methodology involved:

  • Data Preprocessing: Cleaning text data, handling potential class imbalances through oversampling (for Task 2), and encoding labels into a binary format.
  • Tokenization: Using HuggingFace's AutoTokenizer with the bert-base-multilingual-cased model, with padding and truncation to 128 tokens.
  • Model Training: Fine-tuning the mBERT model on the prepared datasets for both tasks.
  • Evaluation: Assessing model performance using metrics like accuracy, precision, recall, and F1 score.

Datasets

The study utilized datasets provided for the DravidianLangTech@NAACL 2025 shared task:

  • Task 1 Dataset: This dataset, derived from previous work on Dravidian languages, consists of English social media posts (from Twitter, Facebook, and YouTube) labeled as either "fake" or "original."
  • Task 2 Dataset: This dataset focuses on Malayalam news articles and categorizes them into four truthfulness levels: "False," "Half True," "Mostly False," and "Partly False." This dataset is based on prior studies.

Key Insights and Ideas

  • Transformer Models Excel in Low-Resource Settings: The study strongly suggests that advanced transformer-based models like mBERT can effectively handle the complexities of underrepresented languages, outperforming traditional methods.
  • Context is King: mBERT's ability to capture long-range linguistic structures and contextual nuances proved crucial for accurate fake news detection, especially in morphologically rich languages like Malayalam.
  • Language-Specific Nuances Matter: While multilingual models are powerful, the paper hints that fine-tuning and potentially domain-specific pre-training are essential for optimal performance in specific language contexts.
  • Data Imbalance is a Real Challenge: The significant imbalance in the Task 2 dataset necessitated oversampling, highlighting a common issue in real-world NLP tasks and the need for robust handling strategies.

Key Results

The mBERT model demonstrated superior performance across both tasks:

  • Task 1 (Binary Classification - Fake vs. Original): mBERT achieved a macro F1-score of 0.83 and an accuracy of 0.85.
  • Task 2 (Multi-class Classification - Truthfulness Levels): mBERT achieved a macro F1-score of 0.54 (though the paper also states an accuracy of 0.80 for this task, the F1 score is more indicative of performance across all classes).

In comparison, the Naive Bayes model achieved a macro F1-score of 80.23 for Task 1, indicating mBERT's significant advantage. The authors ranked 11th and 4th for Task 1 and Task 2, respectively.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Scarcity: The primary challenge remains the lack of large-scale annotated datasets for low-resource languages like Malayalam.
  • Morphological Complexity and Code-Mixing: The inherent complexity of Malayalam, including its morphology and the prevalence of code-mixed text, poses difficulties for even advanced models.
  • Reliance on Pre-trained Models: While effective, existing multilingual models might not fully capture the unique nuances of Malayalam compared to language-specific models.
  • Nuanced Misinformation Detection: Detecting subtle forms of misinformation, such as satire or partially false claims, remains challenging and requires deeper semantic understanding.
  • Data Imbalance: Imbalances in multi-class datasets can affect model generalizability, suggesting a need for more refined balancing techniques.

Future work could involve exploring domain-specific pre-training, developing more sophisticated data augmentation techniques, and investigating hybrid approaches that combine different model architectures or feature sets.

Summary updated

BibTeX
@inproceedings{k2025nlp,
  author = {Srihari V K and Vijay Karthick Vaidyanathan and Thenmozhi Durairaj},
  title = {{NLP\_goats@DravidianLangTech 2025: Detecting Fake News in Dravidian Languages: A Text Classification Approach}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure