LexiLogic@DravidianLangTech 2025: Detecting Fake News in Malayalam and AI-Generated Product Reviews in Tamil and Malayalam

Souvik Bhattacharyya, Pranav Gupta, Niranjan Kumar M, Billodal RoyNAACL 2025

TL;DR

This paper tackles the growing challenge of detecting fake news and AI-generated content in low-resource Dravidian languages, specifically Malayalam and Tamil. The authors present their participation in the NAACL DravidianLangTech 2025 workshop, showcasing their models' performance on these critical tasks and highlighting the difficulties posed by data scarcity and evolving AI capabilities.

What Problem Does This Paper Address?

The proliferation of AI-generated content, including fake news and synthetic product reviews, poses a significant threat across various online domains. While tools for detecting such content exist, they are often limited to high-resource languages, leaving languages like Malayalam and Tamil underserved. This gap is exacerbated by the rapid advancements in Large Language Models (LLMs), making it increasingly difficult to distinguish between human and machine-generated text. Furthermore, the nuances of cultural context and evolving language use in these regions add another layer of complexity to detection efforts. The paper aims to address this by developing and evaluating models for fake news detection in Malayalam and AI-generated product review detection in both Tamil and Malayalam.

Methods

The authors employed a multi-pronged approach, experimenting with various models for each task.

For fake news detection in Malayalam, they explored:

  • TFIDF + Logistic Regression: A traditional baseline method for text representation and classification.
  • FastText: A shallow neural network model known for efficient text classification.
  • GPT-40: An instruction-finetuned LLM from OpenAI, used with specific prompts designed to incorporate cultural context.
  • Malayalam BERT: A monolingual BERT model specifically fine-tuned for Malayalam, leveraging its ability to understand the language's nuances.

For AI-generated product review detection in Tamil and Malayalam, they adopted a fine-tuning approach using transformer-based models:

  • Multilingual BERT (bert-base-multilingual): A model pre-trained on a wide range of languages.
  • Monolingual BERT models (tamil-bert and malayalam-bert): These models were fine-tuned from the multilingual BERT using monolingual corpora.
  • GPT-2 (124M parameter version): A transformer-based decoder-only model.

The fine-tuning process involved initializing models with pre-trained weights, adding a classification head, and using stochastic gradient descent with backpropagation. For binary classification tasks, cross-entropy loss was used. Datasets were split into 80% training and 20% testing, with the ADAM optimizer and an exponential learning rate scheduler.

Datasets

The paper utilized datasets provided by the NAACL DravidianLangTech 2025 workshop organizers.

  • Malayalam Fake News Detection: Datasets for both binary ("original" vs. "fake") and multi-class ("half true," "partly false," "mostly false," "false") classification were used. These datasets included social media posts in pure and code-mixed Malayalam, with varying script usage.
  • AI-Generated Product Review Detection: Separate datasets for Tamil and Malayalam were provided, each containing a mix of human-generated and AI-generated product reviews. These were structured as binary classification problems.

Key Insights and Ideas

  • The importance of language-specific models: The performance of Malayalam BERT in fake news detection suggests that monolingual models, when available, can outperform larger multilingual models for specific low-resource languages by capturing finer linguistic and cultural nuances.
  • LLMs show promise but require careful prompting: GPT-40, despite being a general-purpose LLM, demonstrated strong performance in the multi-class fake news task when provided with carefully crafted prompts that encouraged cultural context consideration. This highlights the potential of LLMs but also the need for effective prompt engineering.
  • Data imbalance is a significant challenge: The multi-class fake news dataset exhibited a notable imbalance, particularly with "partly false" entries being scarce. This made classification more difficult, underscoring the need for strategies to handle imbalanced data.
  • Transformer models generally outperform simpler baselines: For AI-generated review detection, BERT-based models consistently outperformed GPT-2, indicating the effectiveness of transformer architectures for this task.
  • The final test set can be more challenging: The difference in performance between the held-out test set and the internal test set for AI-generated review detection suggests that the final evaluation data contained more complex and subtle examples, posing a greater challenge.

Key Results

  • Malayalam Fake News Detection (Binary Classification): The authors achieved test macro F1 scores of 0.29 and 0.82 for the multi-class and binary sub-tasks, respectively. Their Malayalam BERT model ranked 6th out of 16 and 12th out of 21 submissions in these sub-tasks.
  • Malayalam Fake News Detection (Multi-class Classification): GPT-40 performed best among the tested models, achieving a test macro F1 score of 0.167. TFIDF + Logistic Regression also showed competitive results.
  • AI-Generated Product Review Detection:
  • In Malayalam, their submission ranked 4th among 51 teams with a macro F1 score of 0.9.
  • In Tamil, their submission ranked 30th among 54 teams with a macro F1 score of 0.646.
  • Fine-tuned BERT models (bert-base-multilingual, tamil-bert, malayalam-bert) generally outperformed GPT-2 on the test sets, achieving high F1 scores (e.g., 0.9682 for malayalam-bert on the Malayalam test set).

Limitations and Future Work

The authors acknowledge several limitations:

  • Evolving nature of fake news and AI content: These issues are dynamic, requiring models to be adaptive to shifting patterns and new forms of disinformation.
  • Low-resource language challenges: Detecting content in languages with less available data is inherently more difficult due to limited training resources and complex cultural nuances.
  • Domain adaptation: Models trained on one domain may not perform well when transferred to another due to differences in context, vocabulary, and cultural references.
  • Potential for bias: NLP models can inadvertently inherit biases from their training data, potentially suppressing certain viewpoints or marginalizing demographic groups.
  • Interpretability and explainability: The "black-box" nature of many models necessitates further focus on understanding how they arrive at their decisions to address biases and foster trust.

Future work could involve exploring active and continual learning, developing more robust qualitative feedback mechanisms, and focusing on interpretability to build fairer and more trustworthy AI systems. The authors also suggest further exploration and error analysis for multi-class classification tasks and investigating hyperparameter tuning and data augmentation for imbalanced datasets.

Summary updated

BibTeX
@inproceedings{bhattacharyya2025lexilogic,
  author = {Souvik Bhattacharyya and Pranav Gupta and Niranjan Kumar M and Billodal Roy},
  title = {{LexiLogic@DravidianLangTech 2025: Detecting Fake News in Malayalam and AI-Generated Product Reviews in Tamil and Malayalam}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure