KECLinguAIsts@DravidianLangTech 2025: Detecting AI-generated Product Reviews in Dravidian Languages

Malliga Subramanian, Rojitha R, Mithun Chakravarthy Y, Renusri R V, Kogilavani ShanmugavadivelNAACL 2025

TL;DR

This paper tackles the critical challenge of detecting AI-generated product reviews in low-resource Dravidian languages like Tamil and Malayalam. By evaluating traditional machine learning models and highlighting the limitations of current approaches, the study demonstrates that Logistic Regression offers a robust solution for this complex task, paving the way for more trustworthy online content.

What Problem Does This Paper Address?

The proliferation of AI-generated content online, particularly in product reviews, raises serious concerns about authenticity. While detecting AI-generated text in high-resource languages like English has seen significant progress, this task remains largely underexplored for low-resource languages such as Tamil and Malayalam. These languages present unique hurdles due to their complex morphology, intricate syntactic structures, and frequent code-mixing, making it difficult for existing models to accurately distinguish between human and AI-generated content. The scarcity of annotated datasets further exacerbates this challenge, hindering the development of effective detection models.

Methods

The researchers employed a comparative approach, evaluating six machine learning models for AI-generated review detection: Logistic Regression (LR), Random Forest (RF), and XGBoost for Tamil, and Logistic Regression, Multinomial Naive Bayes (MNB), and Support Vector Machines (SVM) for Malayalam.

The methodology involved:

  • Preprocessing: Text data was cleaned, standardized, converted to lowercase, and stop words were removed. TF-IDF vectorization was used to convert text into numerical features, with tokenization and stemming applied to reduce words to their base forms.
  • Model Training and Evaluation: The models were trained and evaluated using standard metrics: Accuracy, Precision, Recall, and F1-Score.

While the paper acknowledges the potential of transformer-based models like BERT and XLM-R, its primary focus is on evaluating traditional machine learning classifiers due to practical constraints like computational power and the need for extensive fine-tuning on limited datasets.

Datasets

The study utilized a custom dataset consisting of:

  • Training Data: 1,608 samples (808 Tamil, 800 Malayalam)
  • Test Data: 300 samples (100 Tamil, 200 Malayalam)

Each sample was labeled as either "AI" or "HUMAN." The paper includes sample texts from both Tamil and Malayalam datasets to illustrate the nature of the data.

Key Insights and Ideas

  • Logistic Regression Shines: Despite the complexity of Dravidian languages, Logistic Regression emerged as the most effective model for detecting AI-generated content in both Tamil and Malayalam. This highlights its robustness and interpretability, making it a strong candidate for this task.
  • Traditional Models' Resilience: While transformer models are generally powerful, this study demonstrates that well-tuned traditional ML models can still offer competitive performance, especially when dealing with the specific challenges of low-resource languages and limited data.
  • TF-IDF Limitations: The error analysis suggests that TF-IDF, while a useful feature extraction technique, struggles to capture the subtle contextual nuances and code-mixing present in Dravidian languages, leading to misclassifications.

Key Results

  • Training Performance: On the training dataset, Logistic Regression achieved an accuracy of 86% for both Tamil and Malayalam, with high precision and recall. Random Forest and XGBoost also showed strong performance for Tamil.
  • Test Performance: On the test dataset, Logistic Regression maintained its lead, achieving an accuracy of 67% for Malayalam and 68% for Tamil. While transformer-based models are noted to outperform traditional classifiers in general, LR's performance in this study is considered superior among the evaluated traditional models.
  • Misclassifications: The error analysis revealed that the LR model misclassified instances, particularly with code-mixed text, indicating the inherent difficulty in capturing subtle linguistic patterns.

Limitations and Future Work

  • Dataset Size: The primary limitation acknowledged is the scarcity of large, annotated datasets for AI-generated content in Dravidian languages, which restricts model performance.
  • Transformer Model Integration: The authors suggest that future work should focus on creating larger annotated datasets and incorporating transformer-based models to improve contextual understanding and detection accuracy.
  • Hybrid Approaches: Exploring hybrid approaches that combine traditional and advanced models is recommended to further enhance detection capabilities across diverse linguistic contexts.
  • Reproducibility: The paper notes that the datasets and implementation code are publicly available on GitHub to support reproducibility and further research.

Summary updated

BibTeX
@inproceedings{subramanian2025keclinguaists,
  author = {Malliga Subramanian and Rojitha R and Mithun Chakravarthy Y and Renusri R V and Kogilavani Shanmugavadivel},
  title = {{KECLinguAIsts@DravidianLangTech 2025: Detecting AI-generated Product Reviews in Dravidian Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure