TeamVision@DravidianLangTech 2025: Detecting AI generated product reviews in Dravidian Languages

Shankari S R, Sarumathi P, Bharathi BNAACL 2025

TL;DR

This paper tackles the growing challenge of AI-generated product reviews in low-resource Dravidian languages like Tamil and Malayalam. They developed and evaluated several machine learning and deep learning models, finding that BERT significantly outperforms traditional methods in accurately distinguishing AI-generated content. This work is crucial for maintaining trust in online marketplaces and supporting linguistic diversity in AI applications.

What Problem Does This Paper Address?

As AI language models become more sophisticated, they can generate product reviews that are virtually indistinguishable from those written by humans. This poses a significant threat to online platforms, as it can be used to manipulate consumer behavior and create a false sense of consensus. Detecting these AI-generated reviews is vital for ensuring consumer trust and informed decision-making. The problem is particularly acute for low-resource languages like Tamil and Malayalam, which have complex linguistic structures and limited annotated datasets, making it harder to develop effective detection tools.

Methods

The researchers explored a range of techniques for AI-generated text detection. Their methodology involved:

  • Traditional Machine Learning: They implemented Naive Bayes, Logistic Regression, Support Vector Machines (SVM), Random Forest, and K-Nearest Neighbors (KNN).
  • Deep Learning: A key component of their approach was the use of the Multilingual BERT (mBERT) model. mBERT was chosen for its ability to handle multiple languages without requiring separate models for each, making it suitable for the Dravidian languages.
  • Feature Engineering: Standard techniques like Bag of Words (BoW), TF-IDF, and n-grams were used to convert text into numerical formats for machine learning models.

The core innovation lies in applying and evaluating these diverse models, with a particular focus on the effectiveness of mBERT for this specific task and language group.

Datasets

The study utilized a dataset sourced from the "Detecting AI-generated product reviews in Dravidian languages" shared task at DravidianLangTech@NAACL 2025. This dataset was divided into training and testing sets for both Tamil and Malayalam. The training data included fields for ID, data, and label, supporting a supervised learning approach, while the testing data only contained ID and data for prediction.

Key Insights and Ideas

  • BERT's Superiority: The most striking insight is the significant advantage of deep learning models, particularly BERT, over traditional machine learning approaches for AI-generated text detection in these languages. BERT's ability to grasp deep contextual understanding is key.
  • Low-Resource Language Challenges: The paper highlights the inherent difficulties in working with low-resource languages, such as the scarcity of high-quality datasets, the prevalence of code-mixing (blending languages), and complex morphology, which challenge even advanced models.
  • Importance of Context: The findings underscore that while traditional models might struggle with nuanced language, BERT's transformer architecture allows it to better interpret context, sarcasm, and subtle sentiments, leading to higher accuracy.

Key Results

The study reported impressive performance metrics:

  • BERT: Achieved the highest accuracy, with an F1 score of 0.98 for both AI and human text in Tamil. For Malayalam, it reached an F1 score of 0.96 for both AI and human text.
  • Traditional Models: While performing moderately, they lagged significantly behind BERT. For instance, Naïve Bayes achieved around 90% accuracy in Tamil but dropped to 76.87% in Malayalam. SVM also showed good performance but was noted as computationally expensive.
  • Comparison: The BERT-based model significantly surpassed existing AI text detection tools, which often focus on English and lack the adaptability for Dravidian languages.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Size: The availability of large, high-quality datasets for Tamil and Malayalam remains a challenge, potentially leading to model bias and inaccuracies.
  • Language Nuances: Handling code-mixed text, dialects, and the rich morphology of these languages is difficult for current models.
  • Sarcasm and Sentiment: AI models, including BERT, can still misinterpret sarcasm and subtle sentiments.
  • Resource Intensive: Training large models like BERT requires significant computational resources, limiting practical deployment.
  • Model Degradation: Language and user review patterns evolve, necessitating continuous updates and retraining.

Future work could involve fine-tuning transformer models to better handle low-resource languages, improving their ability to understand context and generalize across data. Further research into robust handling of code-mixed text and context-aware features is also suggested.

Summary updated

BibTeX
@inproceedings{r2025teamvision,
  author = {Shankari S R and Sarumathi P and Bharathi B},
  title = {{TeamVision@DravidianLangTech 2025: Detecting AI generated product reviews in Dravidian Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure