RATHAN@DravidianLangTech 2025: Annaparavai - Separate the Authentic Human Reviews from AI-generated one

Jubeerathan Thevakumar, Luheerathan ThevakumarNAACL 2025

TL;DR

This paper tackles the growing problem of AI-generated fake reviews in low-resource languages like Tamil and Malayalam. They developed a transfer learning approach using various pre-trained language models and an ensemble method to achieve high accuracy in distinguishing between human and AI-written product reviews.

What Problem Does This Paper Address?

The proliferation of AI-generated content poses a significant threat to the integrity of online reviews. These fake reviews can mislead consumers, create unfair competition, and erode trust in e-commerce platforms. While research on detecting AI-generated text has focused on high-resource languages like English, there's a critical gap in addressing this issue for low-resource languages such as Tamil and Malayalam. The scarcity of linguistic resources and tools for these languages makes detection particularly challenging. This paper aims to bridge this gap by developing effective AI-generated review detection methods for Tamil and Malayalam.

Methods

The authors propose a multi-stage approach:

  1. Transfer Learning for Embeddings: They leverage pre-trained language models (XLM-ROBERTa, IndicBERT, mT5, and Sentence-BERT) to generate rich embeddings for the review texts. These models are chosen for their multilingual capabilities and effectiveness in capturing semantic and syntactic nuances.
  2. Cross-Validation with XGBoost: To assess the discriminative power of the generated embeddings, they employ a five-fold cross-validation strategy using XGBoost. This step helps in understanding which embeddings are most effective for the task.
  3. Deep Neural Network (DNN) Training: Four independent DNN models are trained, each using the embeddings from one of the four chosen language models.
  4. Weighted Ensemble Model: To further boost performance and robustness, the predictions from the four individual DNN models are combined using a weighted averaging ensemble technique. This approach aims to leverage the complementary strengths of each model.

Datasets

The study utilizes two datasets introduced by Premjith et al. (2025):

  • Tamil Dataset: 808 samples for training and 100 for testing.
  • Malayalam Dataset: 800 samples for training and 200 for testing.

Both datasets are annotated with "Human" and "AI" labels, representing authentic and AI-generated product reviews, respectively. The paper notes that the dataset consists of short texts (1-2 sentences, 3-4 words per sentence) and requires no extensive preprocessing due to its clean and standardized format.

Key Insights and Ideas

  • Transfer Learning is Key for Low-Resource Languages: The success of using pre-trained multilingual models like XLM-ROBERTa and IndicBERT highlights their effectiveness in overcoming the lack of language-specific resources for Tamil and Malayalam.
  • Ensembling Improves Robustness: Combining predictions from multiple models (DNNs trained on different embeddings) significantly enhances overall detection performance, suggesting that different models capture different aspects of AI-generated text.
  • Stylometric Features are Implicitly Captured: While not explicitly extracted as separate features, the underlying language models used for embeddings likely capture stylistic differences that are crucial for distinguishing AI from human text.

Key Results

The proposed method achieved impressive results:

  • Malayalam: 90% accuracy (specifically, the ensemble model achieved 0.940 F1-score on a 21% split of the training data, and 0.90 on the test set).
  • Tamil: 73% accuracy (specifically, the ensemble model achieved 0.73 F1-score on the test set, and 0.982 on a 21% split of the training data).

The paper also presents confusion matrices showing the performance on the test sets, with the Malayalam model demonstrating particularly strong performance in correctly identifying both human and AI-generated reviews.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Biases: AI-generated reviews in the dataset might inherit biases from the language models used to create them, potentially affecting the detection model's fairness and performance.
  • Dataset Size: The limited size of the dataset could constrain the model's ability to learn comprehensively.
  • Performance Drop in Tamil: A noticeable performance drop was observed on the Tamil test set compared to cross-validation results. The authors attribute this discrepancy to potential differences in the distributions of training and test sets, possibly due to different LLM models generating the data.

Future work will focus on refining the ensemble DNN model to ensure uniformity across varying data distributions and addressing the inherent biases and dataset limitations. The source code is publicly available to encourage further research and development in this area.

Summary updated

BibTeX
@inproceedings{thevakumar2025rathan,
  author = {Jubeerathan Thevakumar and Luheerathan Thevakumar},
  title = {{RATHAN@DravidianLangTech 2025: Annaparavai - Separate the Authentic Human Reviews from AI-generated one}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure