Girma@DravidianLangTech 2025: Detecting AI Generated Product Reviews

Girma Yohannis Bade, Muhammad Tayyab Zamir, Olga Kolesnikova, José Luis Oropeza, Grigori Sidorov, Alexander GelbukhNAACL 2025

TL;DR

This paper tackles the growing problem of AI-generated fake product reviews in low-resource languages like Malayalam and Tamil. By fine-tuning a BERT-based model on specific datasets, they achieve promising results, demonstrating BERT's effectiveness in capturing the complex linguistic nuances of these languages for detection tasks.

What Problem Does This Paper Address?

The proliferation of AI-generated content, particularly fake product reviews, poses a significant threat to the authenticity and trustworthiness of e-commerce platforms. While substantial research has focused on detecting these deceptive reviews in high-resource languages like English, there's a notable gap in addressing this issue for low-resource languages such as Malayalam and Tamil. These languages, spoken by millions in South India and its diaspora, present unique linguistic challenges (e.g., high morphology, agglutination) that make them difficult for standard NLP techniques. This lack of attention leaves these communities vulnerable to the negative impacts of fake reviews, which can mislead consumers, damage company reputations, and distort market competition.

Methods

The core of this research lies in leveraging and adapting a BERT-based model for AI-generated review detection. The methodology involves:

  • Fine-tuning BERT: A pre-trained BERT model (specifically, the bert-base-uncased variant) is fine-tuned on custom datasets tailored for Malayalam and Tamil. This process allows the model to learn the specific linguistic patterns and characteristics of these languages.
  • Transformer Architecture: The paper highlights the power of transformer architectures, like BERT, in NLP tasks due to their ability to capture context and deep language features.
  • Data Preprocessing: Standard preprocessing steps are applied, including punctuation removal, emoji removal, and username removal, using Python's re module.
  • Feature Extraction: The BertTokenizer is used to convert text input into numerical representations, a crucial step for AI algorithms.
  • Evaluation Metric: The Macro F1-score is employed as the primary evaluation metric. This metric is chosen for its ability to balance precision and recall, making it suitable for classification problems, especially those with potentially imbalanced datasets.

Datasets

The study utilizes datasets curated for the "Shared Task for Detector AI-generated Product Reviews in Dravidian Languages." These datasets are organized into training and test subsets.

  • Training Data: Contains labeled data (human-written vs. AI-generated) for both Malayalam and Tamil.
  • Tamil: 808 samples (403 HUMAN, 405 AI)
    • Malayalam: 800 samples (400 HUMAN, 400 AI)
  • Test Data: Contains unlabeled data for both languages, where the model's task is to predict the label.
  • Tamil: 100 samples
    • Malayalam: 200 samples

The paper notes that while datasets are crucial, there's a general scarcity of annotated data for Dravidian languages, making the "Dravidian CodeMix" and the shared task datasets particularly valuable.

Key Insights and Ideas

  • BERT's Efficacy in Low-Resource Languages: The results strongly suggest that BERT, when fine-tuned, can effectively capture the complex linguistic features of low-resource languages like Malayalam and Tamil, outperforming simpler methods.
  • Language-Specific Performance Differences: The model shows significantly better performance for Malayalam than for Tamil. This could be attributed to the specific linguistic characteristics of each language and how well BERT's architecture adapts to them.
  • Importance of Domain-Specific Fine-tuning: The success of the BERT model highlights the necessity of fine-tuning general-purpose models on domain-specific data for specialized tasks like fake review detection.
  • Ethical Considerations: The paper touches upon the ethical imperative of detecting AI-generated content to maintain trust in online platforms, especially in vulnerable linguistic communities.

Key Results

The BERT-based model achieved the following Macro F1 scores:

  • Tamil: 0.6394
  • Malayalam: 0.8849

These results indicate that the model is considerably more effective in detecting AI-generated reviews in Malayalam compared to Tamil. The paper also presents a comparative analysis showing the BERT model's performance against other baseline techniques, where it achieved the highest F1-score for Malayalam and a competitive score for Tamil.

Limitations and Future Work

Limitations:

  • Dataset Size: The datasets used for training and testing were limited in size. This constraint means the observed performance might not generalize perfectly to all unseen data.
  • Generalization: While the model performed comparably to other methods in detecting AI-generated reviews on social media posts, its performance in highly competitive environments might be further tested.

Future Work:

  • More Diverse Datasets: The authors plan to improve the work by incorporating more diverse domains and expanding the datasets.
  • Multilingual Models and Cross-Lingual Transfer: Exploring multilingual models and cross-lingual transfer learning setups is a key direction to enhance performance.
  • Explainable AI (XAI): Integrating explainable AI techniques is proposed to increase detection accuracy and support broader applications.
  • Bias Mitigation: Addressing biases in detection systems for these languages is also identified as an area for future improvement.

Summary updated

BibTeX
@inproceedings{bade2025girma,
  author = {Girma Yohannis Bade and Muhammad Tayyab Zamir and Olga Kolesnikova and José Luis Oropeza and Grigori Sidorov and Alexander Gelbukh},
  title = {{Girma@DravidianLangTech 2025: Detecting AI Generated Product Reviews}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure