NLP_goats_DravidianLangTech_2025__Detecting_AI_Written_Reviews_for_Consumer_Trust

Srihari V K, Vijay Karthick Vaidyanathan, Mugilkrishna D U, Thenmozhi DurairajNAACL 2025

TL;DR

This paper tackles the growing challenge of detecting AI-generated product reviews in low-resource Dravidian languages like Tamil and Malayalam. By fine-tuning the mBERT model, the authors achieved strong performance, highlighting the need for language-specific enhancements to maintain trust in e-commerce platforms.

What Problem Does This Paper Address?

The proliferation of AI-generated content poses a significant threat to the authenticity and trustworthiness of online platforms, especially in the context of product reviews. AI-generated reviews can manipulate ratings, deceive consumers, and erode trust. While detecting AI-generated text is already challenging, it becomes even more complex in low-resource languages like Tamil and Malayalam due to their intricate linguistic structures, deep morphology, and code-mixing, coupled with a scarcity of annotated datasets and computational resources. This paper addresses the critical need for effective AI-generated review detection mechanisms in these underrepresented languages to preserve consumer trust in digital marketplaces.

Methods

The core of this research involves fine-tuning transformer-based models for binary classification of AI-generated versus human-written reviews. Specifically, the authors experimented with:

  • mBERT (Multilingual BERT): A pre-trained model that has been trained on a diverse, multilingual corpus, including Dravidian languages. This model was initialized and then fine-tuned on the specific task.
  • XLM-R (Cross-lingual Language Model RoBERTa): Another powerful multilingual transformer model, known for its ability to manage multiple languages and generalize across language structures.
  • Logistic Regression: Used as a baseline model, employing TF-IDF features from the preprocessed text. This serves to gauge the effectiveness of more complex transformer models against a simpler, traditional approach.

The methodology also includes a robust data preprocessing pipeline designed to handle the nuances of Dravidian languages. This involves cleaning text by removing special characters, punctuation, and numerical digits, converting to lowercase, normalizing spaces, and then tokenizing using a BERT-based multilingual tokenizer. Sequences are padded to a fixed length of 256 tokens.

Datasets

The study utilized datasets for detecting AI-generated product reviews in Tamil and Malayalam, sourced from previous studies (specifically mentioning Premjith et al., 2025). The paper provides label distributions for both languages, indicating a balanced dataset for training and testing.

  • Tamil Dataset: Contains AI-generated and human-written reviews.
  • Malayalam Dataset: Also contains AI-generated and human-written reviews.

The paper notes that these datasets were further stratified into training (80%) and testing (20%) subsets to avoid class imbalance.

Key Insights and Ideas

  • Transformer models excel in low-resource languages: The study strongly suggests that transformer-based models like mBERT and XLM-R are significantly more effective than traditional methods (like Logistic Regression) for detecting AI-generated reviews in Dravidian languages.
  • Language-specific challenges persist: Despite the success of transformer models, the performance difference between Tamil and Malayalam highlights the inherent difficulties in processing the latter. This points to the need for further language-specific enhancements.
  • Misclassifications reveal subtle AI writing: The error analysis shows that models sometimes struggle to differentiate human reviews that mimic AI writing styles, leading to false positives. This indicates that AI is becoming increasingly sophisticated in its ability to replicate human nuances.
  • The importance of linguistic features: The paper emphasizes that capturing linguistic variation, context, and style is crucial for accurate classification, especially in languages with complex morphology.

Key Results

The paper reports the following performance metrics:

  • Tamil:
    • mBERT: Achieved a macro F1-score of 0.90.
  • XLM-R: Achieved a macro F1-score of 0.92.
  • Malayalam:
    • mBERT: Achieved a macro F1-score of 0.68.
  • XLM-R: Achieved a macro F1-score of 0.75.

Logistic Regression, as a baseline, performed less effectively, with F1-scores around 0.81-0.83 for both languages. The authors note that XLM-R showed a notable improvement over mBERT in Malayalam, suggesting better management of its language complexities.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scarcity of annotated data and computational resources: This is a primary challenge for low-resource languages like Tamil and Malayalam, impacting the potential for successful AI-generated text detection.
  • Performance gap in Malayalam: The lower macro F1-score in Malayalam (0.68) compared to Tamil (0.90) underscores the difficulty in detecting linguistic subtleties, morphology, and code-mixing.
  • Transformer models still miss subtle signals: Even advanced models can miss subtle contextual signals, leading to misclassifications, particularly in human reviews that closely resemble AI-generated text.
  • Binary classification focus: The current work focuses on binary classification and does not address more complex scenarios like mixed AI and human content.

Future work suggested includes:

  • Using larger datasets.
  • Exploring more sophisticated fine-tuning methods.
  • Developing ensemble models that combine linguistic and behavioral features.
  • Prioritizing enhanced fine-tuning and data augmentation.
  • Investigating hybrid strategies to improve detection precision in varied linguistic contexts.

Summary updated

BibTeX
@inproceedings{k2025nlp,
  author = {Srihari V K and Vijay Karthick Vaidyanathan and Mugilkrishna D U and Thenmozhi Durairaj},
  title = {{NLP\_goats\_DravidianLangTech\_2025\_\_Detecting\_AI\_Written\_Reviews\_for\_Consumer\_Trust}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure