MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews

Oana Ignat, Xiaomeng Xu, Rada MihalceaarXiv 2024 · DOI · arXiv

Abstract

The proliferation of Large Language Models (LLMs) has facilitated the creation of highly realistic, deceptive content, threatening the integrity of online consumer feedback. This paper introduces MAiDE-UP (Multilingual Ai-generateD fakE reviews), the first large-scale multilingual dataset designed to benchmark the detection of AI-generated hotel reviews. Our study demonstrates that while GPT-4 can effectively mimic human writing styles—deceiving human evaluators in nearly 30% of cases—fine-tuned transformer models like XLM-RoBERTa can detect these fabrications with up to 94.8% accuracy across ten languages.

Introduction

The digital economy relies heavily on consumer trust, often mediated through online reviews. However, the advent of LLMs like GPT-4 has enabled the automated generation of "hallucinated" experiences at scale. While detection tools exist for English-language AI text, the global nature of travel necessitates a multilingual approach. This research addresses the gap in multilingual AI-authored deception, investigating whether automated systems can distinguish between genuine human experiences and AI-generated fabrications across diverse linguistic contexts.

Methodology

The study follows a dual-phase approach: generation and detection.

Generation

The researchers utilized GPT-4 to generate 10,000 deceptive reviews. To ensure high fidelity, the model was prompted using a "few-shot" technique, incorporating real examples of the "pros/cons" format characteristic of platforms like Booking.com. The AI was assigned the persona of a "well-traveled native tourist" to enhance the realism of the generated text.

Detection and Analysis

Three primary classification strategies were evaluated:

  1. Interpretable Baselines: Traditional machine learning models (e.g., Random Forest) utilizing features such as word counts and readability metrics.
  2. State-of-the-Art Transformers: A fine-tuned XLM-RoBERTa model, capable of processing multiple languages within a shared embedding space.
  3. Human Baseline: Native speakers were tasked with classifying reviews to provide a benchmark for human detection capabilities.

Linguistic analysis was also performed to identify the "fingerprints" of AI writing, focusing on complexity, descriptiveness, and readability.

The MAiDE-UP Dataset

The authors introduced the MAiDE-UP dataset, comprising 20,000 reviews (10,000 real, 10,000 AI-generated). The dataset is balanced across 10 languages: Chinese, English, French, German, Italian, Korean, Romanian, Russian, Spanish, and Turkish. It covers 10 global capital cities and includes both positive and negative sentiment polarities. Real reviews were sourced from Booking.com to ensure a ground truth based on verified stays.

Key Insights and Results

Linguistic analysis revealed that AI-generated reviews tend to be more descriptive (higher adjective density) but possess lower readability scores compared to human-written reviews. AI models often focus on abstract concepts like "atmosphere" and "service," whereas humans frequently mention platform-specific logistics (e.g., "Booking," "check-in").

Most salient words distinguishing real human reviews from GPT-generated reviews across multiple languages.

The fine-tuned XLM-RoBERTa model significantly outperformed both traditional baselines and human evaluators. While humans achieved an average accuracy of 71.5%, the transformer model reached a peak accuracy of 94.8%. Notably, even with minimal training data, the model achieved high accuracy, surpassing the human baseline early in the training curve.

Detection accuracy of XLM-RoBERTa and Random Forest models across varying training data ratios.

Discussion and Limitations

The study highlights a significant disparity in NLP infrastructure; many linguistic tools are optimized for English, complicating analysis for languages like Turkish or Korean. Furthermore, GPT-4 showed varying levels of "stealth" across languages, with English and Korean being the most difficult to detect, likely due to higher representation in the model's training data.

Future work should explore the generalizability of these findings to open-source models and other high-stakes domains, such as political or medical misinformation. By providing the MAiDE-UP benchmark, this research offers a critical tool for maintaining trust in the global digital marketplace.

Summary updated

Abstract

Deceptive reviews are becoming increasingly common, especially given the increase in performance and the prevalence of LLMs. While work to date has addressed the development of models to differentiate between truthful and deceptive human reviews, much less is known about the distinction between real reviews and AI-authored fake reviews. Moreover, most of the research so far has focused primarily on English, with very little work dedicated to other languages. In this paper, we compile and make publicly available the MAiDE-up dataset, consisting of 10,000 real and 10,000 AI-generated fake hotel reviews, balanced across ten languages. Using this dataset, we conduct extensive linguistic analyses to (1) compare the AI fake hotel reviews to real hotel reviews, and (2) identify the factors that influence the deception detection model performance. We explore the effectiveness of several models for deception detection in hotel reviews across three main dimensions: sentiment, location, and language. We find that these dimensions influence how well we can detect AI-generated fake reviews.
BibTeX
@misc{ignat2024maide,
  author = {Oana Ignat and Xiaomeng Xu and Rada Mihalcea},
  title = {{MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews}},
  year = {2024},
  eprint = {2404.12938},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.12938},
}
Figure