DUTh at SemEval 2024 Task 8: Comparing classic Machine Learning Algorithms and LLM based methods for Multigenerator, Multidomain and Multilingual Machine-Generated Text Detection

Theodora Kyriakou, Ioannis Maslaris, Avi ArampatzisNAACL 2024

TL;DR

This paper tackles the growing challenge of distinguishing AI-generated text from human-written content across various languages and domains. The authors compare traditional machine learning algorithms with large language models (LLMs), finding that LLMs generally outperform traditional methods, especially in multilingual settings.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) has made them incredibly useful, but also raises concerns about the potential for misuse, such as spreading misinformation or undermining critical thinking. A crucial need has emerged for automatic systems that can reliably detect AI-generated text. SemEval 2024 Task 8 specifically addresses this by focusing on "Multigenerator, Multidomain and Multilingual Black-Box Machine-Generated Text Detection," aiming to build robust detectors that can handle diverse text sources, generation models, and languages. The task includes classifying texts as purely human or machine-generated, identifying the specific generator, and even pinpointing the boundary in mixed human-machine texts.

Methods

The core of this work involves comparing the performance of classic machine learning (ML) algorithms against fine-tuned LLMs for text generation detection. The authors experimented with various preprocessing techniques, including lowercasing, removing punctuation and digits, and replacing URLs and emails with special tokens. They also explored different embedding methods like Word2Vec and TF-IDF, alongside BERT embeddings. For ML algorithms, they tested models such as Logistic Regression, XGBoost, Random Forest, and AdaBoost. The LLMs employed were primarily BERT, multilingualBERT (mBERT), and XLM-ROBERTa. The study focuses on fine-tuning these LLMs for the classification task, treating them as powerful feature extractors and classifiers.

Datasets

The datasets used in this study were provided by the SemEval 2024 Task 8 organizers. For SubtaskA (monolingual and multilingual) and SubtaskB, the datasets consist of both human-written texts and machine-generated texts. The human-written sources include "wikihow," "wikipedia," "peerread," "reddit," and "arxiv." The machine-generated texts come from generators like "chatGPT," "cohere," "davinci," "bloomz," and "dolly." The multilingual aspect covers English, Arabic, Bulgarian, Chinese, Indonesian, Urdu, German, and Russian. The authors also created new training and validation sets by consolidating and splitting the provided data to better suit their experimental setup.

Key Insights and Ideas

  • LLMs Dominate: Across the board, fine-tuned LLMs like mBERT, XLM-ROBERTa, and BERT consistently outperformed traditional machine learning algorithms in text generation detection, particularly in multilingual scenarios.
  • Multilingual Advantage: While BERT performed well on English text, XLM-ROBERTa showed superior performance on the multilingual subtask, highlighting the importance of models pre-trained on diverse linguistic data.
  • Preprocessing Nuances: For ML algorithms, specific preprocessing steps like removing stop words and creating custom features (e.g., phrase counts) could significantly boost performance, sometimes bringing them closer to LLM capabilities.
  • No One-Size-Fits-All: The study suggests that while LLMs are generally better, the "best" approach can depend on the specific subtask and dataset. For instance, AdaBoost with optimized Random Forest and removed stop words achieved competitive results on the monolingual subtask.
  • Generalization Challenges: The authors observed a significant drop in performance on the official test sets compared to their custom validation sets, attributing this to the test sets containing texts from new domains and generators (like "outfox" and "gpt-4") that were not present in the training data.

Key Results

  • SubtaskA Monolingual: The best ML model achieved an accuracy of 0.7584 (AdaBoost optimized), while BERT achieved 0.783 and MiniLM 0.7695.
  • SubtaskA Multilingual: XLM-ROBERTa achieved 95.5% accuracy on the custom multilingual validation set. The submitted model scored 76.45 on the official multilingual test set.
  • SubtaskB: The submitted model scored 56.683 on the official SubtaskB test set.
  • General Observation: LLMs generally achieved higher accuracies, with BERT reaching 0.783 on the monolingual task and XLM-ROBERTa excelling in the multilingual setting.

Limitations and Future Work

The primary limitation acknowledged is the models' difficulty in generalizing to unseen domains, generators, and languages, as evidenced by the performance drop on the official test sets. Future work could focus on:

  • Data Augmentation: Employing techniques to increase the diversity of training data and improve generalization.
  • Larger Models: Investigating the performance of even larger LLMs.
  • Improved Preprocessing: Further exploring preprocessing strategies tailored for LLMs to enhance their ability to detect subtle differences in generated text.
  • Domain Adaptation: Developing methods that allow models to adapt more effectively to new domains and generators.

Summary updated

BibTeX
@inproceedings{kyriakou2024duth,
  author = {Theodora Kyriakou and Ioannis Maslaris and Avi Arampatzis},
  title = {{DUTh at SemEval 2024 Task 8: Comparing classic Machine Learning Algorithms and LLM based methods for Multigenerator, Multidomain and Multilingual Machine-Generated Text Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure