L3i++ at SemEval-2024 Task 8: Can Fine-tuned Large Language Model Detect Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text?

Hanh Thi Hong Tran, Tien Nam Nguyen, Antoine Doucet, Senja PollakNAACL 2024

TL;DR

This paper explores the effectiveness of fine-tuned Large Language Models (LLMs) for detecting machine-generated text across diverse sources, languages, and generators. The authors found that LLMs significantly outperform traditional methods and ranked highly in the SemEval-2024 Task 8 competition, demonstrating their potential for identifying AI-generated content.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) like ChatGPT and GPT-4 has led to an explosion of remarkably fluent and coherent machine-generated text. While this offers exciting possibilities, it also raises serious concerns about the potential for misuse, such as spreading misinformation and academic dishonesty. A critical challenge is that humans often struggle to distinguish between human-written and machine-generated text. This paper addresses the need for robust automated systems capable of reliably detecting machine-generated content, especially in complex scenarios involving multiple text generators, diverse domains, and various languages.

Methods

The researchers investigated three main categories of methods for detecting machine-generated text:

  • Metric-based methods: These leverage pre-trained LLMs to extract features from text, such as the probability of words given their context (Log-Likelihood), the rank of words, and their entropy. These metrics are then used to classify text.
  • Traditional sequence-labeling language models: They fine-tuned established models like RoBERTa (for English) and XLM-R (for multilingual tasks) as sequence classifiers. These served as strong baselines.
  • Large Language Models (LLMs): The core of their approach involved fine-tuning a large-scale LLM, specifically LLaMA-2-7b-hf, for the classification task. This approach leverages the advanced generative and understanding capabilities of modern LLMs.

The study focused on two subtasks: binary classification (human vs. machine) and multi-way classification (human vs. specific LLM generators).

Datasets

The study utilized datasets provided by the SemEval-2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection. These datasets included:

  • Subtask A (Monolingual and Multilingual Binary Classification): Data sourced from various origins like arxiv, peerread, reddit, wikihow, and Wikipedia, covering both English and multiple other languages.
  • Subtask B (Multi-Way Classification): Data designed to distinguish between human-written text and text generated by specific LLMs (ChatGPT, cohere, davinci, bloomz, dolly).

The paper notes that while Subtask B's dataset was balanced, Subtask A's monolingual version exhibited label imbalances, with some sources having fewer examples than others.

Key Insights and Ideas

  • LLMs Reign Supreme: Fine-tuned LLMs, particularly LLaMA-2, consistently outperformed both traditional sequence-labeling models and metric-based approaches across all tested tasks. This highlights the power of leveraging large pre-trained models for this detection challenge.
  • Multilingual Advantage: The multilingual version of Subtask A showed strong performance, with the classifier effectively detecting human vs. machine text across languages.
  • Challenges in Attribution: While the LLM classifier was adept at identifying machine-generated text in general, distinguishing between texts from specific LLM generators (especially "cohere" and "davinci") proved more challenging. This suggests potential overlaps in the output characteristics of different LLMs.
  • Human-Written Text Misclassification: In the monolingual binary classification task, the classifier occasionally misclassified human-written texts as machine-generated, a common pitfall in this domain.

Key Results

  • High Rankings: The L3i++ team achieved impressive results in the SemEval-2024 Task 8 competition, ranking 5th out of 62 teams in Multilingual Binary Human-Written vs. Machine-Generated Text Classification and 6th out of 70 teams in Multi-Way Machine-Generated Text Classification.
  • Performance Gains: Compared to the baseline, their fine-tuned LLaMA-2 model showed significant improvements, particularly in the multilingual Subtask A and Subtask B, with an average gain of approximately 10%.
  • Class-wise Accuracy: For Subtask A (multilingual), the classifier achieved up to 94% accuracy. However, in the monolingual version, the accuracy for detecting human-written text dropped to around 76% due to misclassifications. In Subtask B, the accuracy for identifying ChatGPT, bloomz, and dolly was very high (over 99%), while for cohere, it was around 60%.

Limitations and Future Work

The authors acknowledge several limitations:

  • Specificity and Domain Dependence: The classifier's effectiveness might vary across different LLM generators, specific use cases, or domains. Detecting texts from "cohere" and "davinci" served as an example of this challenge.
  • Robustness to Adversarial Attacks: The paper does not extensively explore how well the model withstands techniques like paraphrasing or adversarial perturbations designed to fool detectors.

For future work, the authors propose:

  • Investigating Robustness: Evaluating the classifier's resilience against adversarial attacks.
  • Improving Interpretability: Developing methods to make the model's decisions more understandable and explainable, which is crucial for trust and debugging.
  • Addressing Domain Dependence: Exploring ways to make the model more generalizable across different domains and text types.

Summary updated

BibTeX
@inproceedings{tran2024l,
  author = {Hanh Thi Hong Tran and Tien Nam Nguyen and Antoine Doucet and Senja Pollak},
  title = {{L3i++ at SemEval-2024 Task 8: Can Fine-tuned Large Language Model Detect Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure