Team Innovative at SemEval-2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection

Surbhi Sharma, Irfan MansuriNAACL 2024

TL;DR

This paper introduces "RoBERTa hybrid," a novel approach for detecting machine-generated text by combining the power of RoBERTa with traditional stylometric features and character-level n-grams. The method achieved a respectable 73% accuracy on the SemEval-2024 Task 8, demonstrating a promising step towards more robust machine-generated text detection.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) like ChatGPT has made it increasingly difficult to distinguish between human-written and machine-generated text. This blurring of lines poses significant risks, including the spread of misinformation, fake reviews, and propaganda. The SemEval-2024 Task 8 specifically targets this challenge by requiring systems to detect machine-generated text across multiple generators, domains, and languages, highlighting the urgent need for reliable automated detection methods.

Methods

The core of the proposed solution is the "RoBERTa hybrid" model. This approach leverages a fine-tuned RoBERTa model, a powerful transformer-based language model known for its ability to understand context. To enhance its detection capabilities, the RoBERTa model's output probabilities are combined with two other feature sets:

  • Stylometric Features: These capture the unique writing style of an author. The paper extracts features like text length, word count, average word length, proportion of digits and capital letters, individual letter frequencies, hapax-legomena (words appearing only once), and punctuation frequencies.
  • Hybrid Features: This set includes the 100 most frequent character-level bi-grams and tri-grams, which capture local patterns in the text.

These combined features are then fed into a final Logistic Regression model for classification. The authors also experimented with individual classifiers (Stylometric, Hybrid Features, and RoBERTa + Style) before arriving at their final hybrid model.

Datasets

The experiments were conducted on the dataset provided for SemEval-2024 Task 8, Subtask A. This task utilized a monolingual dataset (primarily English) for distinguishing between human-written and machine-generated text. The dataset was labeled with '1' for machine-generated text (specifically from ChatGPT) and '0' for human-written text. The dataset comprised:

  • Training set: 119,757 samples
  • Development set: 5,000 samples
  • Testing set: 34,272 samples

The training set had a class distribution of 63,351 human-authored texts and 56,406 machine-generated texts. The development set had 2,500 samples, all labeled.

Key Insights and Ideas

  • The Synergy of Multiple Feature Types: The paper strongly suggests that combining deep learning representations (from RoBERTa) with traditional linguistic features (stylometric and n-grams) leads to superior performance compared to using any single approach alone. This highlights that different methods capture complementary aspects of text generation.
  • RoBERTa's Strength in Context: RoBERTa's ability to capture rich contextual information is a significant advantage in detecting subtle differences between human and machine writing.
  • Limitations of Stylometry Alone: The results indicate that traditional stylometric features, while useful, may not be sufficient on their own to accurately distinguish machine-generated text, especially as models become more sophisticated.
  • The "Hybrid" Advantage: The "RoBERTa hybrid" model, which integrates probabilities from RoBERTa, stylometric features, and hybrid features, achieved the highest accuracy, underscoring the effectiveness of this ensemble approach.

Key Results

The "RoBERTa hybrid" model achieved:

  • Validation Accuracy: 89%
  • Test Accuracy: 73%

Individually, the classifiers performed as follows on the test dataset:

  • Stylometric Classifier: 49% accuracy
  • Hybrid Features Classifier: 58% accuracy
  • RoBERTa + Style Classifier: 73% accuracy
  • Hybrid Classifier (RoBERTa + Style + Hybrid): 73% accuracy

The paper notes that their 73% test accuracy secured them the 74th position on the leaderboard for Subtask-A of SemEval-2024 Task 8.

Limitations and Future Work

  • Performance Gap: While the results show promising advancements, the authors acknowledge that their accuracy does not surpass all baseline methods.
  • Monolingual Focus: The current approach was primarily tested on a monolingual dataset (English).
  • Future Directions: The authors suggest several avenues for future research:
  • Exploring additional feature representations.
  • Optimizing model architectures.
  • Addressing the challenges posed by multilingual machine-generated text.
  • Further enhancing the accuracy and robustness of machine-generated text detection systems.

Summary updated

BibTeX
@inproceedings{sharma2024team,
  author = {Surbhi Sharma and Irfan Mansuri},
  title = {{Team Innovative at SemEval-2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure