Groningen Team F at SemEval-2024 Task 8: Detecting Machine-Generated Text using Feature-Based Machine Learning Models

Rina Donker, Björn Overbeek, Dennis Thulden, Oscar ZwagersNAACL 2024

TL;DR

This paper tackles the growing challenge of detecting machine-generated text by developing feature-based machine learning models for the SemEval-2024 Task 8. The team achieved respectable accuracy on both binary and multi-class detection tasks, highlighting the effectiveness of carefully engineered linguistic features over complex LLM fine-tuning for this specific problem.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) like ChatGPT has led to an explosion of fluent, human-like text generation. While beneficial, this also raises significant concerns about the spread of misinformation and potential misuse, particularly in educational settings. The core problem is the urgent need for reliable systems that can distinguish between human-written and machine-generated text. Previous research has often been language-specific or focused on particular LLMs, limiting their generalizability. This paper addresses this by participating in a task designed to create more robust, potentially multilingual, detectors across various LLMs and domains.

Methods

The team employed a feature-based machine learning approach, opting for computationally less expensive models compared to fine-tuning LLMs. They experimented with four supervised models: Support Vector Machines (SVM), Naive Bayes, K-Nearest Neighbors (KNN), and Feed Forward Neural Networks (FFNN). The core innovation lies in the handcrafted features they engineered, categorized into token-level and sentence-level.

Key Features:

  • Token-level:
    • Personal pronouns vs. proper names: Exploiting the observation that humans tend to use pronouns more frequently after introducing a person by name, while LLMs might overuse proper names.
  • POS-tags and dependency tags: Analyzing the grammatical structure of text.
  • Sentence-level:
    • Sentence tense: Investigating patterns in the use of past, present, and future tenses.
  • Sentence voice: Differentiating between active and passive voice usage.
  • Sentence similarity: Measuring how similar a sentence is to its preceding and succeeding sentences.
  • Sentiment: Analyzing the emotional tone of sentences.
  • Domain: Incorporating the text's origin (e.g., Wikipedia, Reddit) as a feature.

These features were then fed into the chosen models for classification.

Datasets

The paper utilized datasets provided by the SemEval-2024 Task 8 organizers. They focused on the monolingual (English) track for both subtasks:

  • Subtask A: Binary classification (human vs. machine-generated). The monolingual training set contained 119,757 instances, and the development set had 5,000 instances.
  • Subtask B: Multi-class classification (identifying specific generators like human, ChatGPT, Cohere, etc.). The English training set had 71,027 instances, and the development set had 3,000 instances.

Key Insights and Ideas

  • Feature Engineering Matters: The study strongly suggests that well-crafted linguistic features can be highly effective for machine-generated text detection, even outperforming more complex models in some scenarios.
  • Linguistic Patterns are Key: Features like sentence tense, sentence voice, sentence similarity, and the ratio of personal pronouns to proper names consistently proved valuable across their best-performing models.
  • Sentiment is Less Informative: The sentiment feature showed minimal impact, likely because many of the text sources (like Wikipedia and arXiv) tend to have a neutral tone.
  • Simpler Models Can Compete: SVMs and FFNNs, when combined with strong features, performed well and were chosen for submission, demonstrating that sophisticated LLM fine-tuning isn't always necessary for good results.

Key Results

The team achieved the following accuracies:

  • Subtask A (Monolingual): 0.74 accuracy with a neural network using sentence tense, sentence voice, sentence similarity, and the ratio of pronouns/named entities. Their final submission (an SVM) ranked 95th out of 137 teams with an accuracy of 0.69.
  • Subtask B: 0.61 accuracy with an SVM model. This ranked them 50th out of 77 participating teams.

Both their best-performing models were outscored by a RoBERTa baseline (0.88 for Subtask A, 0.75 for Subtask B), indicating that while their feature-based approach is effective, state-of-the-art LLMs still hold an advantage.

Limitations and Future Work

The authors acknowledge that their models, while performing well, did not outperform the state-of-the-art. They attribute this partly to using "older techniques" but emphasize the computational efficiency and potential of their feature-based approach.

Future work suggestions include:

  • Experimenting with a wider variety of machine learning models.
  • Developing and incorporating more sophisticated features to further enhance detection capabilities.
  • Exploring multilingual capabilities, as they focused only on the monolingual track for Subtask A.

Summary updated

BibTeX
@inproceedings{donker2024groningen,
  author = {Rina Donker and Björn Overbeek and Dennis Thulden and Oscar Zwagers},
  title = {{Groningen Team F at SemEval-2024 Task 8: Detecting Machine-Generated Text using Feature-Based Machine Learning Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure