UMUTeam at SemEval-2024 Task 8: Combining Transformers and Syntax Features for Machine-Generated Text Detection

Ronghao Pan, José Antonio García-díaz, Pedro José Vivancos-vicente, Rafael Valencia-garcíaNAACL 2024

TL;DR

This paper presents UMUTeam's approach to SemEval-2024 Task 8, focusing on distinguishing machine-generated text from human-written text. They achieved a solid 23rd place by combining the power of RoBERTa, a Transformer-based language model, with traditional syntactic features of text, outperforming the baseline.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) like GPT-3 and ChatGPT has led to an explosion of machine-generated content across various domains, from news to scientific research. While these models offer incredible capabilities, they also raise significant concerns about potential misuse, such as academic dishonesty, the spread of misinformation, and the erosion of trust in authentic content. SemEval-2024 Task 8 was designed to address this challenge by creating and evaluating systems capable of detecting machine-generated text, particularly in a "black-box" setting where the internal workings of the generation models are unknown. The task specifically aimed to differentiate between human-written text, text generated by specific LLMs, and text that is a mix of human and machine contributions.

Methods

UMUTeam's approach for Subtask B (multi-way machine-generated text classification) involved a two-pronged strategy:

  1. Transformer-based Encoding: They utilized RoBERTa, a pre-trained Transformer model, to generate dense vector representations (embeddings) of the input texts. These embeddings capture the semantic and contextual information of the text.
  2. Syntactic Feature Extraction: Alongside the Transformer embeddings, they extracted a suite of traditional syntactic features. These features, which reflect writing style, include:
  3. Average word length
    • Part-of-Speech (POS) tag frequency
    • Average sentence length (in characters and words)
    • Percentage of stopwords
    • Punctuation frequency
    • Special character frequency

These syntactic features were extracted using an open-source tool called authorstyle.

The core of their method was to concatenate the normalized syntactic features with the RoBERTa text embeddings. This combined vector was then fed into a sequence classification layer, and the entire model was fine-tuned on the task-specific dataset. This approach aimed to leverage both the deep contextual understanding of Transformers and the discriminative power of stylistic linguistic features.

Datasets

The authors used the dataset provided by the organizers for SemEval-2024 Task 8. For Subtask B, this dataset consisted of training and validation sets. The dataset was balanced and contained text from five different LLMs (davinci, bloomz, human, chatGPT, dolly, and cohere) and human-written text.

Key Insights and Ideas

  • Syntactic Features Complement Transformers: The paper strongly suggests that traditional syntactic features, which capture writing style, can significantly complement the performance of powerful Transformer-based models like RoBERTa for text generation detection. This indicates that stylistic nuances remain important discriminators, even with advanced generative models.
  • RoBERTa's Suitability: The authors found RoBERTa to be a suitable model for this task, likely due to its strong performance on a wide range of NLP tasks and its robust pre-training.
  • Importance of Feature Normalization and Concatenation: The explicit mention of normalizing syntactic features before concatenating them with text embeddings highlights a crucial step in ensuring that features from different sources contribute effectively to the model without one dominating the other.

Key Results

UMUTeam achieved the 23rd position out of 77 participants in Subtask B, with an accuracy of 75.350%. This score outperformed the baseline (74.606%) by a small margin.

The confusion matrix revealed interesting insights:

  • The model showed strong performance (above 90% accuracy) in predicting texts generated by Bloomz, Dolly, ChatGPT, and Davinci.
  • A notable weakness was observed in classifying human-generated texts, with a 27.73% tendency to misclassify them as generated by the Dolly model.
  • Another significant misclassification occurred with Cohere-generated texts, where the model tended to label them as Davinci-generated 70% of the time.

Limitations and Future Work

  • Misclassification of Human Text: The paper acknowledges a significant tendency to misclassify human-generated texts, particularly as Dolly-generated. This suggests that while the model is good at identifying some machine-generated patterns, it struggles with the nuances of human writing, especially when it might share characteristics with certain LLM outputs.
  • Specific LLM Confusion: The confusion between Cohere and Davinci-generated texts indicates that the model might not be fully capturing the unique stylistic fingerprints of all LLMs, or that these models produce outputs that are very similar in certain aspects.
  • Future Work: The authors mention their ongoing research into evaluating their system on texts containing figurative language and financial language. They anticipate that the ambiguity and creativity of figurative language, and the specialized vocabulary of financial text, will pose challenges for LLM detection systems, requiring careful review and validation.

Summary updated

BibTeX
@inproceedings{pan2024umuteam,
  author = {Ronghao Pan and José Antonio García-díaz and Pedro José Vivancos-vicente and Rafael Valencia-garcía},
  title = {{UMUTeam at SemEval-2024 Task 8: Combining Transformers and Syntax Features for Machine-Generated Text Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure