Team art-nat-HHU at SemEval-2024 Task 8: Stylistically Informed Fusion Model for MGT-Detection

Vittorio Ciccarelli, Cornelia Genz, Nele Mastracchio, Wiebke Petersen, Anna Stein, Hanxin XiaNAACL 2024

TL;DR

This paper tackles the challenge of detecting machine-generated text (MGT) by proposing a novel fusion model. By combining a powerful RoBERTa-based pre-classifier with specialized correction classifiers that leverage linguistic, stylistic, and probabilistic features, the authors significantly improved MGT detection accuracy, achieving 85% on the SemEval-2024 Task 8.

What Problem Does This Paper Address?

The rapid advancement of large language models (LLMs) and generative AI has led to an explosion of machine-generated text. This poses significant challenges, from the spread of misinformation and deepfakes to the creation of fabricated legal documents. Consequently, accurately detecting MGT is becoming increasingly crucial. SemEval-2024 Task 8 specifically addresses this by focusing on multi- and monolingual MGT detection across various domains and models. This paper focuses on Subtask A, which deals with monolingual English MGT detection.

Methods

The core of the proposed solution is a fusion model that combines a pre-classifier with correction classifiers.

  • RoBERTa Pre-classifier: The system starts with a RoBERTa Base OpenAI Detector, a model already fine-tuned for AI-generated text detection. This pre-classifier was further fine-tuned on a small portion (10%) of the training data.
  • Correction Classifiers: To address the limitations of the pre-classifier (which showed a bias towards the "machine" class), two Multi-layer Perceptron (MLP) correction classifiers were introduced. These MLPs were trained on a curated set of linguistic, stylistic, and probabilistic features. The idea is that these features can capture nuances that the RoBERTa model might miss or misinterpret, allowing the correction classifiers to refine the pre-classifier's predictions.
  • Feature Engineering: A comprehensive set of 70 features was extracted, including:
  • Count-based features: Sentence length, punctuation ratios, word type ratios, hapax legomena, negation words.
  • Syntactic features: Part-of-speech ratios, dependency tree depth, passive constructions.
  • Frequency features: Logarithmic frequency of content words, frequent words, hapax legomena, and words from Wiktionary frequency lists.
  • Word difficulty features: Vocabulary lists based on CEFR levels.
  • Stylistic and sentiment features: Opinion word ratios, readability scores, emotion detection logits, sentiment analysis logits, formality detection, and toxicity classification.
  • Features from the pre-classifier: Logits and reduced (via PCA and UMAP) last hidden states of the RoBERTa pre-classifier were also used as input for the correction classifiers.
  • Feature Selection: A rigorous feature selection process was employed, involving standard scaling, removal of highly correlated features, and retention of features with medium or strong correlation to the gold label. This resulted in a final set of 26 features.
  • Model Training: The correction classifiers were trained separately on texts predicted as "human" and "machine" by the pre-classifier, aiming to allow them to learn specific correction patterns.

Datasets

The primary dataset used is the M4 dataset for SemEval-2024 Task 8, Subtask A. This dataset consists of texts from six different LLM generators (Davinci, chatGPT, Dolly, Cohere, BLOOMz, and GPT4) across five genres (Reddit, WikiHow, ArXive, Wikipedia, and peerRead).

  • Training data: 119,757 texts
  • Development data: 5,000 texts
  • Test data: 34,272 texts

The paper notes that no further data augmentation was performed. The dataset has a slight imbalance, with approximately 53% machine-generated texts and 47% human-written texts in the train and test sets.

Key Insights and Ideas

  • The power of fusion: A single powerful model (like RoBERTa) can be biased. Combining it with specialized models that focus on different aspects (like linguistic features) can create a more robust and accurate system.
  • Linguistic features matter: Even with advanced transformer models, traditional linguistic, stylistic, and probabilistic features still hold significant discriminative power for MGT detection.
  • Addressing pre-classifier bias: The RoBERTa pre-classifier, while strong, developed a bias towards the "machine" class. The correction classifiers effectively counteracted this bias, leading to a more balanced performance.
  • Domain-specific fine-tuning is not always necessary: The authors found that their approach, which did not explicitly incorporate domain-specific information, performed well, suggesting that the chosen features and fusion strategy are generalizable.
  • Reproducibility and environmental impact: The use of a pre-trained RoBERTa model and readily available tools like Google Colab makes the approach accessible and potentially more environmentally friendly.

Key Results

  • The proposed fusion model achieved an overall accuracy of 85% on the SemEval-2024 Task 8 test set.
  • This represents a significant improvement over the RoBERTa pre-classifier alone, which achieved an accuracy of 75% on the test set.
  • The MLP correction classifiers, when trained on linguistic features alone, achieved an accuracy of 58%.
  • The fusion model demonstrated balanced precision and recall for both "human" and "machine" classes, unlike the pre-classifier which was biased towards "machine" and the MLP which was biased towards "human."

Limitations and Future Work

  • Limitations:
    • The study did not examine the influence of domain on accuracy, as the test data only contained one domain.
  • The authors acknowledge that the low correlation threshold for feature selection might have led to the inclusion of weakly correlated features, potentially affecting performance.
  • The test set data might have a different distribution of features compared to the training and development data.
  • The pre-classifier's fine-tuning data had an imbalance towards machine-generated texts, contributing to its bias.
  • Future Work:
    • Expand the fusion model by incorporating more semantic-level or complex features, such as contextual predictability.
  • Fine-tune the pre-classifier using more balanced data.
  • Explore different feature selection strategies.
  • Investigate the impact of domain on the model's performance.
  • The authors make their code available on GitHub, encouraging further research and expansion.

Summary updated

BibTeX
@inproceedings{ciccarelli2024team,
  author = {Vittorio Ciccarelli and Cornelia Genz and Nele Mastracchio and Wiebke Petersen and Anna Stein and Hanxin Xia},
  title = {{Team art-nat-HHU at SemEval-2024 Task 8: Stylistically Informed Fusion Model for MGT-Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure