SemEval Task 8: A Comparison of Traditional and Neural Models for Detecting Machine Authored Text

Srikar Kashyap Pulipaka, Shrirang Mhalgi, Joseph Larson, Sandra KüblerNAACL 2024

TL;DR

As Large Language Models (LLMs) become more sophisticated, distinguishing machine-generated text from human text is increasingly challenging. This paper explores a range of traditional and neural models for this task, finding that a fine-tuned XLM-RoBERTa model achieves the best performance, though with a notable discrepancy between development and test set results due to distribution shifts.

What Problem Does This Paper Address?

The rapid advancement and accessibility of Large Language Models (LLMs) have led to a surge in machine-generated content across various domains, from education to fake news generation. This proliferation raises concerns about potential misuse, such as the creation of fake reviews or spam. Crucially, human ability to detect such machine-generated text is often at chance level, highlighting an urgent need for robust automated detection systems. This paper addresses this challenge by comparing the effectiveness of traditional statistical methods against various neural network architectures, including fine-tuned transformer models, for binary classification of human-written versus machine-generated text.

Methods

The authors employed a multi-pronged approach, evaluating both traditional statistical methods and advanced deep learning models.

  • Statistical Features: They extracted features such as stopword ratio and average sentence length, hypothesizing that LLMs might exhibit different patterns in these areas compared to human writing. They also utilized TF-IDF and word unigram features.
  • Statistical Classifiers: These features were fed into several standard machine learning models, including Naïve Bayes, Random Forest, XGBoost, Logistic Regression, and Support Vector Classifier (SVC).
  • Deep Learning Models: The core of their investigation involved fine-tuning pre-trained transformer models. They experimented with BERT, its variants like RoBERTa and ALBERT, and specifically focused on XLM-RoBERTa. For a baseline, they also implemented a simple two-layer fully connected neural network.

The authors used the Hugging Face library for their transformer model implementations and scikit-learn for statistical models. Hyperparameter tuning was performed for the best-performing models.

Datasets

The primary dataset used was the M4 dataset from SemEval-2024 Task 8, specifically the English binary classification subtask. This dataset comprises:

  • Training Set: 119,757 samples (approximately 53% machine-generated).
  • Development Set: 5,000 samples with an equal ratio of human and machine-generated text.
  • Test Set: 34,272 samples (approximately 52.5% machine-generated).

The machine-generated texts in the training set originated from various LLMs including ChatGPT, DaVinci, Dolly, and Cohere. Human texts were sourced from platforms like Reddit, WikiHow, ArXiV, Wikipedia, and PeerRead. Notably, the development set used a different machine generation model (Bloomz) and had a balanced distribution, which the authors point out as a potential source of discrepancy with the test set.

Key Insights and Ideas

  • LLM Sophistication: The paper underscores the growing difficulty in distinguishing machine-generated text, even for advanced models.
  • Fine-tuning is Key: Fine-tuning pre-trained transformer models, particularly XLM-RoBERTa, proved significantly more effective than traditional statistical methods or simpler neural networks.
  • Model Size vs. Performance: Counterintuitively, the larger XLM-RoBERTa model did not outperform the base XLM-RoBERTa model, suggesting that simply increasing parameters doesn't guarantee better performance and that fine-tuning strategy is crucial.
  • Development vs. Test Set Discrepancy: A significant finding is that improvements seen on the development set did not fully translate to the test set. This is attributed to differences in data distribution between the two sets, highlighting the challenge of creating models that generalize well.
  • False Positives are a Challenge: The confusion matrix for the best model revealed a strong tendency to incorrectly classify human text as machine-generated (false positives), indicating an area for improvement.

Key Results

  • Best Performing Model: The optimized XLM-RoBERTa base model achieved the highest accuracy on the development set, reaching 79.90%.
  • Traditional Methods: Among statistical methods, Logistic Regression with ratio features achieved 67.14% accuracy, outperforming TF-IDF features.
  • BERT-based Models: ALBERT achieved 66.78%, while a fully connected neural network reached 70.11%.
  • Official Test Set Results: The authors' submission, using the fine-tuned XLM-RoBERTa system, achieved an accuracy of 74.96% on the test set, ranking 70th out of 137 teams.
  • Baseline Comparison: The official baseline accuracy was 74%, meaning their submission slightly improved upon it. However, the best-performing team achieved a remarkable 96.88%.

Limitations and Future Work

  • Data Processing and Augmentation: The authors acknowledge that they did not extensively explore data preprocessing or augmentation techniques, which could potentially improve model performance.
  • Development vs. Test Set Generalization: The discrepancy in performance between the development and test sets due to distribution shifts is a significant limitation.
  • Future Work: The authors suggest exploring ensemble methods that combine fine-tuned models with models using ratio features. Further work on text preprocessing and feature engineering is also recommended to enhance detection capabilities.

Summary updated

BibTeX
@inproceedings{pulipaka2024semeval,
  author = {Srikar Kashyap Pulipaka and Shrirang Mhalgi and Joseph Larson and Sandra Kübler},
  title = {{SemEval Task 8: A Comparison of Traditional and Neural Models for Detecting Machine Authored Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure