MasonTigers at SemEval-2024 Task 8: Performance Analysis of Transformer-based Models on Machine-Generated Text Detection

Sadiya Sayara Chowdhury Puspo, Nishat Raihan, Dhiman Goswami, Al Nahian Bin Emran, Amrita Ganguly, Özlem UzunerNAACL 2024

TL;DR

This paper presents the MasonTigers team's approach to SemEval-2024 Task 8, a challenge focused on detecting machine-generated text across various languages and domains. They found that ensemble methods, combining multiple transformer models, significantly outperformed individual models, highlighting the complexity of distinguishing AI-generated content from human writing.

What Problem Does This Paper Address?

The rapid proliferation of machine-generated text (MGT) across various platforms, from news and social media to academic writing, presents a dual challenge. On one hand, it offers exciting possibilities for content creation and automation. On the other, it raises serious concerns about the potential for misuse, such as spreading misinformation or enabling academic dishonesty. This necessitates the development of robust automated systems capable of reliably distinguishing between human-written and machine-generated text. While previous research has largely focused on binary classification (human vs. MGT) in English, SemEval-2024 Task 8 broadened the scope to include multigenerator, multidomain, and multilingual MGT detection, along with identifying transitions within mixed human-MGT texts.

Methods

The MasonTigers team employed a multi-pronged strategy, leveraging transformer-based models, sentence transformers, and statistical machine learning techniques. For the core classification tasks (Subtask A: Binary Classification and Subtask B: Multi-Way Classification), they found that ensemble methods were key. Specifically, they utilized a weighted ensemble approach, where the weights were determined by the individual models' accuracies. This involved combining the predictions of models like RoBERTa, DistilBERT, and ELECTRA.

For Subtask A, they also explored zero-shot prompting and fine-tuning of FLAN-T5, a powerful language model. In Subtask C, which focused on detecting the transition point in human-MGT mixed texts, they experimented with embeddings generated by TF-IDF, PPMI, and RoBERTa, followed by Linear Regression and ElasticNet for prediction. An ensemble of these predictions was used to achieve the final result.

Datasets

The experiments were conducted using datasets provided for SemEval-2024 Task 8, which are an expansion of the M4 dataset. These datasets cover various sources, including Wikipedia, WikiHow, Reddit, arXiv, and PeerRead, and encompass multiple languages such as English, Urdu, Indonesian, and Russian. The task was divided into three subtasks:

  • Subtask A (Binary Classification): Distinguishing between human-written and machine-generated text. This had both monolingual and multilingual tracks.
  • Subtask B (Multi-Way Classification): Identifying the source LLM of the machine-generated text (e.g., ChatGPT, Cohere, Davinci, Bloomz, Dolly).
  • Subtask C (Human-Machine Mixed Text Detection): Identifying the point where a text transitions from human-written to machine-generated.

Key Insights and Ideas

  • Ensembles Reign Supreme: The most significant finding was the consistent superiority of ensemble methods over individual models across classification tasks. This suggests that combining diverse model perspectives is crucial for robust MGT detection.
  • FLAN-T5's Mixed Performance: While zero-shot prompting and fine-tuning of FLAN-T5 were explored, they generally yielded less satisfactory results compared to dedicated transformer models and ensembles, especially in the multilingual settings.
  • Subtask B's Nuance: Differentiating between specific LLMs (Subtask B) proved more challenging than simple binary classification, with models showing varying performance across different LLM outputs.
  • Subtask C's Outlier Challenge: Detecting transition points in mixed texts (Subtask C) was complicated by the presence of outliers, leading to a larger gap between development and test set performance.

Key Results

  • Subtask A (Monolingual): The weighted ensemble achieved an accuracy of 0.74 on the test set, outperforming the baseline RoBERTa model.
  • Subtask A (Multilingual): The ensemble achieved 0.60 accuracy on the test set, with XLM-R showing strong performance.
  • Subtask B: The ensemble achieved an accuracy of 0.65 on the test set, with DeBERTa showing strong individual performance.
  • Subtask C: The ensemble achieved a Mean Absolute Error (MAE) of 60.78 on the test set.

Limitations and Future Work

The authors acknowledge several limitations:

  • Computational Cost: The large datasets and complex models led to prolonged execution times and high GPU usage.
  • Text Length: The lengthy nature of the texts added to the processing challenges.
  • No Data Augmentation: The restriction against data augmentation increased the difficulty of achieving optimal performance.
  • Nuanced Distinctions: The subtle differences between human and machine-generated text, which can even fool humans, pose a significant challenge for models.

For future work, the authors suggest addressing the challenges in Subtask C by improving data preprocessing, handling outliers, selecting appropriate features, optimizing model complexity, and ensuring model stability through proper tuning and evaluation. They also highlight the ongoing need for reliable detection methods that work across multiple languages.

Summary updated

BibTeX
@inproceedings{puspo2024masontigers,
  author = {Sadiya Sayara Chowdhury Puspo and Nishat Raihan and Dhiman Goswami and Al Nahian Bin Emran and Amrita Ganguly and Özlem Uzuner},
  title = {{MasonTigers at SemEval-2024 Task 8: Performance Analysis of Transformer-based Models on Machine-Generated Text Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure