SCaLAR at SemEval-2024 Task 8: Unmasking the machine : Exploring the power of RoBERTa Ensemble for Detecting Machine Generated Text

Anand Kumar M, Abhin B, Sidhaarth Sredharan MuraliNAACL 2024

TL;DR

This paper tackles the challenge of identifying text generated by specific AI models, going beyond simply detecting AI-generated text to pinpointing its origin. They achieve this by augmenting data, training multiple RoBERTa models, and ensembling their predictions using a weighted voting classifier, resulting in improved accuracy.

What Problem Does This Paper Address?

The rapid advancement and widespread use of Large Language Models (LLMs) have led to a surge in AI-generated content. While detecting AI-generated text is important for combating misinformation and plagiarism, it's also crucial to distinguish between outputs from different LLMs. This fine-grained classification is vital for understanding model vulnerabilities, transparency, and the impact of training data on model behavior. SemEval-2024 Task 8 specifically addresses this by requiring participants to classify text into one of five specific machine-generated categories.

Methods

The authors propose a methodology centered around ensemble learning and data augmentation.

  • Data Augmentation: To handle longer texts that exceed token limits and to increase data volume, they implement a strategy to split long instances into smaller segments while preserving their original labels. This ensures that no information is lost and the model can learn from the entire context.
  • RoBERTa Ensemble: They leverage the power of RoBERTa, a robustly optimized BERT variant, for text classification. Instead of relying on a single model, they train three different RoBERTa models on distinct subsets of the augmented data.
  • Weighted Voting Classifier: The predictions from these three RoBERTa models are then combined using a weighted voting classifier. The weights are determined by the R2-scores of each individual model's predictions on a validation set. This approach aims to give more influence to models that perform better, leading to a more accurate final prediction.

Datasets

The paper mentions using a dataset with 71,027 instances for training and 3,000 for validation. The specific name of this dataset is not explicitly stated, but it is referred to as the "M4 based dataset" in the context of the SemEval-2024 Task 8.

Key Insights and Ideas

  • Ensembling for Specificity: Combining multiple models, even those based on the same architecture (RoBERTa), can significantly improve performance in fine-grained classification tasks like identifying the source LLM.
  • Data Augmentation for Long Texts: The proposed method for splitting long texts is effective in maintaining model compatibility and preventing information loss, a common challenge with LLM-generated content.
  • R2-Score as a Weighting Mechanism: Using R2-scores to weight the predictions of individual models in an ensemble provides a data-driven way to leverage the strengths of each model.
  • Overfitting Concerns: The significant gap between validation accuracy (97.05%) and test accuracy (76.25%) highlights a common challenge in LLM-based tasks: the tendency for models to overfit to the training data.

Key Results

The proposed ensemble model achieved a validation accuracy of 97.05% and a testing accuracy of 76.25%. This performance surpassed the baseline RoBERTa model, which had an average accuracy of 75%. The individual RoBERTa models achieved validation accuracies ranging from 92.10% to 96.62%. The authors' final submission placed them 18th on the leaderboard.

Limitations and Future Work

The paper acknowledges that the model shows clear signs of over-fitting, with a substantial drop in accuracy from validation to testing. They suggest that techniques like early stopping and further data augmentation could help mitigate this. They also propose that exploring embeddings from LLMs could be a promising direction for tackling this task more effectively in the future.

Summary updated

BibTeX
@inproceedings{m2024scalar,
  author = {Anand Kumar M and Abhin B and Sidhaarth Sredharan Murali},
  title = {{SCaLAR at SemEval-2024 Task 8: Unmasking the machine : Exploring the power of RoBERTa Ensemble for Detecting Machine Generated Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure