I2C-Huelva at SemEval-2024 Task 8: Boosting AI-Generated Text Detection with Multimodal Models and Optimized Ensembles

Alberto Rodero Peña, Jacinto Mata Vazquez, Victoria Pachón ÁlvarezNAACL 2024

TL;DR

This paper tackles the growing challenge of detecting AI-generated text by introducing a novel multimodal approach that combines traditional language models with numerical text features. Their optimized ensemble strategy significantly boosts detection accuracy, outperforming many other systems in the SemEval-2024 competition.

What Problem Does This Paper Address?

The rapid advancement of AI text generators, like GPT-3 and its successors, has made it increasingly difficult to distinguish between human-written and machine-generated content. This poses significant challenges for maintaining the integrity of digital discourse, impacting areas such as content moderation, misinformation prevention, and academic integrity. Existing detection methods often struggle with the diversity of AI models and their evolving capabilities, necessitating more robust and adaptable solutions. This paper specifically addresses the need for effective detection mechanisms for English language texts, aiming to improve accuracy and generalization across various AI generators and text domains.

Methods

The core of the I2C-Huelva system lies in its innovative combination of several techniques:

  • Multimodal Models: This is a key innovation. Instead of relying solely on traditional NLP models (like RoBERTa, ELECTRA, BERT), they integrate numerical text features. These features include aspects like sentence count, lexical diversity, average sentence length, and grammatical errors. The idea is that these quantitative aspects offer a different, complementary signal to the semantic and syntactic information captured by language models.
  • Optimized Ensembles: The system doesn't rely on a single model. Instead, it builds an ensemble of various fine-tuned transformer models and multimodal models. They then employ a rigorous process of testing different voting mechanisms (majority voting, soft voting, etc.) to find the optimal combination of models and voting strategy that maximizes accuracy and robustness.
  • Training Data Optimization: To combat overfitting and improve generalization to unseen AI generators, the authors strategically limit the size of the training dataset. They systematically explored different sample sizes to find an optimal balance, discovering that smaller sample sizes often led to better robustness.
  • Numerical Feature Analysis: For the numerical features, they initially explored a Random Forest classifier, noting its potential for analyzing these aspects, though suggesting further exploration with deep learning models for future work.

Datasets

The study utilized a dataset comprising English texts from diverse sources, including:

  • Wikipedia (March 2022 version)
  • WikiHow
  • Reddit (ELI5)
  • arXiv
  • PeerRead

The machine-generated texts were produced using leading multilingual Large Language Models (LLMs) such as ChatGPT, textdavinci-003, LLaMa, FlanT5, Cohere, Dolly-v2, and BLOOMz. The dataset was specifically curated to ensure a rich variety of genres and styles. The authors focused on the English portion of this dataset for their experiments.

Key Insights and Ideas

  • Multimodality is Key: The integration of numerical text features alongside traditional language models significantly enhances detection capabilities. These features provide a distinct signal that helps differentiate human from machine-generated text, especially when stylistic similarities are high.
  • Ensemble Power: Combining diverse models (both transformer-based and multimodal) through optimized voting strategies is crucial for achieving robust performance. The ensemble approach captures a broader range of linguistic and stylistic patterns.
  • Less is More (Sometimes): Limiting the training data size proved to be an effective strategy for preventing overfitting and improving the model's ability to generalize to new, unseen AI generators. This is a counter-intuitive but important finding for building robust detectors.
  • Task-Specific Ensembles: The optimal ensemble composition varied between binary (Subtask A) and multilabel (Subtask B) classification, highlighting the need for tailored strategies based on the task's complexity. Binary classification benefited from a more diverse set of models.

Key Results

The I2C-Huelva system achieved strong results in the SemEval-2024 Task 8 competition:

  • Subtask A (Monolingual Classification): The system ranked 47th with an accuracy of 0.8079.
  • Subtask B (Multilabel Classification): The system ranked 18th with an accuracy of 0.789.

The paper also shows that multimodal models generally outperformed their non-multimodal counterparts, and the final ensemble model consistently achieved the highest accuracy in both subtasks.

Limitations and Future Work

  • Numerical Feature Exploration: While numerical features proved valuable, the authors suggest that future work could explore more advanced methods like deep learning and XGBoost for analyzing these features, rather than the initial Random Forest approach.
  • Generator Specificity: The system still struggled to consistently differentiate between texts produced by very similar generators, indicating a need for further refinement to capture subtle stylistic variations.
  • Dynamic Ensembles: The authors propose exploring dynamic ensemble configurations that can adapt to the specific characteristics of the text being analyzed.
  • Broader Linguistic Features: Extending multimodal frameworks to incorporate more advanced linguistic and semantic features is a promising avenue for future research.

Summary updated

BibTeX
@inproceedings{pena2024i,
  author = {Alberto Rodero Peña and Jacinto Mata Vazquez and Victoria Pachón Álvarez},
  title = {{I2C-Huelva at SemEval-2024 Task 8: Boosting AI-Generated Text Detection with Multimodal Models and Optimized Ensembles}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure