TL;DR
This paper presents QUST's approach to SemEval-2024 Task 8, focusing on detecting AI-generated text. They achieved 8th place in the multilingual subtask A by combining data augmentation, a Multiscale Positive-Unlabeled (MPU) framework with XLM-R, and a stacking ensemble. The study highlights the effectiveness of these techniques, especially for multilingual settings and short texts.
What Problem Does This Paper Address?
The rapid advancement of Large Language Models (LLMs) has made it increasingly difficult to distinguish between human-written and AI-generated text. This poses significant ethical concerns, particularly regarding misinformation and academic integrity. SemEval-2024 Task 8 aims to address this challenge by developing robust systems for AI-generated text detection, especially in multilingual contexts where linguistic differences and data scarcity exacerbate the problem. Existing multilingual models are less explored for this task compared to their monolingual counterparts, creating a gap in understanding their effectiveness.
Methods
The QUST team employed a multi-pronged strategy:
- Data Augmentation and Cleaning: To address data scarcity and improve quality, they used back-translation to increase monolingual data volume and performed extensive data cleaning to remove noise and irrelevant content. They also re-balanced the multi-class subtask B dataset.
- Multiscale Positive-Unlabeled (MPU) Framework: This framework, designed for short texts, was integrated with models like XLM-R to enhance detection performance. It combines a length-sensitive MPU loss with recurrent models and a text multi-scale module.
- Fine-tuning and Adapters: They explored traditional fine-tuning of pre-trained language models (PLMs) like DeBERTa. To improve parameter efficiency, they also investigated adapters, specifically LoRA (Low-Rank Adapter), which injects trainable matrices into Transformer layers while freezing PLM weights. However, they found adapters sometimes led to unstable performance and ultimately favored direct fine-tuning or MPU-enhanced models for the final submission.
- Stacking Ensemble: To further boost performance and generalization, they implemented a stacking ensemble. This involved using the predicted probabilities (logits) from the top two performing models as input for a linear layer to generate the final predictions.
Datasets
- M4 Dataset (Extended Version): The primary dataset used for SemEval-2024 Task 8, which is an extended version of a previous dataset.
- Monolingual and Multilingual Datasets: The task was divided into monolingual and multilingual tracks for subtask A, and subtask B had its own dataset.
- v1, v2, and v3 Datasets: The authors describe different versions of the dataset: v1 (original), v2 (after data augmentation and cleaning), and v3 (further processing after the official test set release).
Key Insights and Ideas
- MPU Framework's Strength for Short Texts: The MPU framework proved particularly effective for detecting AI-generated text, especially in scenarios with a predominance of short texts, as observed in the dataset.
- XLM-R for Multilingual Tasks: XLM-R, designed for multilingual capabilities, showed superior performance in multilingual settings compared to models like BERT, which are primarily optimized for monolingual tasks.
- Data Augmentation is Crucial: Back-translation and rigorous data cleaning significantly improved model performance, highlighting the importance of data quality and quantity.
- Stacking Ensemble Adds Value: Combining the predictions of top models through stacking provided a notable performance boost, demonstrating the benefit of model fusion.
- Adapter Instability: While adapters offer parameter efficiency, their performance can be unstable, and direct fine-tuning or MPU integration might be more reliable for achieving optimal results in certain complex tasks.
Key Results
- 8th Place in Multilingual Subtask A: The team achieved a strong 8th place ranking in the official multilingual test set for subtask A.
- Performance Improvements:
- MPU framework with RoBERTa-base improved performance by 20 percentage points over direct fine-tuning on the monolingual dataset (Mon1).
- The XLM-R model integrated with the MPU framework outperformed the baseline XLM-R model by 7% on the multilingual dataset.
- Stacking ensemble methods consistently improved accuracy, with one instance showing a nearly 7% improvement over the best single model.
- Subtask B Performance: The XLM-R-MPU model showed poorer results on subtask B, suggesting it might be less suited for multi-class tasks compared to binary classification. Adapter-based DeBERTa-v3-Large and RoBERTa-large models showed significant improvements on subtask B after data cleaning.
Limitations and Future Work
- Adapter Performance Instability: The authors noted that adapter-based fine-tuning could lead to unstable performance, and they ultimately did not use adapters in the final testing phase for some models.
- Subtask B Suitability: The XLM-R-MPU model's performance on subtask B was suboptimal, indicating potential limitations for multi-class classification tasks.
- Data Distribution Discrepancies: Differences in data distribution between pre-training and incremental training, or between processed and raw datasets, could lead to performance gaps.
- Future Work: The team plans to explore advancements in cross-lingual pre-trained models to further enhance their system's understanding of texts across diverse languages and cultural backgrounds.
Summary updated