TL;DR
This paper presents AISPACE, a top-ranked system for detecting machine-generated text from multiple LLM sources. The system leverages fine-tuned transformer models, a class-balanced loss function to handle data imbalance, and a soft-voting ensemble for robust predictions, achieving state-of-the-art performance on the SemEval-2024 Task 8.
What Problem Does This Paper Address?
The rapid advancement of Large Language Models (LLMs) has led to impressive text generation capabilities, blurring the lines between human and machine-written content. This poses significant risks, including the spread of misinformation, academic dishonesty, and the potential degradation of future AI development due to data contamination. SemEval-2024 Task 8 specifically challenges researchers to detect machine-generated text from various LLM generators, across different domains and languages. This paper focuses on Subtask B, which involves classifying English text as either human-written or generated by a specific LLM, with an emphasis on generalization to unseen text sources.
Methods
The AISPACE system employs a multi-pronged approach:
- Fine-tuning Transformer Models: The core of the system involves fine-tuning various pre-trained transformer architectures, including encoder-only (e.g., Roberta, Deberta, Longformer), decoder-only (e.g., XLNet), and encoder-decoder (e.g., T5) models. This systematic study aimed to identify which architectures perform best for this multi-class classification task.
- Class-Balanced Weighted Loss: To address the significant imbalance in the dataset (human-written text being far more prevalent than text from any single LLM), a weighted Cross Entropy loss function was implemented. This ensures that the model doesn't become overly biased towards the majority class.
- Soft-Voting Ensemble: To enhance robustness and generalization, the system utilizes a soft-voting ensemble. This involves combining the probability distributions of multiple well-performing base models, rather than just their final predictions, to make a more reliable final decision.
Datasets
The primary dataset used is derived from SemEval-2024 Task 8, specifically Subtask B. To address data limitations, the training data from Subtask A was merged with Subtask B's training data, resulting in a unified dataset of 127,755 items. The original Subtask B development set, containing 3,000 items, was used for evaluation. Notably, the merged training data did not include texts generated by the BLOOMZ model, allowing for an assessment of the model's generalization ability to unseen generators.
Key Insights and Ideas
- Encoder-only models excel: The study found that encoder-only transformer architectures (like Roberta and Deberta) performed exceptionally well for this multi-class detection task, outperforming decoder-only and encoder-decoder models.
- Data augmentation is crucial: Merging data from Subtask A significantly improved performance, highlighting the importance of a larger and more diverse training set for this task.
- Weighted loss is effective: The class-balanced weighted loss function was essential for mitigating the impact of data imbalance, leading to better performance across all classes.
- Ensembling boosts reliability: The soft-voting ensemble strategy, combining the confidence scores of multiple models, proved more effective than relying on a single best model, leading to improved overall accuracy and robustness.
- Input length matters, but not infinitely: For Longformer, using an input size of 512 tokens was found to be more effective than 1024 tokens, suggesting that while longer contexts are important, excessively long inputs might not always yield better results for this specific task.
Key Results
The AISPACE system achieved an accuracy of 99.46% on the development set, securing the 1st rank in SemEval-2024 Task 8, Subtask B. This performance sets a new state-of-the-art benchmark for detecting machine-generated text from multiple LLM sources. The paper also details ablation studies showing the significant contributions of data merging, weighted loss, and model ensembling.
Limitations and Future Work
The authors acknowledge that the training data did not include texts generated by the BLOOMZ model, which was present in the development set. This means the model's generalization to this specific generator was tested implicitly. Future work includes exploring a wider range of LLMs, investigating few-shot learning, and developing transfer-learning capabilities for similar detection tasks.
Summary updated