TL;DR
This paper tackles the challenge of distinguishing between human-written and AI-generated text across multiple languages and domains. The Groningen team explored fine-tuning large language models (LLMs) like DeBERTa and found that generating additional training data, particularly for underrepresented languages, significantly improved their multilingual model's performance.
What Problem Does This Paper Address?
The rapid advancement of Large Language Models (LLMs) has made it increasingly difficult to discern between human-generated and machine-generated text. This poses significant challenges in various applications, from academic integrity to combating misinformation. SemEval-2024 Task 8 specifically addresses this by focusing on "Multidomain Machine-Generated Text Detection," requiring systems to not only identify AI-generated text but also to do so across diverse domains, multiple languages, and even identify the specific LLM used for generation (Subtask B). The core problem is building robust models that can generalize well across these varied conditions, especially when faced with languages and LLMs not explicitly seen during training.
Methods
The Groningen team primarily employed fine-tuning pre-trained LLMs, with a strong focus on DeBERTa-v3 (large), as it's known to outperform its predecessors.
- Monolingual Subtask A: They experimented with both the base and large variants of DeBERTa, tuning hyperparameters like learning rate, batch size, and epochs. The large DeBERTa model generally performed better.
- Multilingual Subtask A: Instead of a single multilingual model, they opted for a combined approach. This involved using language identification (LangID) to determine the language of each text and then employing either specific monolingual models or a multilingual DeBERTa model for classification. They also explored generating additional training data to augment the existing datasets.
- Subtask B: Similar to Subtask A, they fine-tuned DeBERTa-v3 models. The goal here was more complex: not only detect machine-generated text but also identify the specific LLM used.
- Data Generation: A key methodological contribution was the generation of additional training data using Llama 2 (7B parameter version). They focused on underrepresented languages and aimed to create a balanced dataset of human and machine-generated text. They also generated Wikipedia articles in the style of the original dataset.
Datasets
The primary dataset used was provided by the SemEval-2024 Task 8 organizers, originating from the M4 benchmark. This dataset is comprehensive, covering machine-generated text from diverse generators, domains, and languages including English, Chinese, Russian, Urdu, Indonesian, Italian, German, and Arabic.
The paper also details the generation of additional training data using Llama 2, specifically targeting underrepresented languages to improve model generalization.
Key Insights and Ideas
- Data Augmentation is Crucial: Generating extra training data, especially for underrepresented languages, significantly boosted the performance of their multilingual model. This highlights the importance of diverse and balanced datasets for robust generalization.
- Multilingual Strategy Matters: A combined approach using language identification and tailored models (monolingual or multilingual) proved more effective for multilingual tasks than a single, monolithic multilingual model.
- LLM Bias: Their models showed a tendency to misclassify human-written text, suggesting a potential bias towards identifying text as machine-generated. This indicates that models might be learning the "style" of LLMs more effectively than the nuances of human writing.
- Llama 2 Performance: Even when using Llama 2-generated data, their models struggled with texts generated by Llama 2 itself, particularly when using the smaller 7B parameter version. This suggests that the specific characteristics of the generative model, even within the same family, can impact detection.
Key Results
- Subtask A Monolingual: Achieved 87.80% accuracy on the development set but dropped to 63.68% on the test set, indicating a generalization issue.
- Subtask A Multilingual: Reached 65.90% on the development set and improved to 71.79% on the test set with the addition of generated data. This was still below the baseline of 80.88%.
- Subtask B: Achieved 72.80% on the development set and 61.50% on the test set, struggling to accurately identify the specific language model used for generation.
- Impact of Added Data: The multilingual model saw a notable improvement in score after training on the extra generated data, while the monolingual model's improvement was negligible.
Limitations and Future Work
- Generalization Issues: The models struggled to generalize to unseen data, particularly on the test set for monolingual Subtask A.
- Bias Towards Machine-Generated Text: The models showed a tendency to misclassify human-written text, indicating a potential bias.
- Llama 2 Detection: Performance on texts generated by Llama 2 was poor, possibly due to using the smaller 7B parameter version for data generation.
- Future Work: The authors suggest exploring different model architectures, more data augmentation techniques, training in more languages, using larger pre-trained models (though resource-intensive), and focusing on creating richer, higher-quality training data to improve performance and generalization.
Summary updated