TL;DR
The Unibuc NLP team developed transformer-based and hybrid deep learning models for the SemEval-2024 Task 8, aiming to distinguish between human and machine-generated text across multiple domains and languages. Their transformer model achieved a strong second place in Subtask B, demonstrating its effectiveness, though overfitting remained a challenge for other subtasks.
What Problem Does This Paper Address?
The proliferation of AI-generated text, particularly with advanced models like GPT-3 and beyond, has made it increasingly difficult to discern human-written content from machine-generated output. This poses significant challenges for maintaining information integrity, preventing the spread of misinformation, and ensuring content accountability. SemEval-2024 Task 8 specifically targets this problem by challenging participants to build systems capable of detecting machine-generated text in a multilingual, multi-domain, and black-box setting, meaning the internal workings of the text generators are unknown.
Methods
The Unibuc NLP team explored two main architectural approaches:
- Transformer-based Models: These models leverage pre-trained transformers (like RoBERTa and mdeberta-v3) as feature extractors. The core idea is to pass the transformer's output through a series of fully connected layers for classification. Key challenges addressed include handling long texts (using truncation strategies like "head only") and optimizing fine-tuning to prevent overfitting. They experimented with different layer selection strategies, finding that using the last layer or the last few layers often yielded the best results. A "Head-First Fine-Tuning" (HeFit) strategy was employed, involving freezing transformer layers initially and then fine-tuning selected layers.
- Hybrid Deep Learning Models: For token-level classification (Subtask C), a hybrid approach was used. This involved combining character-level features extracted via a Convolutional Neural Network (CNN) with word embeddings. These combined features were then fed into a Bidirectional Long Short-Term Memory (BiLSTM) network, followed by fully connected layers. For Subtask C, they also experimented with adding a Conditional Random Field (CRF) layer on top of the fully connected layers to improve sequence tagging performance.
Datasets
The datasets used for this task were extensions of the M4 dataset. The paper notes that Subtask C had significantly less data compared to Subtasks A and B. The team also found that augmenting the dataset for Subtask A (monolingual) by including data from Subtask B and removing duplicates was beneficial.
Key Insights and Ideas
- Transformer Power: Transformer models, particularly when fine-tuned appropriately, are highly effective for machine-generated text detection, as evidenced by their strong performance in Subtask B.
- Overfitting is a Persistent Challenge: Despite employing strategies like freezing layers and careful fine-tuning, overfitting remained a significant issue, especially for Subtask A. This suggests that detecting subtle differences between human and AI text requires robust generalization capabilities.
- Truncation Matters: For handling long texts with transformers, the "head only" truncation strategy proved to be the most effective for their dataset.
- Layer Selection is Crucial: Experimentation showed that using the last layer or a few of the last layers of a transformer often captures the most relevant features for this task.
- Preprocessing Trade-offs: For their hybrid model, no text preprocessing (keeping text as-is) yielded the best results, highlighting the robustness of modern language models to raw text.
Key Results
- Subtask B: The Unibuc NLP team secured second place out of 77 teams in Subtask B, achieving an accuracy of 86.95%.
- Subtask A (Monolingual): Achieved an accuracy of 0.85. However, analysis revealed an under-prediction of the human-written class, suggesting a bias.
- Subtask A (Multilingual): Achieved a final score of 0.79. Similar to the monolingual track, the model showed signs of overfitting, with a bias towards predicting machine-generated text.
- Subtask C: The hybrid model overfitted drastically on this dataset, hindering its ability to detect transitions between human and machine-generated text.
Limitations and Future Work
- Overfitting: The authors acknowledge that overfitting was a primary challenge, particularly for Subtask A. They suggest that more time dedicated to preventing overfitting during fine-tuning is necessary.
- Limited Sequence Length: The transformer models' inherent limit of 512 tokens per sequence, combined with resource constraints, prevented training on longer sequences. They plan to explore extended sequence lengths (up to 1024 tokens) with more powerful hardware.
- Model Specialization: The team proposes an ensemble learning approach where specialized models are trained for text generated by specific AI models (e.g., ChatGPT, Cohere).
- LLM Exploration: They intend to investigate the use of large language models (LLMs) like Mistral/Mixtral or Solar, exploring both zero-shot and few-shot learning scenarios to leverage their in-context learning capabilities.
Summary updated