Werkzeug at SemEval-2024 Task 8: LLM-Generated Text Detection via Gated Mixture-of-Experts Fine-Tuning

Youlin Wu, Kaichun Wang, Kai Ma, Liang Yang, Hongfei LinNAACL 2024

TL;DR

This paper introduces Gated Mixture-of-Experts Fine-tuning (GMoEF), a novel method for detecting LLM-generated text. GMoEF tackles the challenges of text anisotropy and dynamic language styles by employing parametric whitening to normalize embeddings and a gated mixture-of-experts architecture to capture diverse linguistic features. The approach achieved a strong #8 ranking out of 70 teams in the SemEval-2024 Task 8.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) has led to text generation capabilities that are increasingly indistinguishable from human writing. While this is a boon for creativity and efficiency, it also presents a significant challenge: reliably detecting LLM-generated text. This is crucial for combating misinformation, ensuring academic integrity, and preventing misuse. Existing methods often rely on fine-tuning smaller language models, but these models can suffer from "anisotropy," where text embeddings cluster too closely, making differentiation difficult. Furthermore, LLMs can adapt their writing style dynamically, making it hard to find consistent patterns for detection.

Methods

The proposed GMoEF method builds upon the standard fine-tuning paradigm for LLM-generated text detection. The core innovations are:

  • Parametric Whitening (PW): To address the anisotropy issue in text embeddings, GMoEF applies a learnable linear transformation. Unlike traditional whitening techniques, GMoEF treats the mean and variance as learnable parameters, allowing for more adaptable normalization. This process aims to create more isotropic representations, where embeddings are spread out more evenly in the latent space.
  • Gated Mixture-of-Experts (MoE): To capture the dynamic and multifaceted nature of LLM-generated text, GMoEF employs an MoE architecture. Here, multiple parametric whitening layers act as "experts," each specializing in learning different aspects of the language style. A "gating router" dynamically assigns weights to these experts, allowing the model to aggregate their outputs and make a more robust decision. This ensemble approach helps in capturing a wider range of linguistic features.

The overall architecture involves using a Pre-trained Language Model (PLM) to encode text, followed by the parametric whitening and gated MoE layers, and finally a feed-forward network for classification.

Datasets

The experiments were conducted on a sampled version of the M4 dataset, which was provided by the SemEval-2024 Task 8 organizers. This dataset is designed for detecting machine-generated text across different scenarios. The paper specifically mentions experiments on:

  • Subtask A (monolingual and multilingual): Focused on detecting single-model generated text.
  • Subtask B: Focused on detecting multi-model generated text.

The paper notes that Subtask C had a different optimization target and was not addressed.

Key Insights and Ideas

  • Anisotropy is a real problem: The paper strongly argues that the anisotropy of text embeddings from standard PLMs is a significant hurdle for accurate LLM-generated text detection.
  • Dynamic styles require dynamic models: The ability of LLMs to mimic various styles necessitates models that can adapt and capture these nuances, which the MoE architecture aims to achieve.
  • [CLS] token is not always optimal: The authors found that using the [CLS] token for text embedding can be too coarse-grained for this task. Their proposed token position pooling strategy proved more effective.
  • Multilingual and multi-model scenarios are harder: The performance improvements of GMoEF were more pronounced in multilingual and multi-model detection tasks, suggesting that these complex scenarios exacerbate the anisotropy issue and benefit more from the proposed approach.

Key Results

GMoEF achieved an impressive #8 ranking out of 70 participating teams on the multi-model generated text detection subtask (Subtask B).

  • Overall Performance: GMoEF showed significant improvements over baseline models on Subtask A (multilingual) and Subtask B.
  • Ablation Studies: Experiments without parametric whitening (w/o PW) and using the [CLS] token (alt. PLM) resulted in immediate performance drops across all subtasks, validating the effectiveness of GMoEF's core components.
  • Number of Experts (k): The optimal number of experts was found to be k=3. Too few experts limited the model's ability to capture dynamic features, while too many experts could lead to conflicting conclusions and overfitting.

Limitations and Future Work

  • Subtask A (monolingual) performance: While GMoEF performed well overall, its advantage over baselines was less pronounced on the monolingual Subtask A. The authors suggest this might be due to the model being better suited for more complex scenarios.
  • Optimal checkpoint selection: For Subtask A (original submission), the authors noted a potential issue in identifying the optimal checkpoint due to a disparity between training and evaluation samples.
  • Future Directions: The authors plan to extend their observations to other text classification tasks and explore using LLMs themselves as detectors for machine-generated text.

Summary updated

BibTeX
@inproceedings{wu2024werkzeug,
  author = {Youlin Wu and Kaichun Wang and Kai Ma and Liang Yang and Hongfei Lin},
  title = {{Werkzeug at SemEval-2024 Task 8: LLM-Generated Text Detection via Gated Mixture-of-Experts Fine-Tuning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure