Team ap-team at PAN: LLM Adapters for Various Datasets

Galina Boeva, German Gritsai, Andrey GrabovoyCLEF (Working Notes) 2024 · OpenReview

Researchers from Advacheck and Université Grenoble Alpes have introduced LAVA (LLM Adapters for Various dAtasets), a modular framework designed to detect machine-generated text. By leveraging specialized QLoRA adapters trained on distinct "families" of AI-generated data, the team successfully addressed the "forgetting" effect inherent in large-scale model training, achieving state-of-the-art results in the PAN 2024 authorship verification challenge.

The Challenge: Catastrophic Forgetting in Detection

As Large Language Models (LLMs) evolve, the difficulty of distinguishing human prose from synthetic text increases. Traditional binary classifiers often suffer from the "forgetting" effect: when a single model is trained on a massive, heterogeneous dataset, it may lose the ability to recognize patterns from earlier training phases. Furthermore, the computational cost of retraining a monolithic detector every time a new generative model is released is prohibitive.

The LAVA Methodology

The LAVA approach shifts from a single-model paradigm to a modular ensemble. The system is built on four pillars:

  1. Base Model: Mistral-v0.2 serves as the foundational model due to its robust performance across NLP benchmarks.
  2. QLoRA Adapters: Instead of full fine-tuning, the team utilized Quantized Low-Rank Adaptation (QLoRA). This allows for the training of lightweight adapters—small sets of additional parameters—while keeping the base model weights frozen.
  3. Specialization: Three separate adapters were trained, each specialized on a specific "family" of generative models (GPT family, Llama/Mistral family, and a diverse "Others" family).
  4. Aggregation Logic: The system employs an "OR" logic gate during inference. A text is classified as human only if all three adapters agree; if any single adapter flags the text as machine-generated, the final label is AI. This design prioritizes high sensitivity to synthetic content.
The QLoRA architecture allows for training lightweight adapters (A and B) while keeping pretrained weights (W0) frozen.

Experimental Setup and Datasets

To train the specialized adapters, the authors curated a massive collection of open-source data, divided into three distinct families:

  • The GPT Family: Texts from GPT-3, GPT-3.5, and GPT-4.
  • The Llama/Mistral Family: Texts from Llama-2 and Mistral-v0.1/v0.2.
  • The "Others" Family: A mix of models including Vicuna, OPT, BLOOM, Alpaca, and Gemini Pro.

This division allows each adapter to capture the specific nuances and statistical distributions unique to different model architectures.

Key Results

The LAVA ensemble (submitted as "marinated-pantone") demonstrated exceptional performance in the PAN 2024 Voight-Kampff Generative AI Authorship Verification competition.

Performance comparison showing that the LAVA ensemble (marinated-pantone) significantly outperforms the single-adapter approach (ferocious-coot) and most baselines.

As shown in the results table, the ensemble approach achieved a ROC-AUC of 0.967, significantly outperforming the single-adapter variant ("ferocious-coot," 0.937) and nearly all official baselines, including Fast-DetectGPT. Across nine test set variants, the model maintained a median accuracy of 0.942, proving that the aggregation of specialized adapters provides a clear and consistent performance boost.

Conclusion and Future Directions

The LAVA framework demonstrates that a "divide and conquer" strategy effectively mitigates the forgetting effect in LLM-based detectors. The modular architecture is inherently future-proof; as new generative models emerge, new adapters can be integrated into the ensemble without retraining existing components.

Future work will focus on more sophisticated aggregation methods beyond simple "OR" logic, exploring weighted voting systems and testing the framework across a broader range of base models like Llama-3 or Phi-3 to further enhance detection accuracy and adversarial robustness.

Summary updated

BibTeX
@inproceedings{boeva2024team,
  author = {Galina Boeva and German Gritsai and Andrey Grabovoy},
  title = {{Team ap-team at PAN: LLM Adapters for Various Datasets}},
  booktitle = {CLEF (Working Notes)},
  year = {2024},
  url = {https://openreview.net/forum?id=1cyZhEtd45},
}
Figure