Team Advacheck at PAN: Multitasking Does All the Magic

Anastasia Voznyuk, German Gritsai, Andrey V. GrabovoyCLEF (Working Notes) 2025

TL;DR

This paper introduces a multi-task learning (MTL) approach for detecting machine-generated text, outperforming single-task models by leveraging shared representations across related tasks. Their Advacheck system achieved near-perfect performance on a binary AI detection task and secured third place in a more complex human-AI collaborative text classification task, demonstrating the power of MTL for robust AI text detection.

What Problem Does This Paper Address?

The proliferation of large language models (LLMs) has led to an explosion of machine-generated text online. While LLMs are incredibly useful, their ability to produce human-like text raises concerns about potential misuse, such as academic dishonesty or the spread of misinformation. Reliably detecting AI-generated content is crucial, but current methods often struggle when faced with variations in domains, generator models, or subtle human edits that obscure the AI's origin. This paper tackles the challenge of building robust AI text detectors that can generalize across these diverse and often noisy real-world scenarios.

Methods

The core of the Advacheck system is a multi-task learning (MTL) architecture built upon a shared Transformer encoder. This approach allows the model to learn a more generalized representation by training on multiple related tasks simultaneously, which helps to mitigate overfitting and improve robustness. Specifically, they employed a hard parameter sharing (HPS) strategy where a common encoder is used for all tasks, feeding into separate custom classification heads (CCHs) for each subtask.

For Task 1 (binary AI detection), they used a binary CCH. To further enhance representations, they also incorporated two additional multiclass CCHs: one for genre classification and another for determining the model family that generated the text. This allowed the model to learn richer, domain-aware features.

For Task 2 (human-AI collaborative text classification), a more complex multiclass CCH was used, designed to distinguish between six novel categories of human-AI text interaction. The training process involved two phases: first, fine-tuning the classification heads with a frozen encoder, and then fine-tuning the entire model. This staged approach helps to guide the encoder's weights effectively.

Datasets

The research utilizes datasets from the PAN 2025 workshop, specifically for the Voight-Kampff Generative AI Detection and Human-AI Collaborative Text Classification tasks. The paper mentions that the training data for Task 1 included information on genres and model families, which were leveraged for their multi-task approach. For Task 2, the dataset comprised six novel classes representing different forms of human-AI text collaboration.

Key Insights and Ideas

  • Multi-task learning is a magic bullet for AI text detection: The paper strongly advocates for MTL, showing it significantly improves performance and robustness compared to single-task models, especially in challenging cross-domain and cross-generator scenarios.
  • Domain and model family information aids detection: By training auxiliary tasks like genre and model family classification, the model learns more nuanced representations that are better at distinguishing AI-generated text, even when it's obfuscated.
  • MTL creates structured representations: The authors observed that training on multiple tasks naturally leads to a cluster-like structure in the text embeddings, where texts from similar domains or families group together. This structure is key to handling noisy data.
  • Robustness to "noisy" data: The proposed MTL approach demonstrates effectiveness even with the presence of a large amount of noisy data, a common issue in real-world text detection.

Key Results

  • Task 1 (Binary AI Detection): The Advacheck system achieved an impressive 99.95% mean metric on the validation set, significantly outperforming baselines.
  • Task 2 (Human-AI Collaborative Text Classification): The system secured third place in the official ranking with a 60.85% macro F1-score on the test set, surpassing the baseline by 13%.

Limitations and Future Work

The paper does not explicitly detail limitations or future work beyond the general statement that the MTL approach helps achieve high results despite noisy data. However, the reliance on specific model families and genres for Task 1 suggests that performance might be sensitive to the availability and quality of such metadata in future datasets. Future work could explore more sophisticated methods for unsupervised domain adaptation or investigate how to further improve generalization to entirely unseen AI models and writing styles.

Summary updated

BibTeX
@inproceedings{voznyuk2025team,
  author = {Anastasia Voznyuk and German Gritsai and Andrey V. Grabovoy},
  title = {{Team Advacheck at PAN: Multitasking Does All the Magic}},
  booktitle = {CLEF (Working Notes)},
  year = {2025},
}
Figure