Unibuc - NLP at "Voight-Kampff" Generative AI Detection PAN 2025

Teodor Marchitan, Claudiu Creanga, Liviu P. DinuarXiv 2026 · arXiv

TL;DR

The Unibuc-NLP team tackled the challenge of distinguishing human-authored text from AI-generated and AI-edited content. Their findings reveal that for simple binary detection, an ensemble of classical machine learning models running on frozen LLM embeddings performs nearly as well as full fine-tuning, but identifying the nuances of human-AI collaboration remains a significantly harder, unsolved problem.

What Problem Does This Paper Address?

As generative AI becomes ubiquitous, the line between human and machine writing has blurred. The "Voight-Kampff" task (a nod to the replicant-detecting test in Blade Runner) at PAN 2025 aims to develop systems that can not only spot purely AI-generated text but also identify "machine-altered" content.

The authors address two specific challenges:

  1. AI Detection Sensitivity: A binary classification task to determine if a text is human-written or AI-written.
  2. Human-AI Collaborative Text Classification: A more complex multi-class problem where the goal is to categorize text into six levels of collaboration, ranging from "fully human-written" to "deeply-mixed" or "machine-humanized" (obfuscated) text.

Methods

The team explored three distinct architectural strategies:

  • Frozen LLM Embeddings + Classical ML: For Subtask 1, they extracted the last-layer embeddings from a small, efficient model (Qwen3-0.6B) and used them as features for an ensemble of classical classifiers: LightGBM, XGBoost, Logistic Regression, and SVM. The final prediction was determined by a majority vote.
  • LLM Fine-tuning: They also tested a more resource-intensive approach by fine-tuning larger models (Llama-3.1-8B, Mistral-7B, and Qwen2.5-0.5B) using QLoRA. They added a classification head on top of the last token to perform the downstream task.
  • Layer-wise Aggregation & Attentive Pooling: For the collaborative task (Subtask 2), they used a RoBERTa-base backbone. Instead of just using the final layer, they implemented a "Layer Aggregation Block" that computes a learnable weighted sum of multiple hidden layers. This was followed by "Attentive Statistical Pooling" (ASP) to summarize the sequence of embeddings into a fixed-size representation for classification.

Datasets

The researchers utilized several datasets provided by the PAN 2025 organizers:

  • Subtask 1: A dataset of ~27,000 samples across essays, news, and fiction. AI texts were generated by models including GPT, Mistral, Gemini, Llama, and DeepSeek.
  • Subtask 2: A much larger dataset (~360,000 samples) covering six collaboration categories. It drew from sources like LLM-DetectAIve, TriBERT, m4gt-bench, and RoFT.

Table 1: Label distribution for Subtask 1 Table 2: Label distribution for Subtask 2

Key Insights and Ideas

  • The "Good Enough" Threshold: One of the most practical takeaways is that classical ML models (like Logistic Regression or SVM) trained on frozen embeddings are remarkably competitive. They achieved an F1 score of ~0.993, which is nearly identical to the 0.997 achieved by fine-tuning a massive Llama-3.1-8B model. For many real-world applications, the speed and lower resource requirements of the classical approach make it the clear winner.
  • Fiction is the "Final Boss": Error analysis showed that the models struggled most with AI-generated fiction (specifically from Gemini 1.5 Pro and Llama 3.1). In embedding space, AI-generated fiction overlaps significantly with human-written fiction, making it the hardest genre to police.
  • The Generalization Gap: The models performed well on data similar to their training sets but struggled with "unseen" data. For example, performance dropped significantly on the m4gt-bench dataset and texts generated by Llama 2, neither of which were present in the training split. This suggests that current detectors are often "memorizing" the specific artifacts of known AI models rather than learning a universal signature of machine writing.

Key Results

For binary detection (Subtask 1), the results were exceptionally high across the board, with the ensemble method proving both robust and efficient.

Table 3: Performance of classical ML models Table 4: Performance of fine-tuned LLMs

However, Subtask 2 proved far more difficult. The RoBERTa-based model achieved a modest Macro F1 of 0.427 on the test set. The difficulty lies in the subtle differences between categories like "human-written, then machine-polished" versus "machine-written, then human-edited."

Limitations and Future Work

The authors acknowledge that their current systems struggle to generalize to generative models they haven't seen before. They suggest that future work should move beyond raw text analysis and incorporate:

  • Metadata and Context: Using information about the document's origin or authorial intent.
  • Adversarial Training: Specifically training models on "obfuscated" text designed to fool detectors.
  • Interpretability: Developing "error-driven" refinements to understand why a model thinks a text is AI-generated, which is crucial for high-stakes applications like journalism or academia.

Summary updated

Abstract

This paper describes the approach of the Unibuc - NLP team in tackling the "Voight-Kampff" Generative AI Detection task from PAN at CLEF 2025. In the Subtask 1: Voight-Kampff AI Detection Sensitivity we have explored different approaches of leveraging embeddings extracted from LLMs, using either classical ML models or further fine-tuning LLMs with a classification head for the downstream binary classification task. Our experiments showed that using a majority vote ensemble of classical ML models (Light GBM, XGB, Logistic Regression and SVM) is comparable with fine-tuning for downstream task achieving around 99.4% F1 score. For the Subtask 2: Human-AI Collaborative Text Classification we explored a way of combining features at different layers extracted using transformer backbone with layer-wise projection and attentive pooling which proved modest results.
BibTeX
@misc{marchitan2026unibuc,
  author = {Teodor Marchitan and Claudiu Creanga and Liviu P. Dinu},
  title = {{Unibuc - NLP at "Voight-Kampff" Generative AI Detection PAN 2025}},
  year = {2026},
  eprint = {2611.16612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16612},
}
Figure