OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution

Lucio La Cava, Andrea TagarelliEMNLP 2025

TL;DR

This paper introduces OpenTuringBench, a comprehensive benchmark and framework for detecting and attributing machine-generated text (MGT) from open large language models (OLLMs). It addresses the growing challenge of distinguishing increasingly sophisticated AI-generated content by providing a diverse dataset and a novel detector, OTBDetector, which outperforms existing methods.

What Problem Does This Paper Address?

The rapid advancement and widespread availability of generative AI, particularly Open Large Language Models (OLLMs), have led to an unprecedented surge in machine-generated text (MGT) across the internet. This influx of MGT poses significant threats to information reliability, credibility, and originality, potentially leading to misinformation, plagiarism, and the contamination of future AI training data. While MGT detectors are crucial for mitigating these risks, existing benchmarks and detectors struggle to keep pace with the evolving capabilities of OLLMs. Specifically, the diversity in OLLM architectures and training data makes MGT detection, especially authorship attribution, more challenging. This paper aims to bridge this gap by introducing a robust benchmark and a high-performing detector specifically designed for OLLM-generated content.

Methods

The core of this work lies in the creation of OpenTuringBench, a novel benchmark designed to train and evaluate MGT detectors. It features a diverse set of OLLMs and includes challenging evaluation tasks that go beyond simple detection, encompassing scenarios like human/machine-manipulated texts, out-of-domain texts, and texts from previously unseen models.

The authors also developed OTBDetector, a contrastive learning framework. This detector leverages a Longformer model as an encoder, chosen for its ability to handle the longer sequences typical of OLLM-generated texts. OTBDetector employs a triplet network architecture to learn a latent similarity space where texts from the same generator are clustered together, while texts from different generators are pushed apart. This approach allows for both MGT detection (Turing Test) and authorship attribution.

Datasets

The primary dataset introduced is OpenTuringBench, comprising over 543,000 texts. This dataset was created using a representative set of OLLMs (Llama3, Gemma2, Qwen, Mistral, Phi3, NeuralChat, and SOLAR) prompted to generate news articles based on headlines from the News Category dataset (spanning 2012-2022). The News Category dataset was chosen for its broad knowledge spectrum and its temporal cut-off, ensuring the human-authored data predates advanced generative AI.

OpenTuringBench includes several evaluation tasks:

  • In-Domain (ID) tasks: Evaluating on news articles from the same source or variations in prompts.
  • Out-of-Distribution (OOD) tasks: Testing on texts from different domains (e.g., essays) or generated by previously unseen models.

Key Insights and Ideas

  • OLLMs are making MGT detection harder: The increased diversity and accessibility of OLLMs present a more complex landscape for MGT detection compared to closed models.
  • Authorship Attribution is the real challenge: While Turing Test (distinguishing human from machine) is becoming relatively easy due to inherent linguistic differences, attributing MGT to a specific generator is significantly more difficult, especially among different OLLMs.
  • Novel evaluation tasks are crucial: The introduction of tasks like human-machine mixing, out-of-domain texts, and previously unseen models highlights the limitations of current detectors in generalizing to real-world scenarios.
  • Contrastive learning is effective for attribution: OTBDetector's success demonstrates the power of contrastive learning in creating a discriminative latent space for authorship attribution.
  • Longer texts require specialized models: The choice of Longformer for OTBDetector underscores the need for models capable of handling longer text sequences generated by modern LLMs.

Key Results

  • OTBDetector consistently outperformed 9 competing detectors across various tasks, achieving an F1 score of 0.996 for the in-domain authorship attribution task.
  • For the Turing Test, most detectors achieved high performance (F1 scores exceeding 0.9), with OTBDetector and LM-D performing exceptionally well.
  • In more challenging tasks like human-machine mixing and out-of-domain essays, OTBDetector still maintained superior performance, though scores dropped significantly for all detectors. For instance, in the essay domain (E5), OTBDetector achieved 0.51 F1, outperforming others by a substantial margin.
  • The study revealed that increasing generation temperature and model size had varying impacts on detector performance, with OTBDetector showing resilience.

Limitations and Future Work

  • Discourse Domains: The current benchmark focuses on news articles. The authors acknowledge the importance of expanding to other domains, such as creative writing, to enhance diversity and specialization.
  • Language: The benchmark is currently limited to English texts. Future work will involve extending it to support multilingual tasks.
  • Continual Learning: Integrating new LLMs requires retraining the system. Developing continual contrastive learning frameworks is suggested to improve scalability and adaptability.
  • Ethical Implications: The authors highlight the difficulty in detecting MGT, which could be exploited for malicious purposes. They urge caution and responsibility in the deployment of these technologies.
  • Broader Impact: While providing a robust benchmark, the research may inadvertently reveal limitations that could be exploited. The authors stress the importance of responsible and ethical use of AI technologies.

Summary updated

BibTeX
@inproceedings{cava2025openturingbench,
  author = {Lucio La Cava and Andrea Tagarelli},
  title = {{OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure