A Practical Examination of AI-Generated Text Detectors for Large Language Models

Brian Tufts, Xuandong Zhao, Lei Li

Abstract

As Large Language Models (LLMs) proliferate, the ability to distinguish machine-generated text from human writing has become a societal necessity. This paper presents a rigorous "black-box" evaluation of seven leading AI detectors across diverse models, tasks, and languages. Our findings suggest that industry-standard metrics like AUROC provide a misleading sense of security. When evaluated at a realistic 1% false positive rate ([email protected]), detector performance drops significantly, especially in multilingual settings and adversarial "rewriting" scenarios where LLMs are used to polish human drafts.

Introduction: The Need for Robust Detection

The rapid adoption of LLMs has created an urgent need for reliable AI-generated text detectors to combat plagiarism, misinformation, and bot-driven interference. However, most existing evaluations suffer from "home-field advantage," testing detectors on the same distributions they were trained on. This study addresses this gap by conducting an out-of-distribution evaluation of prominent detection tools across seven diverse datasets, including code, abstracts, and multilingual dialogue.

Methodology

The researchers evaluated seven detectors, categorized into "Trained Detectors" (e.g., RADAR, T5Sentinel) and "Zero-Shot Detectors" (e.g., Binoculars, Fast-DetectGPT). These were tested against four diverse LLMs: Llama-3-8B, Mistral-v0.3, Phi-3-Mini, and GPT-4o.

The experimental pipeline for evaluating AI-generated text detectors across prompts, models, and human-generated baselines.

The evaluation utilized three distinct prompting strategies: plain prompting, adversarial prompting (instructing the model to "sound human"), and rewriting (using the LLM to polish human-written drafts).

The AUROC Mirage vs. Real-World Utility

A central contribution of this work is the critique of AUROC (Area Under the Receiver Operating Characteristic) as a primary success metric. While a detector might achieve an AUROC of 0.85, its True Positive Rate at a 1% False Positive Rate ([email protected]) might be as low as 0.10. In high-stakes environments like education, where false accusations are unacceptable, the [email protected] is a far more relevant measure of utility. The study found that many tools boasting high AUROC scores failed to catch even 50% of AI text when tuned to this realistic 1% false-alarm threshold.

Key Findings: Evasion and Multilingual Challenges

The study revealed several critical vulnerabilities in current detection technology:

  1. Rewriting as the Ultimate Evasion: When an LLM is used to "rewrite" or polish human text, it retains enough human-like structural "DNA" to bypass most statistical detectors. This scenario represents a significant challenge for detecting "human-in-the-loop" AI usage.
  2. The Multilingual Cliff: Detection performance drops precipitously for non-English text. For many detectors, the [email protected] on tasks in Spanish, French, or Chinese was effectively zero, suggesting these tools are currently English-centric.
  3. Task Sensitivity: Detection success is highly dependent on the nature of the task. Dialogue and reviews are easier to detect, while code and translations—which are naturally more constrained—make AI and human output look statistically identical.
Average [email protected] of detectors for the 'Rewrite' task, illustrating the significant drop in detection performance across various domains and languages.

Conclusion

The results serve as a stark warning: AI detectors are not yet robust enough for universal deployment. Performance is highly sensitive to the specific task and language, and simple adversarial techniques like rewriting can easily defeat them. Future work must focus on improving robustness in "human-in-the-loop" scenarios and expanding capabilities beyond English to ensure these tools can meet the societal needs they were designed for.

Summary updated

BibTeX
@inproceedings{tufts2025practical,
  author = {Brian Tufts and Xuandong Zhao and Lei Li},
  title = {{A Practical Examination of AI-Generated Text Detectors for Large Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.271},
}
Figure