TL;DR
This paper introduces GPT-who, a novel, psycholinguistically-inspired detector for machine-generated text. By leveraging the Uniform Information Density (UID) principle, GPT-who analyzes how information is spread within text to distinguish between human and AI-generated content, achieving state-of-the-art performance with significantly lower computational cost than existing methods.
What Problem Does This Paper Address?
The rapid advancement and widespread adoption of Large Language Models (LLMs) have raised concerns about their potential for misuse, including the generation of misinformation, toxic content, and academic dishonesty. Consequently, there's a growing need for reliable methods to detect machine-generated text. While many detectors exist, they often rely on complex, black-box deep learning models that are computationally expensive and lack interpretability. This paper addresses the need for a more efficient, interpretable, and domain-agnostic detector that can effectively distinguish between human and LLM-generated text, even from sophisticated models.
Methods
GPT-who is built upon the Uniform Information Density (UID) hypothesis, which posits that humans tend to distribute information evenly in their language production to optimize comprehension. The core idea is that LLMs might exhibit different patterns of information distribution compared to humans.
The method involves:
- Surprisal Calculation: Using a pre-trained language model (GPT2-XL), the surprisal (or unexpectedness) of each token in a given text is calculated.
- UID Feature Extraction: Several features are derived from these surprisal values to capture the "smoothness" or "evenness" of information distribution. These include:
- Mean Surprisal: The average surprisal of all tokens.
- UID Variance: The normalized variance of surprisal across the text.
- UID Difference: The average absolute difference in surprisal between consecutive tokens.
- UID Difference²: The average squared difference in surprisal between consecutive tokens.
- Max/Min UID Spans: Features derived from the most and least uniform segments of the text, identified using a sliding window approach.
- Classification: These UID-based features (a fixed-length vector of 44 features) are then fed into a simple Logistic Regression classifier to predict authorship.
This approach is novel because it's the first to apply UID principles specifically for machine-generated text detection, using a psycholinguistic theory to inform feature engineering. It avoids the need for LLM fine-tuning, making it computationally inexpensive and interpretable.
Datasets
The authors evaluated GPT-who on four large-scale benchmark datasets:
- TuringBench: A multi-class authorship attribution dataset with over 168k news articles from 19 neural text generators.
- GPABenchmark (GPT Corpus for Academia): A multi-domain dataset of academic articles (CS, HSS, PHX) with 150k human and 450k ChatGPT-generated articles.
- ArguGPT: A dataset of argumentative essays with over 4k human-written essays and 4k articles generated by 7 recent LLMs.
- InTheWild: A large dataset of over 447k human-written and machine-generated texts across 10 tasks, using 27 recent LLMs.
Key Insights and Ideas
- UID as a Distinguishing Feature: The UID principle, when operationalized through surprisal distributions, effectively captures subtle differences between human and machine-generated text that are not apparent at the lexical or semantic level.
- LLM Families Share UID Signatures: Models belonging to the same LLM family (e.g., GPT-2 variants) tend to exhibit similar UID distributions, suggesting that architectural similarities influence information distribution patterns.
- Human Text is More Non-Uniform: Generally, human-written text shows higher UID variance, indicating more fluctuations and less uniform information spread compared to machine-generated text.
- Interpretability through Features: The UID-based features provide a more interpretable understanding of detection decisions, unlike black-box models. Figure 2 visually demonstrates how different UID spans can reveal distinct information density patterns.
- Efficiency is Key: GPT-who's reliance on pre-computed surprisal and a simple classifier makes it significantly faster and less resource-intensive than fine-tuned transformer models.
Key Results
- State-of-the-Art Performance: GPT-who outperforms existing statistical and non-statistical detectors (GLTR, GPTZero, DetectGPT, OpenAI detector, ZeroGPT) by over 20% across the evaluated datasets.
- Competitive with Fine-tuned Models: GPT-who achieves performance comparable to fine-tuned transformer-based models on several benchmarks, despite not requiring any LLM fine-tuning.
- Generalization: The detector shows good generalization capabilities, performing well even on out-of-distribution datasets.
- Computational Efficiency: GPT-who is significantly faster, with an inference time of 0.8 seconds per text sample compared to DetectGPT's 60 seconds, and requires minimal one-time training.
Limitations and Future Work
- Approximation of Human Distribution: The authors acknowledge that using GPT-2's probability distribution to calculate surprisal might be a poor approximation of the true "human" probability distribution. A closer approximation could potentially reveal different UID patterns.
- Scope of Analysis: The study focused on text detection and did not explore whether UID-based methods could identify other issues like misinformation or plagiarism.
- Resource Constraints: Limitations in publicly available datasets and computational resources restricted the exploration of a wider array of models and tasks.
Future work could involve exploring more sophisticated approximations of human probability distributions, extending the UID analysis to other text generation tasks, and investigating its applicability to detecting other forms of text manipulation.
Summary updated