GPT-generated Text Detection: Benchmark Dataset and Tensor-based Detection Method

Zubair Qazi, William Shiao, Evangelos E. Papalexakis

Introduction

The rapid proliferation of Large Language Models (LLMs) such as ChatGPT has led to an unprecedented surge in synthetic content across the internet. While these models offer significant utility, they also pose substantial risks to the integrity of online discourse, facilitating the spread of misinformation and automated spam. Detecting AI-generated text has become a critical priority, yet current detection methods face significant hurdles in robustness and data integrity.

This paper introduces two major contributions to the field: the GPT Reddit Dataset (GRiD), a meticulously curated benchmark for AI detection, and GpTen, a novel semi-supervised detection method that utilizes tensor decomposition to identify AI-generated text as a statistical anomaly.

The Challenges of AI Detection

Existing research into AI text detection is often hampered by two primary issues:

  1. Dataset Contamination: Many datasets used to train "human" text classifiers were collected after the public release of ChatGPT. Consequently, these datasets may inadvertently contain AI-generated content, blurring the lines between the classes the models are meant to distinguish.
  2. The "Cat-and-Mouse" Game: Most current detectors are fully supervised, meaning they require training on specific examples of both human and AI text. These models often fail when encountering text from newer LLMs (e.g., GPT-4 or Claude) because they have not been exposed to the specific stylistic nuances of those newer models during training.

The GRiD Dataset

To address the issue of contamination, the researchers developed the GPT Reddit Dataset (GRiD). This benchmark focuses on high-quality, informative prose sourced from three specific subreddits: AskHistorians, AskScience, and ExplainLikeImFive.

Data Collection and Integrity

To ensure a "clean" human baseline, the authors only collected Reddit posts submitted before November 2022, the launch date of ChatGPT. This temporal cutoff guarantees that the human samples are genuinely human-authored. For the AI component, the researchers used the titles of these Reddit posts as prompts for gpt-3.5-turbo to generate matching responses.

Preprocessing

The final dataset consists of 6,513 samples (5,145 human and 1,368 GPT). To ensure the models learned linguistic structure rather than superficial patterns, the authors meticulously removed "tells" such as links, special markdown characters, and profanity—elements that GPT models are typically programmed to avoid or that differ significantly in frequency between human and AI text. Furthermore, GPT responses were limited to 100 tokens to match typical Reddit comment lengths, preventing length bias from influencing the detectors.

GpTen: Tensor-Based Anomaly Detection

The core technical innovation of this work is GpTen, a semi-supervised approach that treats AI text detection as an anomaly detection problem. Unlike supervised classifiers, GpTen only needs to learn the "fingerprint" of human writing.

Methodology

  1. Tensor Construction: For every document, the system creates a word co-occurrence matrix, tracking the proximity of terms. These matrices are stacked into a 3D tensor (Terms × Terms × Documents).
  2. Decomposition: The researchers apply Canonical Polyadic Decomposition (CPD) to break the tensor into latent factors that represent the underlying structural patterns of human language.
  3. Reconstruction Error: When a new text sample is evaluated, the model attempts to "reconstruct" it using only the patterns learned from the human training data. The reconstruction error is measured via the Frobenius norm.
  4. Anomaly Scoring: If the reconstruction error is high, it indicates that the text does not follow the learned human linguistic "rules." Kernel Density Estimation (KDE) is then used to model the distribution of these errors and flag outliers as AI-generated.

Experimental Results

The researchers benchmarked GpTen against common supervised baselines using 10-fold cross-validation on the GRiD dataset.

Model F1 Score AUC
BERT (Supervised) 0.934 0.984
SVM (Supervised) 0.813 0.845
Random Forest (Supervised) 0.787 0.825
GpTen (Semi-supervised) 0.667 0.708

While the supervised BERT model achieved the highest raw accuracy, GpTen’s performance is notable because it achieved a respectable F1 score of 0.667 without ever seeing a single example of AI-generated text during training. This "zero-knowledge" approach suggests that GpTen may be more robust when facing "out-of-distribution" AI models that have not yet been released.

Conclusion and Future Work

The introduction of the GRiD dataset provides a clean, uncontaminated benchmark for the research community. While supervised models like BERT currently lead in performance on known distributions, the GpTen method offers a promising path toward more generalized detection. By focusing on what makes text "human" rather than trying to keep pace with every new AI model, semi-supervised tensor methods could provide a more sustainable solution to the evolving challenges of synthetic content detection. Future work will focus on closing the performance gap between semi-supervised and deep learning methods and exploring the impact of adversarial prompt engineering.

Summary updated

BibTeX
@inproceedings{qazi2024gpt,
  author = {Zubair Qazi and William Shiao and Evangelos E. Papalexakis},
  title = {{GPT-generated Text Detection: Benchmark Dataset and Tensor-based Detection Method}},
  booktitle = {The Web Conference},
  year = {2024},
  url = {https://openreview.net/forum?id=fTfobKXxEG},
}
Figure