Trace Is In Sentences: Unbiased Lightweight ChatGPT-Generated Text Detector

Mo Mu, Dianqiao Lei, Chang LiarXiv 2025 · DOI · arXiv

Introduction: The Fragility of Current Detectors

As Large Language Models (LLMs) like ChatGPT become ubiquitous, the need for reliable detection of AI-generated text has grown. However, current detectors are surprisingly fragile. Most rely on word-level patterns—specific vocabulary choices or statistical metrics like perplexity. This approach suffers from two major flaws: sensitivity to paraphrasing and inherent spurious biases (content or style bias).

Word-level modifications through prompting can easily bypass traditional detectors.

Simple "Prompt-based Polishing" (PSP) or manual editing can break word-level detectors. If a user swaps a few adjectives, the "AI signature" often vanishes. Researchers from Tsinghua University argue that while words are easy to change, the underlying logical structure and inter-sentence relationships of AI-generated text are much harder to mask. They propose TRACE, a framework that captures these invariant structural "traces."

The TRACE Framework

The TRACE model shifts the focus from "what words are used" to "how sentences relate to one another." It treats text detection as a sentence-level relation task rather than a token-classification task.

The TRACE architecture overview, showing the progression from raw text to debiased classification.

Architecture and Causal Reasoning

The pipeline consists of four main stages:

  1. Text Preprocessing: The input text is split into a sequence of sentences $S = \{S_1, S_2, \dots, S_n\}$.
  2. Sentence Embedding Generation: A pre-trained RoBERTa-Base model generates high-dimensional embeddings for each sentence.
  3. Learning Inter-sentence Structure: These embeddings are fed into a 3-layer Transformer encoder. Using Multi-Head Attention, the model captures how sentences transition, contrast, or follow one another.
  4. Debiased Classification Output: To ensure the model doesn't rely on superficial keywords, TRACE employs a Structural Causal Model (SCM) and counterfactual learning.

By identifying confounders like word-level patterns and topic content, the authors use counterfactual interventions—such as swapping synonyms (Natural Indirect Effect) or varying topics (Direct Effect)—to force the model to ignore superficial features and focus on the invariant structure.

Experimental Evaluation

TRACE was evaluated across diverse datasets, including the Human ChatGPT Comparison Corpus (HC3), a new Arxiv abstract-comparison dataset, and massive multi-domain FAQ sets covering Finance, Medicine, and Law.

Robustness to Adversarial Attacks

The model was specifically tested against "Cyclical Translation" and "Any Alteration" (combined modifications). On English datasets, while a standard RoBERTa detector's accuracy dropped to 88.64% under modification, TRACE maintained a high accuracy of 98.36%.

Cross-Domain Generalization

In specialized fields, TRACE consistently outperformed baselines by 5-10% in F1-score. This suggests that structural "traces" are a universal feature of LLM outputs, regardless of the specific domain or language (the method proved effective for both English and Chinese).

Conclusion

TRACE demonstrates that the core "trace" of AI generation lies in sentence structure rather than individual words. By operating at the sentence level, the model is not only more robust to paraphrasing but also significantly more lightweight and faster to deploy than massive LLM-based detectors. Future work will explore more complex structural cues and expand the benchmark to further the development of paraphrase-proof detection systems.

Summary updated

Abstract

The widespread adoption of ChatGPT has raised concerns about its misuse, highlighting the need for robust detection of AI-generated text. Current word-level detectors are vulnerable to paraphrasing or simple prompts (PSP), suffer from biases induced by ChatGPT's word-level patterns (CWP) and training data content, degrade on modified text, and often require large models or online LLM interaction. To tackle these issues, we introduce a novel task to detect both original and PSP-modified AI-generated texts, and propose a lightweight framework that classifies texts based on their internal structure, which remains invariant under word-level changes. Our approach encodes sentence embeddings from pre-trained language models and models their relationships via attention. We employ contrastive learning to mitigate embedding biases from autoregressive generation and incorporate a causal graph with counterfactual methods to isolate structural features from topic-related biases. Experiments on two curated datasets, including abstract comparisons and revised life FAQs, validate the effectiveness of our method.
BibTeX
@misc{mu2025trace,
  author = {Mo Mu and Dianqiao Lei and Chang Li},
  title = {{Trace Is In Sentences: Unbiased Lightweight ChatGPT-Generated Text Detector}},
  year = {2025},
  eprint = {2509.18535},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.18535},
}
Figure