CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learning

Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Yu Lan, Chao Shen

Abstract: Machine-Generated Text (MGT) detection, a task that discriminates MGT from Human-Written Text (HWT), plays a crucial role in preventing misuse of text generative models, which excel in mimicking human writing style recently. Latest proposed detectors usually take coarse text sequence as input and output some good results by fine-tune pretrained models with standard cross-entropy loss. However, these methods fail to consider the linguistic aspect of text (e.g., coherence) and sentence-level structures. Moreover, they lack the ability to handle the low-resource problem which could often happen in practice considering the enormous amount of textual data online. In this paper, we present a coherence-based contrastive learning model named CoCo to detect the possible MGT under low-resource scenario. Inspired by the distinctiveness and permanence properties of linguistic feature, we represent text as a coherence graph to capture its entity consistency, which is further encoded by the pretrained model and graph neural network. To tackle the challenges of data limitations, we employ a contrastive learning framework and propose an improved contrastive loss for making full use of hard negative samples in training stage. The experiment results on two public datasets prove our approach outperforms the state-of-art methods significantly.

1. Introduction

As Large Language Models (LLMs) become increasingly adept at mimicking human prose, the risk of their misuse for fake news, spam, and academic dishonesty grows. Most current detectors treat text as a "flat" sequence of tokens, ignoring the high-level linguistic structure and logical flow—what researchers call coherence. Furthermore, high-performing detectors typically require massive amounts of labeled data, which is often unavailable in real-world "low-resource" scenarios.

We introduce CoCo (Coherence-Enhanced Contrastive Learning), a framework that tackles these challenges by modeling the "linguistic glue" of a document through entity-based graphs and utilizing a specialized contrastive learning loss to maximize efficiency with limited data.

2. Methodology

CoCo addresses the limitations of token-based detection by explicitly modeling document structure and using advanced learning objectives.

2.1 Coherence Graph Construction

Inspired by Centering Theory, CoCo represents text as a coherence graph. It identifies entities (people, places, things) as nodes, with edges representing their co-occurrence within or across sentences. This creates a map of how a story "hangs together." As illustrated in Figure 1, human-written text naturally maintains entity consistency in a way that AI often struggles to replicate perfectly over long passages, leading to distinct interaction patterns.

Comparison of coherence patterns in human-written vs. machine-generated text.

2.2 The CoCo Framework

The overall architecture (Figure 2) consists of a Coherence Encoder Module (CEM) that processes the entity graph using a Relation-Aware Graph Convolutional Network (GCN) and an Attention-based LSTM to track coherence evolution. This representation is fused with standard semantic features from RoBERTa. To handle low-resource constraints, the model uses a supervised contrastive learning framework with a dynamic memory bank and an improved contrastive loss that forces the model to focus on "hard negatives"—the samples most difficult to distinguish.

The CoCo framework architecture featuring the Coherence Encoder Module and Contrastive Learning pipeline.

3. Key Results and Insights

  • Superiority in Low-Resource Settings: When trained on only 500 examples, CoCo outperformed state-of-the-art methods by 1.23% to 2.83% in Accuracy and F1-Score across the GROVER and GPT-2 datasets.
  • The GPT-3.5 Paradox: Counter-intuitively, the authors found that text from newer, more advanced models like GPT-3.5 is actually easier to detect than that of their predecessors. This is hypothesized to be because advanced LLMs produce "too perfect" and predictable patterns that act as a statistical signature.
  • Robustness: CoCo maintained its performance even when text was perturbed (e.g., random token deletion or replacement), proving that it relies on structural features rather than just specific keywords.

4. Conclusion

CoCo demonstrates that structural linguistic features are powerful discriminators for MGT detection, especially when data is scarce. By explicitly modeling coherence, the detector gains a structural perspective that token-based models miss. While generating coherence graphs adds computational overhead, the gain in detection accuracy and robustness marks a significant step forward in identifying increasingly sophisticated AI-generated content.

Summary updated

Abstract

Machine-Generated Text (MGT) detection, a task that discriminates MGT from Human-Written Text (HWT), plays a crucial role in preventing misuse of text generative models, which excel in mimicking human writing style recently. Latest proposed detectors usually take coarse text sequence as input and output some good results by fine-tune pretrained models with standard cross-entropy loss. However, these methods fail to consider the linguistic aspect of text (e.g., coherence) and sentence-level structures. Moreover, they lack the ability to handle the low-resource problem which could often happen in practice considering the enormous amount of textual data online. In this paper, we present a coherence-based contrastive learning model named CoCo to detect the possible MGT under low-resource scenario. Inspired by the distinctiveness and permanence properties of linguistic feature, we represent text as a coherence graph to capture its entity consistency, which is further encoded by the pretrained model and graph neural network. To tackle the challenges of data limitations, we employ a contrastive learning framework and propose an improved contrastive loss for making full use of hard negative samples in training stage. The experiment results on two public datasets prove our approach outperforms the state-of-art methods significantly.
BibTeX
@misc{liu2022coco,
  author = {Xiaoming Liu and Zhaohan Zhang and Yichen Wang and Yu Lan and Chao Shen},
  title = {{CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learning}},
  year = {2022},
  eprint = {2212.10341},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.10341},
}
Figure