EnsemJudge: Enhancing Reliability in Chinese LLM-Generated Text Detection Through Diverse Model Ensembles

Zhuoshang Wang, Yubing Ren, Guoyu Zhao, Xiaowei Zhu, Hao Li, Yanan Cao

TL;DR

This paper introduces EnsemJudge, a novel ensemble framework for detecting LLM-generated Chinese text. It addresses the critical need for reliable detection in real-world scenarios, especially against adversarial attacks and out-of-domain data, achieving state-of-the-art performance on a challenging Chinese benchmark.

What Problem Does This Paper Address?

The rapid advancement and widespread adoption of Large Language Models (LLMs) have brought immense benefits, but also significant societal risks. The ability of LLMs to generate human-like text raises concerns about misuse, such as academic dishonesty, the spread of misinformation, and malicious content generation. Consequently, developing robust methods to detect LLM-generated text is crucial. While many detection methods exist, they often struggle with:

  • Generalization: Performance degrades significantly on out-of-domain data or when faced with adversarial attacks (e.g., paraphrasing, text excerpting).
  • Language Bias: Most existing research and evaluation are focused on English text, leaving the detection of LLM-generated Chinese text underexplored.
  • Robustness: Individual models, even those performing well on specific datasets, may not maintain reliability across diverse real-world scenarios.

This paper aims to tackle these challenges by proposing a framework that enhances the reliability and robustness of LLM-generated text detection, specifically for Chinese language.

Methods

EnsemJudge employs an ensemble-based approach to overcome the limitations of individual detection models. The core idea is to leverage the strengths of multiple diverse detection methods and combine their predictions intelligently. Key components of the framework include:

  • Base Models: A collection of existing LLM-generated text detection methods (rule-based, training-free, and training-based) are treated as independent "base models." The framework is designed to be modular, allowing for the integration of any binary classifier.
  • Strategy Assignment Module: This module analyzes lightweight textual features of an input text (e.g., length, perplexity) to cluster texts. For each cluster, a specific subset of base models is selected, and predefined voting weights are assigned to them. This creates tailored "strategies" for different types of text.
  • Decision Support Module: For samples where the ensemble model is uncertain (i.e., scores near the decision threshold), this module uses a large language model with in-context learning (ICL) to provide a "support signal" that helps refine the final prediction. This is particularly useful for handling challenging cases like back-translated samples.
  • Dynamic Voting Module: This module combines the predictions of the selected base models using the assigned weights. It incorporates the support signal from the Decision Support Module to make a final prediction.

The framework dynamically assigns integration strategies and voting weights based on text characteristics, aiming to optimize overall system performance by capitalizing on the diverse strengths of the base models.

Datasets

The study utilizes the dataset provided by the NLPCC2025 Shared Task 1. This dataset is officially in Chinese and includes:

  • Training Set: Derived from the DetectRL-ZH benchmark, designed for detecting Chinese LLM-generated text.
  • Development Set: Used for initial system development and evaluation.
  • Test Set: Contains entirely out-of-domain data in terms of content and LLM source. It is further divided into scenarios for:
  • Normal: Standard LLM-generated and human-written text.
  • Attack: Text subjected to adversarial transformations like paraphrasing and perturbation.
  • Varying Length: Texts of different lengths, including "extreme short," "short," and "general" categories.

The test set's out-of-domain nature and the unknown construction methodology before evaluation highlight the challenge of achieving robust detection.

Key Insights and Ideas

  • Chinese LLM Detection is Underexplored: The paper highlights a significant gap in research concerning LLM-generated text detection for languages other than English, particularly Chinese.
  • Adversarial Attacks are a Major Threat: The study demonstrates that common adversarial strategies like back-translation and text excerpting can severely degrade the performance of many existing detection methods.
  • No Single Model is a Silver Bullet: Different detection methods excel on different types of text or under specific attack scenarios. This inherent specialization necessitates an ensemble approach.
  • Ensembling for Robustness: By dynamically combining diverse models and tailoring their contributions based on text characteristics and potential adversarial modifications, EnsemJudge achieves superior reliability and robustness compared to individual models.
  • Decision Support is Crucial for Uncertainty: The inclusion of a decision support module, leveraging LLMs, proves effective in handling ambiguous cases and improving overall detection accuracy.

Key Results

  • State-of-the-Art Performance: EnsemJudge achieved first place in the NLPCC2025 Shared Task 1, demonstrating its superior performance.
  • High Overall Reliability: The system achieved a macro F1 score of 0.9922 on the full test set, with improvements up to 9.0% on certain individual subsets compared to baseline methods.
  • Robustness to Adversarial Attacks: EnsemJudge showed strong generalization and robustness across various text types and attack settings, outperforming all baseline methods.
  • Rule-based Methods Ineffective: The paper found rule-based methods to be largely ineffective, with macro F1 scores not exceeding 0.6.
  • Training-free vs. Training-based: Training-free methods performed moderately better (around 0.8), while training-based methods showed the best results (up to 0.95) among individual models.
  • Impact of Text Length: Text length significantly impacts detection performance, with specialized models for short texts proving beneficial.

Limitations and Future Work

  • Computational Cost: While the framework is designed to be extensible, ensemble methods can inherently increase computational requirements. The paper does not explicitly detail the computational overhead of EnsemJudge.
  • Hyperparameter Tuning: The performance of the ensemble is sensitive to the selection of base models, strategy assignment, and voting weights, which require careful tuning. The paper provides hyperparameter settings in Table 2, but the process of finding these optimal values is complex.
  • Future Work: The authors suggest that future research could explore more sophisticated ensemble strategies, investigate the impact of different LLM architectures on detection, and extend the framework to other low-resource languages. The paper also notes that the "Decisive Adjustment Module" was omitted in their experiments due to the ensemble's sufficient performance, implying it could be explored further for specific scenarios.

Summary updated

Abstract

Large Language Models (LLMs) are widely applied across various domains due to their powerful text generation capabilities. While LLM-generated texts often resemble human-written ones, their misuse can lead to significant societal risks. Detecting such texts is an essential technique for mitigating LLM misuse, and many detection methods have shown promising results across different datasets. However, real-world scenarios often involve out-of-domain inputs or adversarial samples, which can affect the performance of detection methods to varying degrees. Furthermore, most existing research has focused on English texts, with limited work addressing Chinese text detection. In this study, we propose EnsemJudge, a robust framework for detecting Chinese LLM-generated text by incorporating tailored strategies and ensemble voting mechanisms. We trained and evaluated our system on a carefully constructed Chinese dataset provided by NLPCC2025 Shared Task 1. Our approach outperformed all baseline methods and achieved first place in the task, demonstrating its effectiveness and reliability in Chinese LLM-generated text detection. Our code is available at https://github.com/johnsonwangzs/MGT-Mini.
BibTeX
@inproceedings{wang2025ensemjudge,
  author = {Zhuoshang Wang and Yubing Ren and Guoyu Zhao and Xiaowei Zhu and Hao Li and Yanan Cao},
  title = {{EnsemJudge: Enhancing Reliability in Chinese LLM-Generated Text Detection Through Diverse Model Ensembles}},
  booktitle = {NLPCC},
  year = {2025},
  url = {https://openreview.net/forum?id=JMb7mBk1nW},
}
Figure