Introduction
The rapid advancement of Large Language Models (LLMs) like ChatGPT has introduced significant challenges to academic integrity in computer science education. Students can now generate functional, well-commented solutions for complex programming assignments in seconds. While AI detectors exist for natural language, they often struggle with source code due to its unique structural properties: high repetitiveness, strict syntax, and a lack of the stylistic "flow" found in standard prose. This paper introduces AIGCode, a detector designed to exploit "code naturalness"—the specific patterns of complexity and "burstiness" that distinguish human-written code from AI-generated output.
The AIGCode Framework
The AIGCode Detector operates through a two-stage process: Targeted Perturbation and Unified Scoring.
Targeted Masking Perturbation
Unlike previous methods that mask tokens at random, AIGCode utilizes line-level Perplexity (PPL) to guide the perturbation. Perplexity measures how "surprised" a language model is by a specific sequence. AIGCode identifies lines with high perplexity—representing idiosyncratic or complex human logic—and applies more intense masking to these regions. A fine-tuned CodeBERT model then fills these masks to create a set of perturbed variants.
Unified Scoring and Ranking
The detector evaluates the original code against its perturbed versions using three key metrics:
- Overall Perplexity: AI-generated code typically exhibits very low perplexity as it follows the model's most probable paths.
- Standard Deviation of PPL: Human code is "jittery," with spikes in complexity, whereas AI code is consistently predictable line-by-line.
- Burstiness: This captures the clustering of specific patterns or identifiers.
The core hypothesis is that AI-generated code sits at a local "pinnacle" of probability. Consequently, any perturbation is likely to increase its perplexity. In contrast, human-written code, being less "optimal" from the perspective of an LLM, may actually see its perplexity decrease after a "lucky" modification by the mask-filling model.
Experimental Setup
To evaluate the framework, the authors developed the AIGCode Dataset, a specialized benchmark derived from IBM's CodeNet. The dataset includes 5,214 AI-generated solutions (produced by OpenAI’s text-davinci-003) and 5,214 human-written solutions across six major programming languages: C, C++, C#, Java, JavaScript, and Python.
Results and Discussion
AIGCode demonstrates significant improvements over existing zero-shot detectors. As shown in the performance comparison, the proposed method (labeled as CGCode Detector in experimental results) consistently outperforms baselines like GPTZero and DetectGPT across all tested languages.
Key findings include:
- Superior Accuracy: AIGCode achieved an average AUC of 0.87, compared to 0.56 for GPTZero and 0.51 for DetectGPT.
- Language Sensitivity: The detector performed exceptionally well on C code, reaching an AUC of 0.95.
- Efficiency: By focusing perturbations on high-perplexity areas, the system requires only 50 perturbed samples to reach a decision, a ten-fold reduction compared to the 500 samples required by DetectGPT.
Limitations and Future Work
Despite its success, AIGCode faces challenges. The detector's performance is somewhat dependent on the model used to calculate perplexity; a significant mismatch between the generation and detection models can impact accuracy. Furthermore, the tool is vulnerable to "Code Blending"—the mixing of human and AI segments—which increased the bypass rate to 0.75 in adversarial testing. Future research will focus on developing supervised deep learning models to recognize perplexity patterns directly from embeddings, potentially eliminating the need for the computationally expensive perturbation step.
Summary updated