TL;DR
This paper introduces AIDER, a novel framework for detecting AI-generated text that significantly improves robustness against adversarial attacks and generalization to unseen topics. By combining an ALBERT-based detector with a sophisticated data augmentation module, AIDER achieves state-of-the-art performance in challenging detection scenarios.
What Problem Does This Paper Address?
The rapid advancement of large language models (LLMs) has made AI-generated text increasingly indistinguishable from human-written text. This poses significant challenges for academic integrity, journalistic authenticity, and combating misinformation. Existing AI-generated text detectors often struggle with two main issues: they are vulnerable to adversarial attacks (e.g., paraphrasing) that drastically reduce accuracy, and they lack the ability to generalize to topics not seen during training. This creates an "arms race" where detectors are constantly being outsmarted by evasion techniques, and their practical applicability is limited to specific domains.
Methods
AIDER is a two-component framework: a detector and an augmentor.
- Detector: Built on the lightweight ALBERT model, it employs a two-stage training process.
- Stage 1: Pre-training on a large, diverse corpus of human and AI texts to learn fundamental differences.
- Stage 2: Fine-tuning using a novel contrastive label prediction module and a topic disentanglement module.
- Topic Disentanglement Module: This module uses a Gradient Reversal Layer (GRL) and a Variational Information Bottleneck (VIB) to separate topic-specific features from generation-specific features. This is crucial for generalizing to unseen topics.
- Contrastive Label Prediction Module: This module uses triplet loss, cross-entropy loss, and a novel balance loss to ensure robust classification by making embeddings of the same class (human/AI) closer and embeddings of different classes farther apart.
- Augmentor: This module leverages LLMs (like GPT-4 Turbo) to generate challenging adversarial samples. It supports various attack types (paraphrasing, prompt-based, adversarial) and can perform recursive attacks. This augmented data is then used to train the detector, significantly enhancing its robustness.
Datasets
- Pre-training Dataset: An aggregated dataset of approximately 500,000 unpaired human-AI texts from publicly available sources like Deepfake, CHEAT, and MGTBench.
- AIGen Dataset: A newly introduced dataset comprising around 24,000 human-written, AI-generated, and AI-augmented triplets. It amalgamates data from existing datasets (OUTFOX, HC3, Daigt-v2) and human-written datasets (IMDB, SemEval, SubjQA, Writing-Prompts), covering diverse topics and generated by various LLMs and attack strategies.
Key Insights and Ideas
- Topic Disentanglement is Key for Generalization: By explicitly separating topic-related features from generation-specific ones, AIDER can effectively detect AI text across a wide range of topics, even those it hasn't seen during training.
- Augmentation as a Robustness Booster: The augmentor's ability to generate diverse and challenging adversarial samples is critical for making the detector resilient to evasion techniques. This creates a more robust model that can withstand sophisticated attacks.
- Synergy of Contrastive Learning and Topic Disentanglement: Combining these two approaches allows AIDER to learn representations that are both discriminative for AI/human text and invariant to topic, leading to superior performance.
- The "Arms Race" Can Be Addressed: AIDER demonstrates that a well-designed augmentation strategy can significantly mitigate the effectiveness of adversarial attacks, rather than just reacting to them.
Key Results
- Superior Robustness: AIDER achieves state-of-the-art performance on augmented datasets, significantly outperforming baselines like OUTFOX and DIPPER in terms of F1 score and Average Recall, especially under recursive attacks. For instance, on the DIPPER-augmented data, AIDER achieved the highest recall (82.9%) and second-best F1 (83.2%).
- Excellent Topic Generalization: AIDER consistently achieved the highest F1 scores (89.36% to 93.28%) and AvgRec across three unseen topics from the AIGen dataset, outperforming models like HC3 Detector, RoBERTa-base, and DetectGPT.
- Minimal Performance Degradation: While adversarial attacks do impact performance, AIDER shows minimal degradation compared to other methods, particularly under OUTFOX augmentation.
- Effective Augmentation: The AIGen dataset, created using AIDER's augmentor, proved to be more challenging than the OUTFOX dataset, highlighting the effectiveness of the augmentation strategy.
Limitations and Future Work
- Vulnerability to New Generators: AIDER's generalization capability for texts generated by entirely new, out-of-distribution models (not encountered during training) has not been fully addressed.
- Data Shift in Human Texts: The study used historical human texts, which might introduce a temporal data shift compared to contemporary human writing, potentially impacting performance on very recent human-generated content.
- Future Work:
- Refining data augmentation strategies by integrating more sophisticated attack techniques.
- Exploring advanced attack models.
- Investigating cross-lingual detection capabilities.
- Incorporating a wider range of target generators (GPTs, GLMs, Claudes) to enhance applicability.
Summary updated