Catching Chameleons: Detecting Evolving Disinformation Generated using Large Language Models

Bohan Jiang, Chengshuai Zhao, Zhen Tan, Huan Liu

Abstract

As Large Language Models (LLMs) rapidly evolve, the disinformation they generate becomes a "moving target" for detection systems. Static models often become obsolete as new generators emerge, and standard fine-tuning methods frequently suffer from catastrophic forgetting—losing the ability to detect older disinformation patterns while learning new ones. This paper introduces DELD (Detecting Evolving LLM-generated Disinformation), a parameter-efficient framework that utilizes concatenated "soft prompts" to learn the unique fingerprints of new LLMs sequentially. By keeping the backbone model frozen and only updating small prompt vectors, DELD preserves knowledge of previous generators while effectively adapting to new ones, significantly outperforming traditional fine-tuning approaches.

Introduction

The "post-ChatGPT" era has fundamentally altered the landscape of digital misinformation. LLMs can now generate highly credible, context-aware fake news at an unprecedented scale. However, the primary challenge for defenders is not just the volume of content, but its evolution. New models like GPT-4, LLaMA-3, and Claude—along with their numerous fine-tuned variants—emerge constantly, each with distinct linguistic signatures.

Existing detection methods face two critical hurdles:

  1. Computational Inefficiency: Retraining a full detector for every new LLM is resource-intensive and slow.
  2. Catastrophic Forgetting: When a single model is sequentially trained on different LLM outputs (e.g., learning to detect GPT-2, then LLaMA, then Vicuna), it often loses the patterns associated with earlier models, leading to a sharp decline in overall robustness.

This research addresses the "evolving disinformation" problem by proposing a framework that accumulates knowledge over time rather than replacing it.

Methodology: The DELD Framework

The authors propose DELD, a model-agnostic framework built upon frozen Pre-trained Language Models (PLMs) such as BERT or T5. The core innovation lies in the use of Parameter-Efficient Fine-Tuning (PEFT) via Soft Prompts.

Soft Prompt Concatenation

Instead of updating the millions of parameters within the backbone PLM, DELD trains a unique "soft prompt" for every new disinformation generator encountered. A soft prompt is a small set of trainable vectors prepended to the input text.

As the system encounters new LLMs, the specific soft prompts are concatenated sequentially. This architecture allows the model to "remember" the characteristics of previous generators while learning the nuances of the current one. Because the core language model remains frozen, the risk of catastrophic forgetting is virtually eliminated. Only the small prompt vectors and the final classification layer are updated during the learning phase.

Optimal Configuration

Through rigorous experimentation, the authors identified two key structural insights:

  • Prepending vs. Appending: Placing soft prompts at the beginning of the news article is significantly more effective than placing them at the end, suggesting the prompt acts as a specialized "lens" for the model.
  • Prompt Length: A length of 12 tokens was found to be the "sweet spot," providing sufficient context for detection without unnecessary computational overhead.

Experimental Setup and Datasets

To simulate a real-world evolving landscape, the study utilized a hybrid dataset:

  • Human Baseline: Sourced from the Welfake dataset to represent traditional disinformation.
  • AI-Generated Content: Fake news generated specifically for this study using ChatGPT (GPT-4), LLaMA, and Vicuna.

The final experimental set included approximately 4,002 true news articles and 3,970 disinformation pieces across these four categories.

Key Insights and Results

Linguistic Fingerprints

Analysis revealed that different LLMs leave distinct "fingerprints" on their output. For instance, ChatGPT-generated disinformation often utilizes technical and medical jargon to project authority, whereas Vicuna tends toward dynamic, action-oriented descriptions. DELD successfully captures these stylistic nuances through its soft prompts.

Performance Metrics

DELD was tested against several baselines, including zero-shot detection and standard sequential fine-tuning (FT-Seq). The results highlight a significant performance gap:

  • Accuracy: In sequential scenarios, standard fine-tuning on BERT achieved 73.40% average accuracy. DELD improved this to 85.19%. When using T5 as the backbone, DELD reached an impressive 90.44% average accuracy.
  • Mitigating Forgetting: Using a "Forgetting" (Fgt) metric, the authors demonstrated that DELD maintained near-zero or even negative forgetting (indicating improved performance on old tasks), while standard models saw forgetting scores as high as 15.7.
  • Forward Transfer: Interestingly, the sequential concatenation sometimes improved performance on new models, suggesting that the detector learns generalizable features of "machine-ness" that apply across different LLMs.

Limitations and Future Directions

While DELD represents a major advancement, the authors acknowledge several limitations:

  • Identity Dependency: The framework currently requires knowledge of which LLM generated the text during training to assign the correct soft prompt. Future iterations may need to automate this identification.
  • Multimodal Content: The current study is limited to text. As disinformation evolves to include AI-generated images and video, detection frameworks must become multimodal.
  • Adversarial Resilience: Further research is needed to determine how DELD handles "jailbroken" models or LLMs specifically prompted to mimic human writing styles to evade detection.

Conclusion

DELD offers a scalable, efficient, and robust solution to the problem of evolving AI-generated disinformation. By moving away from full-model retraining and toward parameter-efficient knowledge accumulation, it provides a sustainable defense mechanism for an information environment in constant flux.

Summary updated

BibTeX
@inproceedings{jiang2024catching,
  author = {Bohan Jiang and Chengshuai Zhao and Zhen Tan and Huan Liu},
  title = {{Catching Chameleons: Detecting Evolving Disinformation Generated using Large Language Models}},
  booktitle = {CogMI},
  year = {2024},
  doi = {10.1109/cogmi62246.2024.00034},
}
Figure