Can Backdoor Attacks Survive Time-Varying Models?

Huiying Li, Arjun Nitin Bhagoji, Ben Y. Zhao, Haitao Zheng

Abstract

Existing research on training-time attacks for deep neural networks (DNNs), such as backdoors, largely assume that models are static once trained, and hidden backdoors trained into models remain active indefinitely. In practice, models are rarely static but evolve continuously to address distribution drifts in the underlying data. This paper explores the behavior of backdoor attacks in time-varying models, whose model weights are continually updated via fine-tuning to adapt to data drifts. Our theoretical analysis shows how fine-tuning with fresh data progressively "erases" the injected backdoors, and our empirical study illustrates how quickly a time-varying model "forgets" backdoors under a variety of training and attack settings. We also show that novel fine-tuning strategies using smart learning rates can significantly accelerate backdoor forgetting. Finally, we discuss the need for new backdoor defenses that target time-varying models specifically.

Introduction: The Static Model Assumption

In AI security, backdoor attacks represent a significant threat. An attacker poisons a small subset of training data, causing the model to behave normally on clean inputs but misclassify inputs containing a specific "trigger." Traditionally, the community has assumed that once a backdoor is successfully injected, it remains a permanent vulnerability. However, real-world production models are dynamic; they undergo continuous fine-tuning to account for data drift—changes in environmental factors like lighting or camera angles.

Visualizing common backdoor attacks: Original images (e.g., a cat) are modified with triggers (Badnets, Blend, Wanet) to be misclassified as a target label (e.g., airplane).

This research investigates the "shelf life" of these attacks. If a model is updated periodically with fresh, clean data, does the backdoor eventually degrade? The authors propose that the evolution of models acts as a natural, albeit unintentional, defense mechanism.

Methodology

The study employs a two-pronged approach to evaluate backdoor survivability in evolving systems:

  1. Theoretical Framework: The authors provide a mathematical analysis of backdoor behavior during periodic fine-tuning. Using strongly convex loss functions as a model, they prove that fine-tuning with clean data acts as a forgetting mechanism, shifting model weights away from the poisoned state toward a benign state.
  2. Empirical Evaluation: Using standard benchmarks (MNIST and CIFAR-10) modified to simulate data drift (e.g., rotating digits or changing image hue), the researchers measured backdoor survivability—the number of update rounds an attack remains effective (Attack Success Rate > 50%) after poisoning ceases. They tested various architectures including ResNet-9, ResNet-18, and DenseNet-121.
  3. Accelerated Forgetting: To enhance this natural decay, the authors introduced the use of Slanted Triangular Learning Rates (STLR). This strategy ramps up the learning rate quickly before decaying it, helping the model escape the local minima associated with the backdoor features.

Key Results and Insights

The empirical results confirm that backdoors are indeed perishable. In standard fine-tuning scenarios on CIFAR-10, backdoors typically lose their effectiveness within 5 to 10 update rounds.

Empirical results showing the decay of backdoor Attack Success Rate (ASR) over successive model update events for MNIST and CIFAR-10 datasets.

Several critical insights emerged from the data:

  • Injection Timing Matters: Backdoors injected during the initial "from-scratch" training are significantly more resilient than those introduced later during fine-tuning. The former are more deeply integrated into the model's feature extraction layers.
  • Stealth vs. Survivability: Counter-intuitively, less stealthy attacks (those with more obvious triggers) tend to survive longer. These attacks create more distinct decision manifolds that require more clean data to overwrite.
  • STLR as a Defense: The use of Slanted Triangular Learning Rates proved exceptionally effective, often reducing backdoor survivability to zero in a single update round while maintaining high classification accuracy (91-99%).

Discussion and Future Work

The findings suggest that current backdoor defenses, such as Neural Cleanse or TABOR, are ill-equipped for time-varying models. These tools often rely on detecting 100% attack success rates; as backdoors naturally degrade, they become harder to detect even while remaining dangerous.

While this study highlights a natural defense, it assumes "one-shot" poisoning. If an attacker can persistently poison the incoming data stream used for fine-tuning, the backdoor could be maintained indefinitely. Future research should expand these findings to other domains, such as Large Language Models (LLMs) and Reinforcement Learning, where model evolution is equally prevalent.

Summary updated

Abstract

Existing research on training-time attacks for deep neural networks (DNNs), such as backdoors, largely assume that models are static once trained, and hidden backdoors trained into models remain active indefinitely. In practice, models are rarely static but evolve continuously to address distribution drifts in the underlying data. This paper explores the behavior of backdoor attacks in time-varying models, whose model weights are continually updated via fine-tuning to adapt to data drifts. Our theoretical analysis shows how fine-tuning with fresh data progressively "erases" the injected backdoors, and our empirical study illustrates how quickly a time-varying model "forgets" backdoors under a variety of training and attack settings. We also show that novel fine-tuning strategies using smart learning rates can significantly accelerate backdoor forgetting. Finally, we discuss the need for new backdoor defenses that target time-varying models specifically.
BibTeX
@misc{li2022can,
  author = {Huiying Li and Arjun Nitin Bhagoji and Ben Y. Zhao and Haitao Zheng},
  title = {{Can Backdoor Attacks Survive Time-Varying Models?}},
  year = {2022},
  eprint = {2206.04677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.04677},
}
Figure