On the Permanence of Backdoors in Evolving Models

Huiying Li, Arjun Nitin Bhagoji, Yuxin Chen, Hai-Tao Zheng, Ben Y. Zhao

Abstract

Existing research on training-time attacks for deep neural networks (DNNs), such as backdoors, largely assume that models are static once trained, and hidden backdoors trained into models remain active indefinitely. In practice, models are rarely static but evolve continuously to address distribution drifts in the underlying data. This paper explores the behavior of backdoor attacks in time-varying models, whose model weights are continually updated via fine-tuning to adapt to data drifts. Our theoretical analysis shows how fine-tuning with fresh data progressively "erases" the injected backdoors, and our empirical study illustrates how quickly a time-varying model "forgets" backdoors under a variety of training and attack settings. We also show that novel fine-tuning strategies using smart learning rates can significantly accelerate backdoor forgetting. Finally, we discuss the need for new backdoor defenses that target time-varying models specifically.

Introduction: The Static Model Fallacy

Most security research on Deep Neural Networks (DNNs) assumes a "train once, deploy forever" lifecycle. In this static view, a backdoor injected during training remains active indefinitely, acting as a permanent "sleeper agent." However, real-world models are rarely static; they are continuously updated through fine-tuning with new data to account for distribution drift—the natural tendency for data to change over time due to sensor aging, environmental shifts, or evolving user behavior.

This paper challenges the assumption of backdoor permanence by asking: How long can a backdoor survive in a model that is constantly evolving?

Methodology and Experimental Setup

The researchers simulated a "time-varying" environment where a ResNet-9 model is updated over 15 rounds. To ensure realism, they introduced controlled data drifts in each round, such as rotating digits in MNIST or shifting color properties in CIFAR-10.

They evaluated three prominent backdoor attacks, ranging from visible to invisible triggers:

  • Badnets: Uses a visible patch (trigger) to misclassify images.
  • Blend: Blends the trigger into the image, making it less perceptible.
  • Wanet: Uses warping-based transformations to create invisible triggers.
Visualizing the three backdoor attack types: Badnets (visible patch), Blend (faded trigger), and Wanet (warping).

Key Insights: The Forgetting Phenomenon

1. Backdoors are Not Permanent

Theoretical analysis and empirical simulations reveal that fine-tuning on clean data acts as a natural "cleansing" process. As the model learns new, legitimate features to adapt to distribution drift, it progressively overwrites the malicious triggers. The authors define $\gamma$-survivability as the number of update cycles a backdoor remains effective after poisoning stops.

2. Natural Decay of Attack Success Rate (ASR)

Without any specialized defense, most one-shot backdoors see their effectiveness plummet within just a few update cycles. The model essentially "forgets" the malicious behavior as it prioritizes the new, clean data distribution.

The Attack Success Rate (ASR) for Badnets, Blend, and Wanet decays rapidly over successive model update events in both MNIST and CIFAR-10 datasets.

3. The Learning Rate Lever and STLR

The speed of forgetting is highly dependent on the learning rate. Higher learning rates accelerate the erasure of backdoors but can compromise overall accuracy. To solve this, the authors propose using Slanted Triangular Learning Rates (STLR). This strategy rapidly increases the learning rate before decaying it, which was found to reduce backdoor survivability to zero in a single update cycle while maintaining high classification accuracy (99% for MNIST, 91% for CIFAR-10).

The Defense Paradox

An ironic finding of the study is that as a backdoor weakens through fine-tuning, it becomes harder for existing defense tools (like Neural Cleanse) to detect. These tools are designed to identify strong anomalies; when a backdoor is partially forgotten, it blends into the model's natural noise, remaining dormant but potentially dangerous if the specific trigger is encountered before full erasure.

Conclusion

Backdoor attacks are surprisingly fragile in the face of model evolution. While "one-shot" poisoning is easily mitigated by natural fine-tuning or optimized schedulers like STLR, the study highlights a new frontier for security: persistent poisoning, where an attacker continuously injects small amounts of poison into every update cycle. Future defense mechanisms must move beyond static detection and account for the dynamic, time-varying nature of modern AI deployments.

Summary updated

Abstract

Existing research on training-time attacks for deep neural networks (DNNs), such as backdoors, largely assume that models are static once trained, and hidden backdoors trained into models remain active indefinitely. In practice, models are rarely static but evolve continuously to address distribution drifts in the underlying data. This paper explores the behavior of backdoor attacks in time-varying models, whose model weights are continually updated via fine-tuning to adapt to data drifts. Our theoretical analysis shows how fine-tuning with fresh data progressively "erases" the injected backdoors, and our empirical study illustrates how quickly a time-varying model "forgets" backdoors under a variety of training and attack settings. We also show that novel fine-tuning strategies using smart learning rates can significantly accelerate backdoor forgetting. Finally, we discuss the need for new backdoor defenses that target time-varying models specifically.
BibTeX
@misc{li2022permanence,
  author = {Huiying Li and Arjun Nitin Bhagoji and Yuxin Chen and Hai-Tao Zheng and Ben Y. Zhao},
  title = {{On the Permanence of Backdoors in Evolving Models}},
  year = {2022},
  eprint = {2206.04677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.04677},
}
Figure