BeDKD: Backdoor Defense based on Dynamic Knowledge Distillation and Directional Mapping Modulator

Zhengxian Wu, Juan Wen, Wanli Peng, Yinghan Zhou, Changtong dou, Yiming Xue

Abstract

Although existing backdoor defenses have gained success in mitigating backdoor attacks, they still face substantial challenges. In particular, most of them rely on large amounts of clean data to weaken the backdoor mapping but generally struggle with residual trigger effects, resulting in persistently high attack success rates (ASR). Therefore, in this paper, we propose a novel Backdoor defense method based on Directional mapping module and adversarial Knowledge Distillation (BeDKD), which balances the trade-off between defense effectiveness and model performance using a small amount of clean and poisoned data. We first introduce a directional mapping module to identify poisoned data, which destroys clean mapping while keeping backdoor mapping on a small set of flipped clean data. Then, the adversarial knowledge distillation is designed to reinforce clean mapping and suppress backdoor mapping through a cycle iteration mechanism between trust and punish distillations using clean and identified poisoned data. Experimental results demonstrate that BeDKD surpasses the state-of-the-art defenses and reduces the ASR by 98% without significantly reducing the CACC.

Introduction

Deep Neural Networks (DNNs) are increasingly vulnerable to backdoor attacks, where malicious triggers (e.g., specific words or syntactic patterns) are injected into training data. An infected model performs normally on clean inputs but produces attacker-specified outputs when the trigger is present. Current defenses are categorized into data-level filtering and model-level erasure. However, these methods often suffer from a "defense trade-off": they either require prohibitive amounts of clean data or fail to fully eliminate the backdoor, leaving a high residual Attack Success Rate (ASR). BeDKD addresses these issues by achieving robust defense with minimal data and effectively "unlearning" the backdoor mapping.

Methodology

The BeDKD framework consists of two primary stages: Poisoned Data Identification and Adversarial Knowledge Distillation.

The BeDKD framework: (a) DMM Distillation, (b) Poisoned Data Identification, and (c) Adversarial Knowledge Distillation.

Directional Mapping Module (DMM)

To identify poisoned samples without extensive manual inspection, BeDKD utilizes a Directional Mapping Module. The DMM is initialized with the poisoned model's parameters and fine-tuned on a tiny subset of clean data with intentionally flipped labels. This process exploits the inherent stability of backdoor mappings; while the clean mapping is weakened by the noisy labels, the trigger-to-target relationship remains robust. By calculating the Mean Error of Probability Distributions (MEPD) between the original model and the DMM, the system can accurately isolate poisoned samples from the training set.

Adversarial Knowledge Distillation (AKD)

Following identification, the defense employs a cycle iteration mechanism between two distillation processes:

  1. Trust Distillation: A student model learns the correct clean mapping from a teacher model using a small clean dataset.
  2. Punish Distillation: The student model is explicitly penalized for following the backdoor mapping on the identified poisoned samples through a specialized penalty loss function. This dual-path approach ensures the model retains primary task performance while aggressively suppressing the backdoor.

Experimental Results

BeDKD was evaluated against six diverse attacks (word-level, syntactic, and style-based) across the SST2, AGnews, and OLID datasets.

Performance comparison of BeDKD against state-of-the-art defenses across multiple datasets and attack types.

The results indicate that BeDKD consistently reduces ASR to near-zero levels (averaging 0.81% to 1.79% across datasets) while maintaining Clean Accuracy (CACC) within 1% of the original performance. Notably, BeDKD demonstrates superior efficiency; for instance, on SST2, it completes the defense in 232 seconds, significantly faster than W2SDefense (4,000+ seconds) or TextGuard (30,000+ seconds). Furthermore, the method proves robust even when the clean data used for distillation is from a different domain than the training data.

Conclusion

BeDKD represents a significant advancement in model-level backdoor defense by leveraging the stability of backdoor mappings for identification and adversarial distillation for erasure. It breaks the traditional data-dependency barrier, requiring only a small fraction of data to achieve state-of-the-art results. Future work will focus on extending these techniques to generative Large Language Models (LLMs).

Summary updated

Abstract

Although existing backdoor defenses have gained success in mitigating backdoor attacks, they still face substantial challenges. In particular, most of them rely on large amounts of clean data to weaken the backdoor mapping but generally struggle with residual trigger effects, resulting in persistently high attack success rates (ASR). Therefore, in this paper, we propose a novel Backdoor defense method based on Directional mapping module and adversarial Knowledge Distillation (BeDKD), which balances the trade-off between defense effectiveness and model performance using a small amount of clean and poisoned data. We first introduce a directional mapping module to identify poisoned data, which destroys clean mapping while keeping backdoor mapping on a small set of flipped clean data. Then, the adversarial knowledge distillation is designed to reinforce clean mapping and suppress backdoor mapping through a cycle iteration mechanism between trust and punish distillations using clean and identified poisoned data. We conduct experiments to mitigate mainstream attacks on three datasets, and experimental results demonstrate that BeDKD surpasses the state-of-the-art defenses and reduces the ASR by 98% without significantly reducing the CACC. Our code are available in https://github.com/CAU-ISS-Lab/Backdoor-Attack-Defense-LLMs/tree/main/BeDKD.
BibTeX
@misc{wu2025bedkd,
  author = {Zhengxian Wu and Juan Wen and Wanli Peng and Yinghan Zhou and Changtong dou and Yiming Xue},
  title = {{BeDKD: Backdoor Defense based on Dynamic Knowledge Distillation and Directional Mapping Modulator}},
  year = {2025},
  eprint = {2508.01595},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.01595},
}
Figure