AGKD-BML: Defense Against Adversarial Attack by Attention Guided Knowledge Distillation and Bi-directional Metric Learning

Hong Wang, Yuefan Deng, Shinjae Yoo, Haibin Ling, Yuewei Lin

Abstract

Deep Neural Networks (DNNs) are highly susceptible to adversarial examples—images with imperceptible perturbations that trigger misclassification. We present AGKD-BML, a dual-pronged defense strategy that combines Attention Guided Knowledge Distillation (AGKD) and Bi-directional Metric Learning (BML). By distilling clean attention patterns from a teacher model and constraining the feature space through a novel bi-directional triplet loss, AGKD-BML significantly improves model robustness against state-of-the-art adversarial attacks.

Introduction: The Brittleness of Deep Learning

Despite their success, DNNs remain "brittle." A "German Shepherd" can be easily misclassified as a "Planetarium" through minor pixel manipulations. Standard defenses like adversarial training often struggle with generalization. This research identifies two primary drivers of adversarial success:

  1. Attention Corruption: Adversarial noise shifts the model's internal focus toward irrelevant image regions.
  2. Feature Space Drift: Adversarial examples (AEs) are pushed far from their original class clusters in the model's internal representation, often crossing decision boundaries.

The AGKD-BML Framework

To address these issues, the authors propose a framework consisting of two synergistic modules designed to maintain focus and structural integrity in the feature space.

1. Attention Guided Knowledge Distillation (AGKD)

AGKD employs a Teacher-Student architecture. A pre-trained, fixed "Teacher" model provides a "class-irrelevant" spatial attention map to a "Student" model undergoing adversarial training. Unlike traditional distillation that focuses on class labels, AGKD teaches the student where to look. By forcing the student to mirror the teacher's clean spatial attention, the model learns to ignore adversarial noise and focus on the actual object.

2. Bi-directional Metric Learning (BML)

While standard metric learning pulls an AE toward its clean counterpart, BML introduces a more rigid constraint using a bi-directional approach:

  • Forward AE: A clean image is attacked to resemble its most confusing class.
  • Backward AE: A random image from the confusing class is attacked to resemble the original class.

By minimizing the distance between clean images and their AEs while maximizing the distance between forward and backward AEs, the model creates a stronger "pull" back toward the correct class cluster.

The AGKD-BML framework overview, showing the Teacher-Student attention distillation and the bi-directional mapping to the feature space.

Experimental Results

The model was evaluated on CIFAR-10 and SVHN benchmarks against rigorous attacks including PGD, CW, and AutoAttack (AA).

  • CIFAR-10: Under a 20-step PGD attack, AGKD-BML achieved 71.02% accuracy, a significant improvement over standard adversarial training (~45%) and other metric-learning defenses (~51%).
  • SVHN: The model reached 74.94% accuracy under PGD-20, maintaining a substantial lead over competing methods.

Visualizing Feature Space Robustness

The effectiveness of BML is most apparent in t-SNE visualizations. In competing models, adversarial examples (represented as red dots) remain scattered and confused across class boundaries. In AGKD-BML, these points are tightly clustered within their original class regions, demonstrating superior feature space stability.

t-SNE visualizations of different defense methods. AGKD-BML shows the most effective clustering of adversarial examples (red dots) within their correct class regions.

Conclusion and Future Work

AGKD-BML demonstrates that maintaining clean attention patterns and enforcing bi-directional constraints in the feature space are powerful tools for adversarial defense. While the method shows state-of-the-art performance on standard benchmarks, scaling to more complex datasets like Tiny ImageNet remains a challenge due to the increasing difficulty of optimizing the "most confusing class" logic. Future research will focus on improving computational efficiency—as training currently takes roughly 29 hours on CIFAR-10—and enhancing generalization for high-resolution, multi-class environments.

Summary updated

Abstract

While deep neural networks have shown impressive performance in many tasks, they are fragile to carefully designed adversarial attacks. We propose a novel adversarial training-based model by Attention Guided Knowledge Distillation and Bi-directional Metric Learning (AGKD-BML). The attention knowledge is obtained from a weight-fixed model trained on a clean dataset, referred to as a teacher model, and transferred to a model that is under training on adversarial examples (AEs), referred to as a student model. In this way, the student model is able to focus on the correct region, as well as correcting the intermediate features corrupted by AEs to eventually improve the model accuracy. Moreover, to efficiently regularize the representation in feature space, we propose a bidirectional metric learning. Specifically, given a clean image, it is first attacked to its most confusing class to get the forward AE. A clean image in the most confusing class is then randomly picked and attacked back to the original class to get the backward AE. A triplet loss is then used to shorten the representation distance between original image and its AE, while enlarge that between the forward and backward AEs. We conduct extensive adversarial robustness experiments on two widely used datasets with different attacks. Our proposed AGKD-BML model consistently outperforms the state-of-the-art approaches. The code of AGKD-BML will be available at: https://github.com/hongw579/AGKD-BML.
BibTeX
@inproceedings{wang2021agkd,
  author = {Hong Wang and Yuefan Deng and Shinjae Yoo and Haibin Ling and Yuewei Lin},
  title = {{AGKD-BML: Defense Against Adversarial Attack by Attention Guided Knowledge Distillation and Bi-directional Metric Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00756},
}
Figure