Attention-based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devices

Lu Zhang, Sangarapillai Lambotharan, Gan Zheng, Guisheng Liao, Basil AsSadhan, Fabio Roli

Introduction

Automatic Modulation Classification (AMC) is a critical component of modern wireless communication systems, enabling spectrum monitoring and interference management. While deep learning models, particularly Transformers, have significantly improved AMC performance, they remain highly susceptible to adversarial attacks. These attacks involve small, intentionally crafted perturbations that can lead a model to misclassify signals, posing a severe security risk in hostile electronic environments.

The Robustness Gap in IoT

Deploying robust AMC models on IoT devices presents a unique challenge. These devices are constrained by limited power and memory, necessitating the use of compact models. While large-scale Transformers can be made resilient through standard adversarial training (AT), compact versions often fail to retain this robustness even when trained with identical techniques. This "robustness gap" leaves low-power devices vulnerable to even simple adversarial techniques like the Fast Gradient Method or Projected Gradient Descent (PGD).

The ATARD Framework

To bridge this gap, the researchers proposed ATARD (ATtention-based Adversarial Robustness Distillation). Unlike traditional knowledge distillation, which typically focuses on matching the final output (logits) of a teacher and student, ATARD focuses on the internal Attention Maps.

The ATARD framework showing the transfer of Adversarial Attention Maps (AAM) from a robust teacher to a compact student model.

As shown in the framework above, the process involves:

  1. Teacher-Student Architecture: A large, adversarially-trained Transformer acts as the teacher. The student is a significantly smaller model, reducing the parameter count from approximately 801k to 230k.
  2. Adversarial Attention Map (AAM) Transfer: During training, the student is exposed to adversarial examples. It is optimized to minimize a loss function that includes the difference between its own attention maps and those of the robust teacher (Loss2).
  3. Function Smoothing: By mimicking the teacher’s attention patterns under attack, the student learns a "smoother" decision function. This makes the model less sensitive to small input changes, forcing an attacker to use larger, more detectable perturbations to achieve a misclassification.

Experimental Evaluation and Results

The effectiveness of ATARD was evaluated using two primary benchmarks: RML2016.10a (a standard GNU Radio dataset) and RDL2021.12 (a complex dataset featuring Rayleigh and Rician fading).

Accuracy vs. Perturbation-to-Noise Ratio (PNR) for various defense methods, showing ATARD's superior performance.

Key results from the study include:

  • Superior Robustness: Against strong PGD attacks at a Perturbation-to-Noise Ratio (PNR) of -10 dB, ATARD achieved roughly 13% higher accuracy than standard adversarial training and outperformed other state-of-the-art distillation methods like ARD, IAD, and AKD.
  • Efficiency Gains: The student model maintained high robustness while being nearly 70% smaller than the teacher model, making it ideal for resource-constrained IoT hardware.
  • Resilience in Fading Channels: In challenging Rayleigh fading scenarios, ATARD provided a 19% accuracy improvement over standard AT at a PNR of -20 dB.

Conclusion

ATARD demonstrates that preserving attention-based inter-patch relations is key to maintaining robustness in compact Transformer models. By distilling adversarial attention from a large teacher, the method provides a path toward secure and efficient radio signal classification for low-power IoT devices. Future work will focus on testing these architectures on physical hardware to evaluate real-time latency and exploring defenses against more sophisticated adaptive attacks.

Summary updated

BibTeX
@article{zhang2023attention,
  author = {Lu Zhang and Sangarapillai Lambotharan and Gan Zheng and Guisheng Liao and Basil AsSadhan and Fabio Roli},
  title = {{Attention-based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devices}},
  journal = {IEEE Internet of Things Journal},
  year = {2023},
  doi = {10.1109/jiot.2022.3215188},
}
Figure