Attention-Based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devices

Lu Zhang, Sangarapillai Lambotharan, Gan Zheng, Guisheng Liao, Basil AsSadhan, Fabio Roli

TL;DR

This paper introduces ATARD, a novel method to make transformer-based radio signal classifiers robust against adversarial attacks, particularly for low-power IoT devices. It achieves this by distilling "adversarial attention maps" from a larger, robust teacher model to a smaller student model, outperforming existing defense techniques.

What Problem Does This Paper Address?

The rapid growth of IoT devices necessitates efficient radio spectrum management, which relies on accurately classifying radio signal modulations. While transformer networks have shown great promise in this area, they are vulnerable to adversarial attacks – subtle input modifications that can cause misclassifications. These attacks are a significant concern for security and reliability, especially in environments where devices might be targeted by adversaries. Existing defense mechanisms, like adversarial training, often require large models and significant computational resources, making them unsuitable for low-power IoT devices. The paper aims to bridge this gap by developing an efficient and robust defense mechanism for transformer-based radio signal classification on resource-constrained devices.

Methods

The core of the proposed solution is the Attention-based Adversarial Robustness Distillation (ATARD) method. Unlike previous adversarial distillation techniques that focus on transferring model logits (penultimate layer outputs), ATARD uniquely transfers the adversarial attention map from a robustly trained large transformer (teacher model) to a smaller transformer (student model).

Here's a breakdown of the key methodological aspects:

  • Teacher-Student Architecture: A large, pre-trained transformer model (teacher) serves as the source of robustness. A smaller, more efficient transformer model (student) is trained to mimic the teacher's behavior.
  • Adversarial Attention Map Distillation: During training, adversarial examples are generated for the input signals. Both the teacher and student models process these adversarial inputs. The ATARD method then forces the student model's attention maps (which capture how the model focuses on different parts of the input) to match those of the teacher model. This is achieved by minimizing the L2 norm of the difference between the teacher's and student's adversarial attention maps across multiple encoder layers.
  • Combined Loss Function: The student model's training objective includes two main components:
  • Cross-Entropy Loss: Standard classification loss on adversarial samples to ensure correct classification.
  • Attention Map Distillation Loss: The L2 norm difference between teacher and student adversarial attention maps, encouraging the student to learn the robust attention patterns.
  • Compact Transformer Architecture: The paper emphasizes the use of a compact transformer architecture for the student model to meet the low-power IoT device requirements. This involves reducing parameters by using fewer encoder layers and smaller embedding dimensions.

Datasets

The paper utilizes two publicly available datasets for evaluation:

  • GNU Radio ML dataset RML2016.10a: This dataset contains 220,000 samples across 11 modulation types at various Signal-to-Noise Ratios (SNRs).
  • RDL 2021.12 dataset: This dataset includes two noise types (Gaussian and alpha-stable) and considers Rayleigh and Rician fading channels. It was pre-processed to mitigate impulsive noise.

Key Insights and Ideas

  • Attention Maps as a Source of Robustness: The paper posits that adversarial attention maps, which reveal how a transformer model processes perturbed inputs, contain crucial information for robustness. Transferring these maps, rather than just output probabilities, is a novel way to imbue smaller models with resilience.
  • Distilling Robustness for Low-Power Devices: ATARD offers a practical solution for deploying robust AI on resource-constrained IoT devices, where training large, robust models from scratch is infeasible.
  • Smoothness as a Measure of Robustness: The paper demonstrates that ATARD leads to a "smoother" loss function landscape, meaning larger perturbations are required to cause misclassifications. This is a key theoretical underpinning for its improved robustness.
  • Transferability of Adversarial Examples: The study also investigates how adversarial examples generated for one model can fool another, highlighting the importance of robust defenses that generalize across architectures.

Key Results

  • Superior Robustness: ATARD consistently outperforms existing defense methods (NT, AT, ARD, IAD, AKD, RSLAD) against both Fast Gradient Method (FGM) and Projected Gradient Descent (PGD) attacks across various PNR (Perturbation-to-Noise Ratio) values on the RML dataset. For example, at PNR = -10 dB, ATARD achieved significantly higher classification accuracy compared to other methods.
  • Performance on RDL Dataset: Similar improvements were observed on the RDL dataset under Rayleigh and Rician fading channels, especially at higher PGNR values. ATARD showed substantial accuracy gains over AT and NT.
  • Smoother Loss Function: Experimental results (Table III) show that ATARD has the smallest average L2 norm of the loss gradient with respect to the input, indicating it is less sensitive to perturbations.
  • Transferability Defense: ATARD demonstrates good performance even when adversarial examples are generated from different transformer architectures (Model-1 and Model-2), suggesting its robustness is not overly specific to the teacher model's architecture.

Limitations and Future Work

  • White-Box Attacks: The current evaluation primarily focuses on white-box attacks, where the attacker has full knowledge of the defense system. While this represents a worst-case scenario, future work could explore robustness against black-box and gray-box attacks.
  • Specific Transformer Architectures: The proposed method is tailored for transformer-based networks. Its applicability to other network architectures (e.g., CNNs) might require further investigation.
  • Generalization to Other Tasks: While effective for radio signal classification, the ATARD approach's effectiveness on other domains (e.g., image classification) could be explored.
  • Computational Cost of Teacher Model: The ATARD method still relies on a large, robust teacher model for distillation. Optimizing the teacher model's training or exploring alternative ways to obtain robust attention patterns could be future directions.

Summary updated

Abstract

Due to great success of transformers in many applications, such as natural language processing and computer vision, transformers have been successfully applied in automatic modulation classification. We have shown that transformer-based radio signal classification is vulnerable to imperceptible and carefully crafted attacks called adversarial examples. Therefore, we propose a defense system against adversarial examples in transformer-based modulation classifications. Considering the need for computationally efficient architecture particularly for Internet of Things (IoT)-based applications or operation of devices in an environment where power supply is limited, we propose a compact transformer for modulation classification. The advantages of robust training such as adversarial training in transformers may not be attainable in compact transformers. By demonstrating this, we propose a novel compact transformer that can enhance robustness in the presence of adversarial attacks. The new method is aimed at transferring the adversarial attention map from the robustly trained large transformer to a compact transformer. The proposed method outperforms the state-of-the-art techniques for the considered white-box scenarios, including the fast gradient method and projected gradient descent attacks. We have provided reasoning of the underlying working mechanisms and investigated the transferability of the adversarial examples between different architectures. The proposed method has the potential to protect the transformer from the transferability of adversarial examples.
BibTeX
@article{zhang2023attention,
  author = {Lu Zhang and Sangarapillai Lambotharan and Gan Zheng and Guisheng Liao and Basil AsSadhan and Fabio Roli},
  title = {{Attention-Based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devices}},
  journal = {IEEE Internet of Things Journal},
  year = {2023},
  doi = {10.1109/jiot.2022.3215188},
}
Figure