Integrated Robust Optimization for Lightweight Transformer Models in Low-Resource Scenarios

Hui Huang, Hengyu Zhang, Yusen Wang, Haibin Liu, Xiao Jie Chen, Yiling Chen, Yuan LiangSymmetry 2025 · DOI

TL;DR

This paper tackles the critical challenge of deploying robust AI models on resource-constrained edge devices by enhancing lightweight Transformer models. They introduce a novel evaluation framework and a suite of adversarial training techniques that significantly improve robustness against various attacks, making powerful AI more accessible and secure on devices like smartphones.

What Problem Does This Paper Address?

The widespread adoption of AI is leading to an increasing demand for AI capabilities on edge devices such as smartphones, cameras, and embedded systems. However, these devices have limited storage, computational power, and network bandwidth, making it difficult to deploy large, powerful AI models like standard Transformers. While lightweight Transformer models (e.g., MobileViT) offer a solution by reducing model size and computational needs, they often suffer from insufficient robustness, especially against adversarial attacks. This lack of robustness poses a significant security risk in critical edge applications. This paper aims to bridge this gap by developing methods to make lightweight Transformers robust enough for real-world edge deployment.

Methods

The authors propose a multi-pronged approach:

  • Robustness Evaluation Framework: They introduce a comprehensive framework to assess lightweight Transformer robustness across three key dimensions:
  • Noise Robustness: Evaluating performance under various image corruptions (blur, weather, digital noise).
  • Distributional Robustness: Assessing performance on out-of-distribution (OOD) data, like sketches or images with different artistic styles.
  • Adversarial Robustness: Testing resilience against malicious inputs generated by attacks like FGSM and PGD. This framework also investigates the impact of model size and hardware constraints on robustness.

  • Adversarial Robustness Enhancement Strategy: To improve robustness, they combine several techniques:

  • Stabilized Adversarial Training: They address training instability in lightweight models using:
  • Static Parameter Adjustment: Optimizing learning rate, momentum, and weight decay.
  • Gradient Clipping: Constraining gradient updates to prevent explosion.
  • Layer-wise Unfreezing: Gradually unfreezing model layers during training to mitigate overfitting.
  • Decision Boundary Optimization: They incorporate advanced techniques to balance accuracy and robustness:
  • TRADES (Trade-off between Robustness and Accuracy): A loss function that jointly optimizes for clean accuracy and adversarial robustness.
  • SMART (Smoothness-inducing Adversarial Regularization and Bregman proximal point optimization): A method that encourages smoother decision boundaries and more consistent predictions for similar inputs.

Datasets

The study utilizes several standard datasets for evaluation and training:

  • Tiny-ImageNet-C: Used for evaluating noise robustness.
  • ImageNet-R and ImageNet-Sketch: Used for evaluating distributional robustness.
  • ImageNet-A: Used for evaluating robustness against naturally occurring adversarial examples.
  • ImageNet-1k validation set and CIFAR-10: Used for adversarial robustness evaluation and training.

Key Insights and Ideas

  • Lightweighting Hurts Robustness: Compressing Transformer models significantly degrades their adversarial robustness, even more so than noise or distributional shifts.
  • Standard Adversarial Training is Problematic for Lightweight Models: Lightweight Transformers exhibit training instability (overfitting, gradient explosion) with traditional adversarial training methods.
  • A Multi-faceted Approach is Necessary: Combining stabilization techniques (gradient clipping, layer-wise unfreezing) with decision boundary optimization (TRADES, SMART) is crucial for achieving significant robustness gains.
  • Model Size Matters, but Not Always as Expected: While larger models generally show better noise robustness, the relationship with adversarial robustness is more complex and can be severely impacted by aggressive compression.
  • Edge Deployment is Feasible: The proposed robust lightweight models can be successfully deployed and run on resource-constrained mobile devices, demonstrating practical applicability.

Key Results

  • Adversarially trained lightweight models achieved a 22.14% improvement in robustness accuracy against FGSM attacks and a 47.59% improvement against PGD attacks compared to non-adversarially trained models.
  • The proposed enhanced training strategy boosted adversarial robustness against PGD attacks by an additional 33.7%.
  • On mobile devices, the adversarially trained MobileViT model correctly classified an adversarial image with a confidence score of 0.82, whereas the original model misclassified it with low confidence.
  • The DALI library was shown to reduce data preprocessing and loading time by 95.68% compared to standard PyTorch, significantly speeding up adversarial training.
  • The SMART+ method achieved optimal adversarial robustness with 3 update iterations, outperforming traditional baselines.

Limitations and Future Work

  • Focus on Adversarial Training: The current work primarily focuses on adversarial training. Future research could explore improving noise and data robustness further.
  • Vision Tasks Only: The study is limited to image classification. Future work could extend to other computer vision tasks like object detection and semantic segmentation.
  • Mobile Devices as Primary Scenario: While mobile devices are a key low-resource scenario, future work could explore other environments like embedded systems.
  • Transformer Architectures: The research focuses on Transformer models. Extending these findings to other model architectures could be beneficial.

Summary updated

Abstract

With the rapid proliferation of artificial intelligence (AI) applications, an increasing number of edge devices—such as smartphones, cameras, and embedded controllers—are being tasked with performing AI-based inference. Due to constraints in storage capacity, computational power, and network connectivity, these devices are often categorized as operating in resource-constrained environments. In such scenarios, deploying powerful Transformer-based models like ChatGPT and Vision Transformers is highly impractical because of their large parameter sizes and intensive computational requirements. While lightweight Transformer models, such as MobileViT, offer a promising solution to meet storage and computational limitations, their robustness remains insufficient. This poses a significant security risk for AI applications, particularly in critical edge environments. To address this challenge, our research focuses on enhancing the robustness of lightweight Transformer models under resource-constrained conditions. First, we propose a comprehensive robustness evaluation framework tailored for lightweight Transformer inference. This framework assesses model robustness across three key dimensions: noise robustness, distributional robustness, and adversarial robustness. It further investigates how model size and hardware limitations affect robustness, thereby providing valuable insights for robustness-aware model design. Second, we introduce a novel adversarial robustness enhancement strategy that integrates lightweight modeling techniques. This approach leverages methods such as gradient clipping and layer-wise unfreezing, as well as decision boundary optimization techniques like TRADES and SMART. Together, these strategies effectively address challenges related to training instability and decision boundary smoothness, significantly improving model robustness. Finally, we deploy the robust lightweight Transformer models in real-world resource-constrained environments and empirically validate their inference robustness. The results confirm the effectiveness of our proposed methods in enhancing the robustness and reliability of lightweight Transformers for edge AI applications.
BibTeX
@inproceedings{huang2025integrated,
  author = {Hui Huang and Hengyu Zhang and Yusen Wang and Haibin Liu and Xiao Jie Chen and Yiling Chen and Yuan Liang},
  title = {{Integrated Robust Optimization for Lightweight Transformer Models in Low-Resource Scenarios}},
  booktitle = {Symmetry},
  year = {2025},
  doi = {10.3390/sym17071162},
}
Figure