TinyFormer: Efficient Sparse Transformer Design and Deployment on Tiny Devices

Jianlei Yang, Jiacheng Liao, Fanding Lei, Meichen Liu, Lingkun Long, Junyi Chen, Han Wan, Bei Yu, Weisheng ZhaoIEEE Transactions on Circuits and Systems I Regular Papers 2025 · DOI

TL;DR

This paper introduces TinyFormer, a novel framework for designing and deploying powerful transformer models on resource-constrained microcontrollers (MCUs). By combining neural architecture search with sparsity optimization and a specialized deployment engine, TinyFormer achieves high accuracy (96.1% on CIFAR-10) while meeting strict memory and storage limits, enabling advanced AI on tiny devices.

What Problem Does This Paper Address?

The proliferation of IoT devices has led to a growing interest in running AI models directly on microcontrollers (MCUs). However, these devices have extremely limited computational power, memory (RAM), and storage (Flash). While transformers have shown remarkable success in various AI tasks, their large size and computational demands make them unsuitable for MCUs. Existing solutions often involve significant accuracy degradation or are not optimized for the unique constraints of MCUs. This paper tackles the challenge of bridging this gap, enabling the deployment of sophisticated transformer models on these tiny devices without sacrificing performance.

Methods

TinyFormer is a three-part framework:

  • SuperNAS (Supernet Neural Architecture Search): This component searches for an optimal "supernet" – a large, over-parameterized model that encompasses a wide range of potential transformer architectures. It's designed to be hardware-aware, considering memory and storage constraints during the search.
  • SparseNAS (Sparse Neural Architecture Search): Once a suitable supernet is identified, SparseNAS searches within it for the best single-path sparse transformer model. This involves a two-stage process: an initial selection stage to find promising sparse models and a compression stage for fine-tuning and accuracy recovery. It jointly optimizes model architecture and sparsity configuration.
  • SparseEngine: This is a specialized deployment library and code generator for MCUs. It efficiently handles sparse model storage, memory allocation, and generates optimized C++ code for inference on STM32 MCUs. SparseEngine is highlighted as the first framework capable of performing sparse transformer inference on MCUs, featuring optimizations for operations like convolution, linear layers, and softmax.

Datasets

The paper primarily uses the CIFAR-10 dataset for evaluation. It also mentions using ImageNet-32 for evaluating models trained on higher-resolution datasets.

Key Insights and Ideas

  • Transformers on MCUs are feasible: The core insight is that with careful design and optimization, powerful transformer architectures can be adapted for extremely resource-constrained environments like MCUs.
  • Joint optimization is crucial: The paper emphasizes that optimizing model architecture and sparsity configuration together is key to achieving high accuracy under strict hardware limitations.
  • Hardware-aware search is essential: Simply searching for the smallest dense model isn't enough. The search process must explicitly consider MCU hardware constraints (memory, storage) to yield practical results.
  • Specialized deployment is a bottleneck: Even with an optimized model, efficient deployment on MCUs requires dedicated tools. SparseEngine addresses this by optimizing sparse model handling and code generation.
  • LayerNorm and Softmax are critical optimization targets: The paper identifies LayerNorm and Softmax as computationally expensive operations in transformers that require specific optimizations (like Scaled-LayerNorm and a bitmap lookup table) for MCU deployment.

Key Results

  • High Accuracy on CIFAR-10: TinyFormer achieves an accuracy of 96.1% on CIFAR-10 while adhering to hardware constraints of 1MB storage and 320KB memory.
  • Significant Speedups: SparseEngine provides inference speedups of up to 12.2x compared to the CMSIS-NN library.
  • Storage Reduction: TinyFormer models achieve substantial storage reduction, up to 78% compared to CMSIS-NN.
  • Improved Accuracy over Baselines: TinyFormer-300K achieves higher accuracy than CCT-7/3×1 while using 41% less peak memory and 74% less storage. TinyFormer-120K improves accuracy over MobileViT-XS by 4.4% with less memory and storage.
  • Optimized LayerNorm: The proposed Scaled-LayerNorm achieves near-equivalent accuracy to FP32 LayerNorm while being significantly faster (38.8x to 52.0x acceleration).

Limitations and Future Work

  • Search Space Tuning: The paper notes that the search space design significantly impacts the sampled model size and performance. While they explore different search space sizes, further research into automatically adapting these spaces could be beneficial.
  • Generalizability: While evaluated on CIFAR-10 and ImageNet-32, the framework's performance on a wider range of tasks and datasets for TinyML applications could be explored.
  • Hardware Diversity: The current evaluation focuses on STM32 MCUs. Extending SparseEngine to support a broader range of MCU architectures would increase its applicability.
  • Real-world Applications: While the paper demonstrates feasibility, deploying TinyFormer in actual, complex real-world TinyML applications would be a natural next step.

Summary updated

Abstract

Developing deep learning models on tiny devices (e.g. Microcontroller units, MCUs) has attracted much attention in various embedded IoT applications. However, it is challenging to efficiently design and deploy recent advanced models (e.g. transformers) on tiny devices due to their severe hardware resource constraints. In this work, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TinyFormer</i>, a framework specifically designed to develop and deploy resource-efficient transformer models on MCUs. TinyFormer consists of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SuperNAS</i>, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SparseNAS</i>, and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SparseEngine</i>. Separately, SuperNAS aims to search for an appropriate supernet from a vast search space. SparseNAS evaluates the best sparse single-path transformer model from the identified supernet. Finally, SparseEngine efficiently deploys the searched sparse models onto MCUs. To the best of our knowledge, SparseEngine is the first deployment framework capable of performing inference of sparse transformer models on MCUs. Evaluation results on the CIFAR-10 dataset demonstrate that TinyFormer can design efficient transformers with an accuracy of 96.1% while adhering to hardware constraints of 1MB storage and 320KB memory. Additionally, TinyFormer achieves significant speedups in sparse inference, up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$12.2\times $</tex-math> </inline-formula> comparing to the CMSIS-NN library. TinyFormer is believed to bring powerful transformers into TinyML scenarios and to greatly expand the scope of deep learning applications.
BibTeX
@article{yang2025tinyformer,
  author = {Jianlei Yang and Jiacheng Liao and Fanding Lei and Meichen Liu and Lingkun Long and Junyi Chen and Han Wan and Bei Yu and Weisheng Zhao},
  title = {{TinyFormer: Efficient Sparse Transformer Design and Deployment on Tiny Devices}},
  journal = {IEEE Transactions on Circuits and Systems I Regular Papers},
  year = {2025},
  doi = {10.1109/tcsi.2025.3638448},
}
Figure