Lightweight Software Kernels and Hardware Extensions for Efficient Sparse Deep Neural Networks on Microcontrollers

Francesco Daghero, Daniele Jahier Pagliari, Francesco Conti, Luca Benini, Massimo Poncino, Alessio Burrello

Introduction

Deploying Deep Neural Networks (DNNs) on ultra-low-power microcontrollers (MCUs) is a significant challenge due to severe memory and computational constraints. While weight pruning is a common technique to reduce model size, the resulting unstructured sparsity often leads to inefficient execution on simple processors. This paper introduces a full-stack framework—encompassing software kernels, hardware extensions, and a compiler—to enable efficient N:M semi-structured sparsity on RISC-V based MCUs.

The Sparsity Bottleneck on Edge Devices

Pruning techniques typically result in unstructured sparsity, where non-zero weights are distributed randomly. While this maximizes compression, it creates a "random mess" of zeros that standard MCU architectures struggle to skip, often spending more cycles on index management and address calculation than on actual arithmetic.

Comparison of sparsity patterns: Dense, Unstructured, Semi-structured N:M, and Structured-Blockwise.

The industry has gravitated toward N:M semi-structured pruning (e.g., 1:4 or 1:8) as a compromise. This format enforces a fixed number of non-zero elements within a block, making it more amenable to hardware acceleration. However, existing MCU architectures lack the dedicated support found in high-end GPUs to exploit this structure. This paper bridges that gap, providing the first comprehensive stack to make N:M sparsity a reality for ultra-low-power edge devices.

A Three-Tiered Solution for N:M Sparsity

The proposed framework addresses the indexing bottleneck through a three-tiered approach:

1. Optimized Software Kernels

The authors developed specialized C kernels for various sparsity ratios (1:4, 1:8, 1:16). These kernels utilize a "Decimate Im2col" strategy, which avoids processing zeros by intelligently selecting only the required input activations for multiplication with non-zero weights.

2. Hardware Extension (xDecimate)

To eliminate the indexing bottleneck, a custom instruction was added to the RISC-V ISA. The xDecimate instruction performs index unpacking and indirect memory loads in a single cycle, significantly reducing the overhead of sparse data management. This surgical hardware intervention targets the primary bottleneck of sparse computation: figuring out where the data is, rather than the math itself.

Detailed representation of N:M semi-structured sparsity with non-zero values and index masks.

3. MATCH Compiler Integration

The framework is integrated into the MATCH compiler (based on Apache TVM), which automatically identifies N:M patterns and generates optimized tiled code that leverages the custom kernels and hardware instructions.

Experimental Evaluation

The framework was evaluated using ResNet18 (CIFAR100) and Vision Transformer (ViT-Small, CIFAR10) models.

  • Performance: Hardware-accelerated kernels achieved up to a 3.21× end-to-end speedup for ResNet18 at 1:16 sparsity. Software-only optimizations alone provided up to 3.4× speedups for Fully-Connected layers.
  • Efficiency: The xDecimate instruction provided an additional 1.9× speedup over optimized software-only kernels with only a 5% increase in core area, making it highly practical for commercial production.
  • Accuracy: Accuracy degradation remained below 1.5% even at high sparsity levels (1:16). In some cases, such as 1:4 sparsity, the sparse models actually performed slightly better than the dense baselines.

Conclusion

This work demonstrates that N:M semi-structured sparsity is a highly effective "Goldilocks" solution for edge AI—structured enough for efficient SIMD operations but flexible enough to maintain high accuracy. By combining surgical hardware interventions with optimized software and compiler support, the authors provide a practical path for running sophisticated DNNs on low-cost, low-power microcontrollers.

Summary updated

Abstract

The acceleration of pruned Deep Neural Networks (DNNs) on edge devices such as Microcontrollers (MCUs) is a challenging task, given the tight area- and power-constraints of these devices. In this work, we propose a three-fold contribution to address this problem. First, we design a set of optimized software kernels for N:M pruned layers, targeting ultra-low-power, multicore RISC-V MCUs, which are up to 2.1$\times$ and 3.4$\times$ faster than their dense counterparts at 1:8 and 1:16 sparsity, respectively. Then, we implement a lightweight Instruction-Set Architecture (ISA) extension to accelerate the indirect load and non-zero indices decompression operations required by our kernels, obtaining up to 1.9$\times$ extra speedup, at the cost of a 5\% area overhead. Lastly, we extend an open-source DNN compiler to utilize our sparse kernels for complete networks, showing speedups of 3.21$\times$ and 1.81$\times$ on a ResNet18 and a Vision Transformer (ViT), with less than 1.5\% accuracy drop compared to a dense baseline.
BibTeX
@inproceedings{daghero2025lightweight,
  author = {Francesco Daghero and Daniele Jahier Pagliari and Francesco Conti and Luca Benini and Massimo Poncino and Alessio Burrello},
  title = {{Lightweight Software Kernels and Hardware Extensions for Efficient Sparse Deep Neural Networks on Microcontrollers}},
  booktitle = {MLSys},
  year = {2025},
  url = {https://openreview.net/forum?id=NJvVOgN81a},
}
Figure