Multi-Component Optimization and Efficient Deployment of Neural-Networks on Resource-Constrained IoT Hardware

Bharath Sudharsan, Dineshkumar Sundaram, Pankesh Patel, John G. Breslin, Muhammad Intizar Ali, Schahram Dustdar, Albert Y. Zomaya, Rajiv RanjanarXiv 2022 · DOI · arXiv

Introduction

The proliferation of the Internet of Things (IoT) has traditionally relied on cloud-based processing for "intelligent" features. However, transmitting sensor data to remote servers introduces significant latency, privacy risks (e.g., GDPR compliance), and bandwidth costs. Moreover, cloud-dependent devices fail entirely without a stable internet connection.

To address these issues, the field of TinyML aims to run artificial intelligence directly on the "edge"—specifically on microcontrollers (MCUs). Unlike powerful servers, MCUs like the ARM Cortex-M series are extremely resource-constrained, often featuring less than 256KB of SRAM and clock speeds below 100MHz. Standard deep learning models are far too large for these environments. This paper proposes a comprehensive multi-component optimization sequence to bridge the gap between high-performance AI and tiny hardware.

The Multi-Component Optimization Pipeline

The researchers developed an end-to-end optimizer consisting of five distinct phases, designed to shrink model size and accelerate execution without sacrificing predictive accuracy.

1. Pre-training and Post-training Optimization

The pipeline begins with Pruning (removing redundant network connections) and Quantization-Aware Training (QAT). These techniques prepare the model for a smaller footprint during the learning phase. Following training, weights are converted from 32-bit floating points to 8-bit integers (Int8) or 16-bit floats (Float16). A "Joint" approach, combining pre-training preparation with post-training quantization, was found to be particularly effective.

2. Graph-Level Simplification

Graph optimization treats the neural network as a mathematical structure to be simplified. Key techniques include:

  • Arithmetic Simplification: Reducing computational complexity by re-writing expressions (e.g., replacing repeated additions with multiplications).
  • Hoisting: Moving loop-invariant sub-graphs out of execution loops to minimize redundant cycles.
  • Redundancy Removal: Identifying and deleting "NoOp" (no operation) nodes that do not impact the final output.

3. Operations and Kernel Tuning

At a lower level, standard 2D convolutions are decomposed into Depthwise Separable Convolutions, significantly reducing the multiplication operations required. Finally, the pipeline applies kernel-level C++ optimizations, such as 2x2 matrix multiplication kernels tailored for specific CPU register counts, and input data reduction via low-pass filters to manage high-frequency sensor data.

Experimental Evaluation and Hardware Targets

The optimization sequence was evaluated using the MNIST Fashion and MNIST Digits datasets. These benchmarks provided a baseline to ensure that the aggressive compression did not degrade accuracy.

Figure 1: Comparison of various microcontroller units (MCUs) used in the study, highlighting their diverse SRAM, Flash, and clock speed constraints.

As shown in Figure 1, the study targeted a wide range of hardware, from the ultra-constrained ATmega328P to more capable modules like the ESP32 and nRF52840. These devices vary significantly in memory (8 KB to 520 KB SRAM) and processing power (16 MHz to 240 MHz), representing the heterogeneous landscape of modern IoT deployment.

Key Results

The experimental results demonstrate the efficacy of the multi-component approach:

  • Model Compression: The pipeline achieved compression ratios up to 12.06x. In one instance, a 271.4 KB model was reduced to just 22.5 KB, allowing it to fit comfortably on devices with limited Flash memory.
  • Inference Performance: Optimized models reached unit inference times as low as 0.06 ms, a massive improvement over unoptimized baselines which could take upwards of 300 ms.
  • Accuracy Preservation: Remarkably, accuracy remained stable, with some optimized models showing a slight improvement (up to +0.27%). This suggests that pruning and quantization can act as regularizers, reducing overfitting.

Conclusion and Future Directions

This research proves that a sequenced, hardware-aware optimization strategy can successfully deploy neural networks on even the most resource-starved microcontrollers. By treating optimization as a multi-stage process—from high-level graph simplification to low-level kernel tuning—developers can achieve high-speed, local AI. Future work will focus on scaling this pipeline to more complex architectures like Tiny-YOLO and exploring automated Neural Architecture Search (NAS) to further streamline deployment for specific hardware targets.

Summary updated

BibTeX
@misc{sudharsan2022multi,
  author = {Bharath Sudharsan and Dineshkumar Sundaram and Pankesh Patel and John G. Breslin and Muhammad Intizar Ali and Schahram Dustdar and Albert Y. Zomaya and Rajiv Ranjan},
  title = {{Multi-Component Optimization and Efficient Deployment of Neural-Networks on Resource-Constrained IoT Hardware}},
  year = {2022},
  eprint = {2204.10183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.10183},
}
Figure