DiTMoS: Delving into Diverse Tiny-Model Selection on Microcontrollers

Xiao Ma, Shengfeng He, Hezhe Qiao, Dong Ma

Introduction

The deployment of Deep Neural Networks (DNNs) on microcontrollers (MCUs) represents a significant frontier in pervasive sensing. However, this transition is hindered by severe hardware constraints, particularly the limited Static Random-Access Memory (SRAM), which often falls below 256KB. The traditional industry paradigm for addressing these limits is a "top-down" approach: taking a high-capacity, accurate model and applying compression techniques such as pruning, quantization, or knowledge distillation.

While effective to a point, this approach eventually encounters a "transferability limit." As models are compressed into the "tiny" regime, their representation capability degrades sharply. DiTMoS (Diverse Tiny-Model Selection) proposes a paradigm shift toward a "bottom-up" ensemble strategy. Instead of shrinking a single generalist model, DiTMoS intelligently selects from a diverse ensemble of specialized tiny models, leveraging their collective strengths to achieve high accuracy within strict resource bounds.

The DiTMoS Framework

The core of the DiTMoS framework is a Selector-Classifier architecture. In this setup, a dedicated "Selector" network analyzes the input data and routes it to the most appropriate "Classifier" (a specialized tiny model) for final inference. This approach is built upon three technical pillars designed to maximize accuracy and minimize memory overhead.

1. Diverse Training Data Splitting

To ensure that the ensemble consists of truly specialized models, the authors employ a data-driven splitting strategy. A pre-trained "strong" model is used to extract high-dimensional features from the training dataset. These features are then clustered using K-Means into $m$ distinct subsets. Each tiny classifier is initially trained on a specific subset, forcing it to develop expertise in a particular region of the feature space. This diversity is crucial; while individual tiny models may have low overall accuracy, their "union accuracy"—the probability that at least one model in the set is correct—is significantly higher.

2. Adversarial Selector-Classifiers Training

To optimize the routing process, the selector and classifiers are trained through an asynchronous, iterative loop. This adversarial-style training ensures the selector learns the specific strengths and weaknesses of each specialist. As the selector improves its routing logic, the specialists further refine their knowledge based on the specific samples they are assigned, creating a virtuous cycle of specialization and coordination.

3. Heterogeneous Feature Aggregation and Network Slicing

To enhance the capacity of the tiny classifiers without significantly increasing their parameter count, DiTMoS utilizes "Feature Aggregation." The classifiers "borrow" intermediate feature representations from the selector network. Since the early layers of the selector are often task-neutral, they provide a rich feature set that acts as a "free" boost in depth for the classifiers.

To manage the SRAM impact of these additional features, the authors introduced Network Slicing. This technique involves storing intermediate activations in the larger, non-volatile Flash memory and reloading them into SRAM only when needed. This strategy effectively trades a marginal increase in latency for a substantial reduction in peak SRAM usage.

Experimental Evaluation

The framework was evaluated on a Nucleo STM32F767ZI board (Cortex-M7) across three diverse time-series tasks:

  • UniMiB-SHAR: Human Activity Recognition (17 classes) using accelerometer data.
  • Speech Commands: Keyword spotting (30 classes) using audio MFCC features.
  • DEAP: Emotion recognition (4 classes) based on EEG signals.

Key Results

DiTMoS demonstrated superior performance compared to standard baselines, including Knowledge Distillation and Mixture of Experts (MoE):

  • Accuracy Gains: The framework achieved an accuracy boost of 7.0% on UniMiB-SHAR, 4.3% on Speech Commands, and 13.4% on the DEAP dataset.
  • Memory Efficiency: Through the use of Network Slicing, peak SRAM usage was maintained at 6.2 KB, nearly identical to the 6.1 KB required by a single standard tiny model. This demonstrates that the ensemble approach does not necessarily lead to a proportional increase in runtime memory requirements.
  • Latency Trade-off: The overhead introduced by network slicing was minimal, ranging from 0.6ms to 2.1ms depending on the specific task configuration, which is well within the requirements for most real-time sensing applications.

Discussion and Future Work

DiTMoS proves that a "bottom-up" ensemble of specialized models can significantly outperform a single "top-down" compressed model on microcontrollers. However, certain considerations remain for future development. While SRAM usage is optimized, the ensemble approach requires more Flash storage to house multiple models. While Flash is typically more abundant than SRAM on MCUs, this could become a constraint for very large ensembles.

Future research directions include integrating Neural Architecture Search (NAS) to automatically optimize the shapes of the selector and classifiers, as well as applying 8-bit quantization to further reduce the overall resource footprint and potentially improve latency.

Summary updated

BibTeX
@inproceedings{ma2024ditmos,
  author = {Xiao Ma and Shengfeng He and Hezhe Qiao and Dong Ma},
  title = {{DiTMoS: Delving into Diverse Tiny-Model Selection on Microcontrollers}},
  booktitle = {PerCom},
  year = {2024},
  doi = {10.1109/percom59722.2024.10494422},
}
Figure