Tiny Deep Ensemble: Uncertainty Estimation in Edge AI Accelerators via Ensembling Normalization Layers with Shared Weights

Soyed Tuhin Ahmed, Michael Hefenbrock, Mehdi B. Tahoori

Abstract

The applications of artificial intelligence (AI) are rapidly evolving, and they are also commonly used in safety-critical domains, such as autonomous driving and medical diagnosis, where functional safety is paramount. In AI-driven systems, uncertainty estimation allows the user to avoid overconfidence predictions and achieve functional safety. Therefore, the robustness and reliability of model predictions can be improved. However, conventional uncertainty estimation methods, such as the deep ensemble method, impose high computation and, accordingly, hardware (latency and energy) overhead because they require the storage and processing of multiple models. Alternatively, Monte Carlo dropout (MC-dropout) methods, although having low memory overhead, necessitate numerous ($\sim 100$) forward passes, leading to high computational overhead and latency. Thus, these approaches are not suitable for battery-powered edge devices with limited computing and memory resources. In this paper, we propose the Tiny-Deep Ensemble approach, a low-cost approach for uncertainty estimation on edge devices. In our approach, only normalization layers are ensembled $M$ times, with all ensemble members sharing common weights and biases, leading to a significant decrease in storage requirements and latency. Moreover, our approach requires only one forward pass in a hardware architecture that allows batch processing for inference and uncertainty estimation. Furthermore, it has approximately the same memory overhead compared to a single model. Therefore, latency and memory overhead are reduced by a factor of up to $\sim M\times$. Nevertheless, our method does not compromise accuracy, with an increase in inference accuracy of up to $\sim 1\%$ and a reduction in RMSE of $17.17\%$ in various benchmark datasets, tasks, and state-of-the-art architectures.

Introduction

Deploying deep learning on "the edge"—battery-powered sensors, drones, or medical wearables—requires a careful balance between performance and resource constraints. In safety-critical applications, uncertainty estimation is vital for functional safety, yet standard methods are often too resource-intensive. Deep Ensembles require storing and running $M$ full models, while MC-Dropout requires many forward passes, causing prohibitive latency.

The authors identified a key opportunity: in modern architectures like ResNet-32, convolutional layer weights and biases constitute over 98% of total parameters, while normalization layers (e.g., Batch Norm) account for less than 1%. Tiny-DE exploits this lopsidedness by ensembling only the "cheap" parts of the network, drastically reducing the storage and compute footprint.

Parameter distribution in ResNet-32 showing that weights dominate the footprint.

Methodology

The core innovation of Tiny-DE is to share all "heavy" convolutional and linear weights across the ensemble while maintaining $M$ distinct sets of parameters for the normalization layers. This approach provides sufficient internal state manipulation to produce high-quality uncertainty estimates at a fraction of the cost.

The paper introduces two hardware-optimized inference modes:

  1. Sequential Inference: The model runs $M$ times, with a "router" cycling through different normalization parameters for each pass. This is ideal for Computation-in-Memory (CiM) accelerators.
  2. Single-Shot (Parallel) Inference: Designed for vectorized hardware like FPGAs or TPUs, this mode uses an EnsembleNorm layer. It repeats the input $M$ times to create a batch and applies all $M$ sets of normalization parameters simultaneously in a single forward pass.
Comparison of Deep Ensemble, BatchEnsemble, and the proposed Tiny-DE in serial and parallel modes.

Results and Discussion

Tiny-DE was evaluated across diverse tasks including image classification (CIFAR-10/100), regression (UCI benchmarks), time-series forecasting, and semantic segmentation.

Key findings include:

  • Efficiency: Latency and memory overhead are reduced by up to $\sim M\times$ compared to standard Deep Ensembles.
  • Accuracy: Ensembling normalization layers often improves base accuracy (up to 1%) and achieved a 17.17% reduction in RMSE across regression benchmarks.
  • Uncertainty Quality: In Out-of-Distribution (OoD) tests, Tiny-DE demonstrated a high relative change in predictive entropy, effectively distinguishing between seen and unseen data.
  • Augmentation: Training ensemble members with varied data augmentations further sharpened uncertainty maps in complex tasks.

Conclusion

Tiny-DE demonstrates that diversity in uncertainty estimation can be achieved "cheaply" by focusing on normalization layers. While ensembling only these layers might hit a performance ceiling on extremely high-dimensional datasets compared to full ensembles, the current results show it punches well above its weight class for edge applications. Future work includes exploring different normalization types and further optimizing router logic for low-power digital implementations.

Summary updated

Abstract

The applications of artificial intelligence (AI) are rapidly evolving, and they are also commonly used in safety-critical domains, such as autonomous driving and medical diagnosis, where functional safety is paramount. In AI-driven systems, uncertainty estimation allows the user to avoid overconfidence predictions and achieve functional safety. Therefore, the robustness and reliability of model predictions can be improved. However, conventional uncertainty estimation methods, such as the deep ensemble method, impose high computation and, accordingly, hardware (latency and energy) overhead because they require the storage and processing of multiple models. Alternatively, Monte Carlo dropout (MC-dropout) methods, although having low memory overhead, necessitate numerous ($\sim 100$) forward passes, leading to high computational overhead and latency. Thus, these approaches are not suitable for battery-powered edge devices with limited computing and memory resources. In this paper, we propose the Tiny-Deep Ensemble approach, a low-cost approach for uncertainty estimation on edge devices. In our approach, only normalization layers are ensembled $M$ times, with all ensemble members sharing common weights and biases, leading to a significant decrease in storage requirements and latency. Moreover, our approach requires only one forward pass in a hardware architecture that allows batch processing for inference and uncertainty estimation. Furthermore, it has approximately the same memory overhead compared to a single model. Therefore, latency and memory overhead are reduced by a factor of up to $\sim M\times$. Nevertheless, our method does not compromise accuracy, with an increase in inference accuracy of up to $\sim 1\%$ and a reduction in RMSE of $17.17\%$ in various benchmark datasets, tasks, and state-of-the-art architectures.
BibTeX
@inproceedings{ahmed2024tiny,
  author = {Soyed Tuhin Ahmed and Michael Hefenbrock and Mehdi B. Tahoori},
  title = {{Tiny Deep Ensemble: Uncertainty Estimation in Edge AI Accelerators via Ensembling Normalization Layers with Shared Weights}},
  booktitle = {ICCAD},
  year = {2024},
  doi = {10.1145/3676536.3676804},
}
Figure