Tiny Deep Ensemble: Uncertainty Estimation in Edge AI Accelerators via Ensembling Normalization Layers with Shared Weights

Soyed Tuhin Ahmed, Michael Hefenbrock, Mehdi B. Tahoori

In safety-critical applications like autonomous driving or medical diagnostics, it’s not enough for an AI to be right; it needs to know when it might be wrong. This is the domain of uncertainty estimation. While "Deep Ensembles" are the gold standard for this, they are notoriously resource-heavy, requiring multiple full models to be stored and run.

A new paper, "Tiny Deep Ensemble: Uncertainty Estimation in Edge AI Accelerators via Ensembling Normalization Layers with Shared Weights," proposes a clever workaround: what if we only ensembled the "cheap" parts of the network?

TL;DR

Tiny-DE is a lightweight uncertainty estimation method that ensembles only the normalization layers of a neural network while sharing all other weights. It achieves near-state-of-the-art uncertainty quantification with up to $M\times$ lower memory and latency overhead than traditional ensembles, making it a perfect fit for resource-constrained edge hardware.

What Problem Does This Paper Address?

Deploying deep learning on "the edge"—think battery-powered sensors, drones, or medical wearables—is a balancing act. You have very little memory and limited compute power.

Standard uncertainty methods fail this test:

  • Deep Ensembles: You have to store and run $M$ different models. If $M=10$, you need 10x the memory and 10x the time.
  • MC-Dropout: You run one model, but you have to do it ~100 times with different dropout masks to get a reliable estimate. The latency is a dealbreaker for real-time use.

The authors identified a key opportunity: in modern architectures like ResNet-32, the actual weights and biases of convolutional layers make up over 98% of the parameters. The normalization layers (like Batch Norm) account for less than 1%. Tiny-DE exploits this lopsidedness.

Parameter distribution in ResNet-32 showing that weights dominate the footprint.

Methods

The core idea of Tiny-DE is to share all the "heavy" convolutional and linear weights across the ensemble, but maintain $M$ distinct sets of parameters for the normalization layers.

The paper introduces two clever ways to run this on hardware:

  1. Sequential Inference: The model runs $M$ times, but a "router" cycles through different normalization parameters for each pass. This is ideal for specialized hardware like Computation-in-Memory (CiM) accelerators.
  2. Single-Shot (Parallel) Inference: This is the "aha" moment for standard hardware like FPGAs or TPUs. The authors created an EnsembleNorm layer. It repeats the input $M$ times to create a batch, then applies all $M$ sets of normalization parameters simultaneously in a single forward pass.
Comparison of Deep Ensemble, BatchEnsemble, and the proposed Tiny-DE in serial and parallel modes.

Datasets

The authors put Tiny-DE through a gauntlet of diverse tasks:

  • Image Classification: CIFAR-10 and CIFAR-100, including Out-of-Distribution (OoD) tests using SVHN and STL-10.
  • Regression: 10 different UCI benchmark datasets (e.g., Boston Housing, Concrete Strength).
  • Time-Series Forecasting: Atmospheric $CO_2$ concentration data.
  • Semantic Segmentation: Challenging datasets like CamVid (automotive), Kvasir-SEG (medical), and Pascal VOC.

Key Insights and Ideas

  • Diversity is "Cheap": The most surprising finding is that ensembling only the normalization layers—which represent a tiny fraction of the model—provides enough "internal state manipulation" to produce high-quality uncertainty estimates.
  • Hardware-First Design: Unlike many algorithmic papers, Tiny-DE is built with specific hardware constraints in mind. By offering both serial and parallel modes, it caters to both emerging CiM architectures and traditional vectorized accelerators.
  • The "Free" Accuracy Boost: Interestingly, ensembling normalization layers often improves the base accuracy of the model (up to 1%) while simultaneously providing uncertainty data.
  • Augmentation Matters: The authors found that training each ensemble member with different data augmentations significantly sharpens the uncertainty maps, especially in complex tasks like semantic segmentation.

Key Results

Tiny-DE punches well above its weight class:

  • Efficiency: It reduces latency and memory overhead by a factor of up to $\sim M\times$ compared to standard Deep Ensembles.
  • Regression: Achieved a 17.17% reduction in Root Mean Square Error (RMSE) across various benchmarks.
  • Uncertainty Quality: In OoD tests, Tiny-DE showed a much higher relative change in predictive entropy than baselines, meaning it is much better at "knowing what it doesn't know."
Predictive entropy distributions showing Tiny-DE's ability to clearly distinguish between seen (CIFAR-10) and unseen (SVHN) data.
  • Time Series: For $CO_2$ forecasting, it reduced RMSE by 17.7% compared to a single model.
Time-series prediction of CO2 levels, showing the ensemble mean tracking the ground truth closely with a clear uncertainty band.

Limitations and Future Work

While impressive, the paper notes a few areas for further exploration:

  • Training Memory: In the "Single-Shot" parallel mode, the effective batch size increases by $M$. While this is fine for inference on the edge, it can lead to memory overflow on GPUs during the training phase if $M$ is very large.
  • Diversity Ceiling: Ensembling only normalization layers might eventually hit a performance ceiling compared to full ensembles on extremely high-dimensional, complex datasets.
  • Future Directions: The authors suggest exploring different types of normalization (like Group or Layer Norm) and further optimizing the "router" logic for even lower-power digital implementations.

Summary updated

Abstract

The applications of artificial intelligence (AI) are rapidly evolving, and they are also commonly used in safety-critical domains, such as autonomous driving and medical diagnosis, where functional safety is paramount. In AI-driven systems, uncertainty estimation allows the user to avoid overconfidence predictions and achieve functional safety. Therefore, the robustness and reliability of model predictions can be improved. However, conventional uncertainty estimation methods, such as the deep ensemble method, impose high computation and, accordingly, hardware (latency and energy) overhead because they require the storage and processing of multiple models. Alternatively, Monte Carlo dropout (MC-dropout) methods, although having low memory overhead, necessitate numerous ($\sim 100$) forward passes, leading to high computational overhead and latency. Thus, these approaches are not suitable for battery-powered edge devices with limited computing and memory resources. In this paper, we propose the Tiny-Deep Ensemble approach, a low-cost approach for uncertainty estimation on edge devices. In our approach, only normalization layers are ensembled $M$ times, with all ensemble members sharing common weights and biases, leading to a significant decrease in storage requirements and latency. Moreover, our approach requires only one forward pass in a hardware architecture that allows batch processing for inference and uncertainty estimation. Furthermore, it has approximately the same memory overhead compared to a single model. Therefore, latency and memory overhead are reduced by a factor of up to $\sim M\times$. Nevertheless, our method does not compromise accuracy, with an increase in inference accuracy of up to $\sim 1\%$ and a reduction in RMSE of $17.17\%$ in various benchmark datasets, tasks, and state-of-the-art architectures.
BibTeX
@inproceedings{ahmed2024tiny,
  author = {Soyed Tuhin Ahmed and Michael Hefenbrock and Mehdi B. Tahoori},
  title = {{Tiny Deep Ensemble: Uncertainty Estimation in Edge AI Accelerators via Ensembling Normalization Layers with Shared Weights}},
  booktitle = {ICCAD},
  year = {2024},
  doi = {10.1145/3676536.3676804},
}

Loading related content...

Figure