Comprehensive Analysis of Neural Network Inference on Embedded Systems: Response Time, Calibration, and Model Optimisation

Patrick Huber, Ulrich Göhner, Mario Trapp, Jonathan Zender, Rabea LichtenbergSensors 2025 · DOI

TL;DR

This paper provides a comprehensive analysis of neural network inference on embedded systems, focusing on response times and model calibration. It reveals that while frameworks like TensorFlow Lite (TFLite) offer efficiency, their performance and calibration can be highly platform- and configuration-dependent, necessitating careful benchmarking and consideration beyond just accuracy.

What Problem Does This Paper Address?

The increasing adoption of Industry 4.0 and data-driven technologies necessitates performing complex computations, like neural network inference, directly on edge devices. However, these devices have limited computational power, making response time and efficient resource utilization critical. Furthermore, for safety-critical applications like predictive maintenance, the reliability of these predictions, quantified by model calibration, is paramount. Existing research often focuses on specific vision tasks or high-performance hardware, leaving a gap in understanding how generic neural network architectures perform across diverse embedded systems, especially concerning the interplay between network structure, hardware, and conversion to optimized formats like TFLite.

Methods

The authors conducted a systematic empirical study to evaluate neural network inference performance on three distinct embedded platforms: NXP 8MPLUSLPD4-EVK, Raspberry Pi 4 Model B, and NVIDIA Jetson AGX XAVIER. They focused on fully connected neural networks with varying architectures (number of layers, neurons per layer, input/output dimensions) and activation functions (ReLU, ELU, Sigmoid, TanH). Response times were measured using wall time over 100,000 invocations to ensure consistency and minimize interrupt-induced distortions. Model calibration was assessed using the Expected Calibration Error (ECE). A key aspect of their methodology was analyzing the impact of converting Keras models to TensorFlow Lite (TFLite) on both response times and calibration.

Datasets

The study utilized synthetic input data for response time measurements, as the focus was on performance rather than classification accuracy. For model calibration analysis, two specific use cases were employed:

  • Image Recognition: The Vision Transformer model ViT-B/16, pre-trained on ImageNet 2012, was used. The entire validation dataset of ImageNet 2012 was utilized for evaluation.
  • Signal Processing: A lightweight convolutional neural network model from the publicly available TFLite Speech Recognition demo was used, trained on the Speech Commands v2 test dataset.

Key Insights and Ideas

  • Platform Dependency is King: Response times are not just about the model but heavily influenced by the specific embedded hardware and its underlying architecture. What works efficiently on one platform might perform poorly on another.
  • Multi-threading Isn't Always Faster: Counterintuitively, multi-threading didn't always lead to improved response times. In some cases, it actually increased them, suggesting inefficiencies in current task partitioning and scheduling strategies within frameworks like TFLite, especially on heterogeneous hardware.
  • Activation Functions Matter for Parallelism: Beyond their computational complexity, different activation functions interact differently with multi-threaded execution. ReLU generally showed stable performance, while TanH struggled, indicating potential runtime scheduling issues.
  • TFLite Conversion: A Double-Edged Sword: While TFLite conversion generally preserves global calibration, it can alter local confidence patterns. This means that for applications relying on precise confidence estimates, post-conversion evaluation is crucial.
  • Calibration is More Than Just Accuracy: The study highlights that focusing solely on accuracy can be misleading. Mismatched confidence levels can lead to unreliable predictions, which is critical in safety-sensitive embedded applications.

Key Results

  • Response Time Correlation: A linear correlation was found between response times and the total number of trainable parameters in a network.
  • Hardware Ranking: For single-threading, the Jetson platform generally offered the best performance (lowest response times), followed by Raspberry Pi, and then IMX8. This ranking shifted for multi-threading.
  • Activation Function Performance: Across platforms, ReLU consistently yielded the lowest response times, followed by ELU, Sigmoid, and TanH, which generally showed the highest response times.
  • TFLite Calibration Impact: In the image recognition use case, TFLite conversion resulted in a marginal improvement in calibration (ECE decreased slightly). However, in the signal processing use case, while the overall ECE difference was minimal, there were shifts in confidence distributions and per-bin calibration errors, indicating subtle but potentially important changes.

Limitations and Future Work

The authors acknowledge that their study focused on fully connected networks and did not explore recurrent or attention-based architectures, which are becoming increasingly prevalent. They also note the lack of transparency in TFLite's multi-threading behavior, suggesting future work in reverse-engineering runtime scheduling.

Promising future directions include:

  • Targeted Pruning for Calibration: Investigating pruning strategies to improve model calibration, especially for larger networks that tend to be overconfident.
  • Extension to Other Architectures: Analyzing response times and calibration for more complex network types like recurrent and attention-based models.
  • Adaptive Scheduling Mechanisms: Developing frameworks that can dynamically adjust scheduling strategies based on runtime behavior to optimize inference performance.
  • Runtime Scheduling Analysis: Applying automated pattern recognition to TFLite's source code and runtime data to better understand scheduling dependencies and bottlenecks.

Summary updated

Abstract

The response time of Artificial Neural Network (ANN) inference is critical in embedded systems processing sensor data close to the source. This is particularly important in applications such as predictive maintenance, which rely on timely state change predictions. This study enables estimation of model response times based on the underlying platform, highlighting the importance of benchmarking generic ANN applications on edge devices. We analyze the impact of network parameters, activation functions, and single- versus multi-threading on response times. Additionally, potential hardware-related influences, such as clock rate variances, are discussed. The results underline the complexity of task partitioning and scheduling strategies, stressing the need for precise parameter coordination to optimise performance across platforms. This study shows that cutting-edge frameworks do not necessarily perform the required operations automatically for all configurations, which may negatively impact performance. This paper further investigates the influence of network structure on model calibration, quantified using the Expected Calibration Error (ECE), and the limits of potential optimisation opportunities. It also examines the effects of model conversion to Tensorflow Lite (TFLite), highlighting the necessity of considering both performance and calibration when deploying models on embedded systems.
BibTeX
@inproceedings{huber2025comprehensive,
  author = {Patrick Huber and Ulrich Göhner and Mario Trapp and Jonathan Zender and Rabea Lichtenberg},
  title = {{Comprehensive Analysis of Neural Network Inference on Embedded Systems: Response Time, Calibration, and Model Optimisation}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25154769},
}
Figure