Interrupting Real-Time IoT Tasks: How Bad Can It Be to Connect Your Critical Embedded System to the Internet?

Ilja Behnke, Lukas Pirl, Lauritz Thamsen, Robert Danicki, Andreas Polze, Odej Kao

TL;DR

This paper investigates the disruptive impact of network connectivity on the real-time performance of embedded systems. The authors demonstrate that network traffic, particularly interrupts, can significantly degrade the predictability and timeliness of critical tasks, potentially invalidating real-time guarantees. Their findings highlight a critical challenge for the widespread adoption of IoT in safety-critical applications.

What Problem Does This Paper Address?

Embedded systems are the backbone of many critical applications, from industrial automation to automotive systems, where precise timing and predictable performance are paramount. Traditionally, these systems operate in isolated environments, managing their tasks deterministically using Real-Time Operating Systems (RTOSs). However, the rise of the Internet of Things (IoT) is pushing these systems to connect to networks, exposing them to external stimuli like network packets. These packets trigger hardware interrupts, which can disrupt the carefully orchestrated execution of real-time tasks. The core problem this paper addresses is quantifying how bad this disruption can be, specifically examining the impact of network-generated interrupts and the overhead of networking tasks on the timing guarantees of critical embedded systems.

Methods

The authors employ a quantitative experimental approach to measure the impact of network loads on real-time tasks. They designed two experimental setups to isolate different aspects of the problem:

  1. Real Network Packets: This setup involves sending actual network packets to the devices over Wi-Fi. The framework-supplied drivers and networking tasks handle these packets.
  2. Simulated Packets: To control the experiment and isolate the interrupt generation, a pseudo network driver was implemented. This simulation mimics the actions of a packet reception by triggering an interrupt via an input pin, which then queues a packet descriptor for processing. This allows for a more controlled analysis of interrupt-induced delays versus the full network stack overhead.

They measured two key metrics:

  • Lateness: This quantifies how much a critical periodic task exceeds its deadline. They report accumulated lateness per second.
  • Relative MPU Utilization: This measures the number of task cycles completed within a fixed interval, indicating how much processing time is consumed by networking operations under varying network loads.

Experiments were conducted on two common IoT development boards: an ESP32 (dual-core) and a Particle Photon (single-core ARM Cortex M3). They also varied the priority of the observed critical task relative to the networking tasks to understand the impact of preemption.

Datasets

Not discussed in the paper. The experiments utilize custom network traffic generation and simulation rather than pre-existing datasets.

Key Insights and Ideas

  • Network Interrupts are a Real Threat: The study unequivocally shows that network-generated interrupts are not a trivial concern for real-time embedded systems. They can significantly degrade performance and potentially break timing guarantees.
  • ISR Overhead is Measurable: While often minimized in traditional systems, the paper quantifies the lateness introduced by Interrupt Service Routines (ISRs) themselves, attributing up to 1% lateness per packet per second solely to ISR delays.
  • Full Network Stack is the Bigger Culprit: The combined overhead of networking tasks, drivers, and ISRs leads to a much larger performance degradation, with critical task performance loss reaching up to 6.67% per received packet per second.
  • Task Priority Matters, But Has Limits: While adjusting task priorities can mitigate some issues (e.g., a high-priority critical task can starve networking tasks), it doesn't eliminate the problem entirely and can lead to system instability or crashes.
  • Frameworks Obscure Control: The use of vendor-provided frameworks, while simplifying development, often hides low-level details like driver source code, limiting developers' ability to fine-tune or fully control the system's real-time behavior.
  • Unused Ports Can Be Worse: Interestingly, network traffic directed to unused ports sometimes showed a higher impact, possibly due to partial deactivation of networking tasks when buffers are full.

Key Results

  • Lateness: A critical task's lateness can increase by up to 6.67% per received packet per second due to network traffic. ISR-generated delays alone can account for up to 1% lateness per packet per second.
  • Task Performance Degradation: In utilization experiments, task performance decreased by 2.15% to 6.67% depending on the network load and priority configurations.
  • System Instability: The experiments revealed that even low network traffic loads could lead to Wi-Fi driver crashes and system halts on the Particle Photon, highlighting robustness issues.
  • ESP32 vs. Particle Photon: While both platforms showed similar trends, the ESP32's dual-core architecture offered some resilience, whereas the single-core Photon was more susceptible to crashes.

Limitations and Future Work

Limitations:

  • Initial Study: The authors acknowledge this is an "initial study" and call for a "more comprehensive study with a broader range of IoT devices and test scenarios."
  • Framework Dependence: The results are tied to specific vendor frameworks, and the limited access to low-level driver source code restricts the depth of analysis and potential for optimization.
  • Specific Hardware: The experiments were conducted on two specific development boards, and results may vary on other hardware.

Future Work:

  • Investigate Mitigation Techniques: The authors plan to explore mitigation techniques specifically for NIC-generated ISR delays in real-time systems.
  • Broader Device Range: Expanding the study to include a wider variety of IoT devices and architectures.
  • More Complex Scenarios: Testing with more complex network protocols and traffic patterns.

Summary updated

Abstract

Embedded systems have been used to control physical environments for decades. Usually, such use cases require low latencies between commands and actions as well as a high predictability of the expected worst-case delay. To achieve this on small, low-powered microcontrollers, Real-Time Operating Systems (RTOSs) are used to manage the different tasks on these machines as deterministically as possible. However, with the advent of the Internet of Things (IoT) in industrial applications, the same embedded systems are now equipped with networking capabilities, possibly endangering critical real-time systems through an open gate to interrupts.This paper presents our initial study of the impact network connections can have on real-time embedded systems. Specifically, we look at three aspects: The impact of network-generated interrupts, the overhead of the related networking tasks, and the feasibility of sharing computing resources between networking and real-time tasks. We conducted experiments on two setups: One treating NICs and drivers as black boxes and one simulating network interrupts on the machines. The preliminary results show that a critical task performance loss of up to 6.67% per received packet per second could be induced where lateness impacts of 1% per packet per second can be attributed exclusively to ISR-generated delays.
BibTeX
@inproceedings{behnke2020interrupting,
  author = {Ilja Behnke and Lukas Pirl and Lauritz Thamsen and Robert Danicki and Andreas Polze and Odej Kao},
  title = {{Interrupting Real-Time IoT Tasks: How Bad Can It Be to Connect Your Critical Embedded System to the Internet?}},
  booktitle = {IPCCC},
  year = {2020},
  url = {https://openreview.net/forum?id=dNOC8sKTvF},
}
Figure