Cloud Is Closer Than It Appears: Revisiting the Tradeoffs of Distributed Real-Time Inference

Hang Qiu, Pragya Sharma, Hang Qiu 0001, Mani B. Srivastava

TL;DR

This paper challenges the conventional wisdom that cloud-based inference is too slow for safety-critical real-time systems. By developing a formal analytical model and conducting high-fidelity simulations, the authors demonstrate that cloud inference can, under specific conditions, outperform on-device inference in terms of response time and safety, even for tasks like emergency braking.

What Problem Does This Paper Address?

Modern cyber-physical systems (CPS), such as autonomous vehicles, increasingly rely on deep neural networks (DNNs) for perception and decision-making. While DNNs offer unparalleled accuracy, they demand significant computational resources, posing a challenge for real-time control deadlines. Traditionally, to ensure low latency and avoid network variability, inference has been performed on-device. However, this approach burdens local hardware with high energy consumption and computational demands, limiting the complexity of models that can be deployed. This paper questions the assumption that cloud-based inference is inherently unsuitable for latency-sensitive tasks and investigates the conditions under which it can be a viable, or even superior, alternative.

Methods

The authors propose a multi-faceted approach:

  • Formal Analytical Model: They develop a generalized model that captures the interplay between sensing frequency, inference delay (both on-device and cloud), network latency, system load (modeled as M/M/1 queues), and task-specific safety constraints. This model allows for a principled evaluation of different model-platform configurations.
  • Lemmas for Analysis: Three key lemmas are introduced to formalize the analysis:
  • Lemma 1: Defines the condition under which cloud inference yields lower total response latency than on-device inference, highlighting the importance of queue-amortized inference latency and network delay.
  • Lemma 2: Addresses temporal misalignment in control actuation due to inference and network delays, introducing a "reaction-time budget" that must be met for safety.
  • Lemma 3: Incorporates detection delay, showing how model accuracy and sensitivity influence the overall temporal pipeline and affect safety.
  • Hardware-in-the-Loop Simulation: The analytical findings are validated using the CARLA simulator, which models realistic vehicular dynamics and sensor data. This allows for controlled experiments across various vehicle types, speeds, and driving conditions.
  • Empirical Profiling: The performance of different YOLOv11 object detection models is profiled on both an on-device (NVIDIA Jetson AGX Orin) and a cloud (RTX A5000 GPU) platform, measuring inference latency, detection accuracy, and energy consumption.

Datasets

The paper does not explicitly mention training datasets. For evaluation, they utilize:

  • CARLA simulator: A high-fidelity platform for urban driving environments, used to simulate vehicle dynamics, sensor data, and traffic scenarios.
  • COCO dataset: The YOLOv11 models used in the experiments are pre-trained on the COCO dataset.
  • AWS EC2 local zone measurements: Synthetic round-trip latencies were generated based on measurements of transmitting CARLA-generated image frames over urban Wi-Fi and 5G cellular networks.

Key Insights and Ideas

  • Cloud is "Closer Than It Appears": The central idea is that with modern high-throughput cloud GPUs and improved networking, the perceived distance and latency of the cloud can be effectively managed, making it competitive for real-time tasks.
  • Beyond Simple Latency: The paper emphasizes that a holistic view is crucial, considering not just raw inference latency but also network delays, queuing effects, and crucially, detection timeliness.
  • Queueing Delays Matter Even for On-Device: Even dedicated on-device platforms can experience queuing delays if inference time exceeds the frame inter-arrival interval, a scenario that can be exacerbated by runtime perturbations.
  • Detection Delay is Critical: A significant finding is that delayed detection, even with fast inference, can severely reduce the effective decision window and lead to unsafe outcomes, highlighting the importance of model accuracy and sensitivity.
  • Context is King: The optimal inference placement is highly dependent on the specific application, operating context (speed, vehicle type, obstacle size), and workload conditions. There is no universal "best" solution.
  • Cloud Scalability Advantage: Under multi-tenant workloads, cloud platforms demonstrate better scalability and more graceful degradation compared to on-device systems, which can exhibit sharp latency spikes.

Key Results

  • Cloud Outperforms On-Device in Baseline: In a baseline scenario, cloud-based inference (using YOLOv11-xlarge) consistently enabled earlier detection and shorter stopping distances compared to on-device inference (using YOLOv11-medium), even with a median round-trip network delay of 22 ms.
  • Tail Latency is a Concern: Under high network latency (tail latency), the performance gap between cloud and on-device narrowed, and in some critical cases (e.g., high-speed truck), cloud deployment led to unsafe braking.
  • Concurrent Workloads Impact: While cloud GPU utilization increased with concurrent workloads, latency remained relatively stable. On-device GPUs, however, showed significant latency spikes even under low additional load.
  • Smaller Obstacles Reduce Gap: For smaller obstacles, the performance gap between cloud and on-device reduced, as detection delays became more prominent for both.
  • Cloud Resilience to Obstacle Size: The cloud configuration proved more resilient to variations in obstacle size due to its ability to deploy larger, more accurate models.

Limitations and Future Work

  • Simplified Network Model: The paper uses a simplified M/M/1 queue model for network and platform delays. Real-world network conditions can be more complex.
  • Focus on Single Task: The analysis is focused on emergency braking. Extending the framework to other safety-critical CPS tasks is suggested.
  • No Sensor Fusion: The current analysis abstracts away sensor fusion, which is common in robust perception systems.
  • Future Work:
    • Investigating hybrid architectures where cloud and on-device systems collaborate.
  • Incorporating dynamic frame rate adjustment based on context.
  • Analyzing the impact of data encryption on feasibility.
  • Extending the model to include real-time tracking and trajectory forecasting.
  • Generalizing the framework to a wider range of real-time systems beyond braking.

Summary updated

BibTeX
@inproceedings{qiu2025cloud,
  author = {Hang Qiu and Pragya Sharma and Hang Qiu and Mani B. Srivastava},
  title = {{Cloud Is Closer Than It Appears: Revisiting the Tradeoffs of Distributed Real-Time Inference}},
  booktitle = {Icccn},
  year = {2025},
  doi = {10.1109/icccn65249.2025.11133852},
}
Figure